![首頁 Logo Keith ]p !!](/themes/mytheme/logo.png){kind=logo}
最後更新: 2014-10-22
目錄
- instance to instance
- flat ip
- float ip
- nova-network 的 iptables
Networking
In any setup with flat networking
The hosts providing the nova-network service are responsible for forwarding traffic from the private network.
They also run and configure dnsmasq as a DHCP server listening on this bridge, usually on IP address 10.0.0.1
* in multi-host mode every compute node has its own nova-network
The compute can determine the NAT entries for each network, although sometimes NAT is not used
Network:
Virtual Networking Infrastructure (VNI)
Physical Networking Infrastructure (PNI)
Legacy networking primarily involves compute nodes.
Controller node:
# 設定
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api nova
# Restart the Compute services:
service openstack-nova-api restart
service openstack-nova-scheduler restart
service openstack-nova-conductor restart
Compute node:
安裝
yum install openstack-nova-network openstack-nova-api
設定
# network_api 及 network_manager 的設定
openstack-config --set /etc/nova/nova.conf DEFAULT network_api_class nova.network.api.API
openstack-config --set /etc/nova/nova.conf DEFAULT security_group_api nova
openstack-config --set /etc/nova/nova.conf DEFAULT network_manager nova.network.manager.FlatDHCPManager
openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.libvirt.firewall.IptablesFirewallDriver
# INTERFACE_NAME with the actual interface name for the external network
INTERFACE_NAME=eth0
# nova.network.manager - Number of addresses in each private subnet
openstack-config --set /etc/nova/nova.conf DEFAULT network_size 254
# defined in nova.virt.firewall
openstack-config --set /etc/nova/nova.conf DEFAULT allow_same_net_traffic False
# all compute hosts share the same dhcp address
openstack-config --set /etc/nova/nova.conf DEFAULT multi_host True
openstack-config --set /etc/nova/nova.conf DEFAULT send_arp_for_ha True
openstack-config --set /etc/nova/nova.conf DEFAULT share_dhcp_address True
openstack-config --set /etc/nova/nova.conf DEFAULT force_dhcp_release True
# nova.network.manager
openstack-config --set /etc/nova/nova.conf DEFAULT flat_network_bridge br100
# FlatDhcp will bridge into this interface if set, (它亦是 compute node 互連的 interface)
openstack-config --set /etc/nova/nova.conf DEFAULT flat_interface $INTERFACE_NAME
# Interface for public IP addresses (此 interface 多數會成為 bridge 的 member)
openstack-config --set /etc/nova/nova.conf DEFAULT public_interface $INTERFACE_NAME
# Startup Service
service openstack-nova-network start
service openstack-nova-metadata-api start
chkconfig openstack-nova-network on
chkconfig openstack-nova-metadata-api on
To create the network
Controller:
source admin-openrc.sh
NETWORK_CIDR=192.168.123.96/30
nova network-create demo-net --bridge br100 --multi-host T --fixed-range-v4 $NETWORK_CIDR
* --fixed-range-v4- specifies the network subnet.
* --fixed-cidr specifies a range of fixed IP addresses to allocate (subset of the --fixed-range-v4 argument )
# Verify
nova net-list
+--------------------------------------+----------+--------------------+ | ID | Label | CIDR | +--------------------------------------+----------+--------------------+ | 17046477-72c6-4d79-bbf1-2a9291407c10 | demo-net | 192.168.123.128/29 | +--------------------------------------+----------+--------------------+
Network node iptables
NAT
iptables -t nat -S
-A nova-network-OUTPUT -d 10.3.4.1/32 -j DNAT --to-destination 192.168.32.3 -A nova-network-POSTROUTING -s 192.168.32.0/22 -d 192.168.88.211/32 -j ACCEPT -A nova-network-POSTROUTING -s 192.168.32.0/22 -d 192.168.32.0/22 -m conntrack ! --ctstate DNAT -j ACCEPT -A nova-network-POSTROUTING -s 192.168.32.3/32 -m conntrack --ctstate DNAT -j SNAT --to-source 10.3.4.1 -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.88.211:8775 -A nova-network-PREROUTING -d 10.3.4.1/32 -j DNAT --to-destination 192.168.32.3 -A nova-network-float-snat -s 192.168.32.3/32 -d 192.168.32.3/32 -j SNAT --to-source 10.3.4.1 -A nova-network-float-snat -s 192.168.32.3/32 -o eth0 -j SNAT --to-source 10.3.4.1 -A nova-network-snat -j nova-network-float-snat -A nova-network-snat -s 192.168.32.0/22 -o eth0 -j SNAT --to-source 192.168.88.211
Filter
# OUTPUT 及 FORWARD 都會跳到 nova-filter-top
-A FORWARD -j nova-filter-top -A OUTPUT -j nova-filter-top
# nova-filter-top
-A nova-filter-top -j nova-compute-local
-A nova-filter-top -j nova-network-local
-A nova-filter-top -j nova-api-local# nova-compute-local
-A nova-compute-local -d 192.168.32.3/32 -j nova-compute-inst-19# nova-compute-inst-19
-A nova-compute-inst-19 -m state --state INVALID -j DROP -A nova-compute-inst-19 -m state --state RELATED,ESTABLISHED -j ACCEPT -A nova-compute-inst-19 -j nova-compute-provider -A nova-compute-inst-19 -s 192.168.32.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT -A nova-compute-inst-19 -s 192.168.32.0/22 -j ACCEPT -A nova-compute-inst-19 -p tcp -m multiport --dports 1:65535 -j ACCEPT -A nova-compute-inst-19 -p icmp -m icmp --icmp-type 8/0 -j ACCEPT -A nova-compute-inst-19 -j nova-compute-sg-fallback
# Last Rule
-A nova-compute-sg-fallback -j DROP
openstack@compute-1:~$ ip a
dnsmasq 設定
# The nova-network service is responsible for starting up dnsmasq processes.
dnsmasq_config_file=/etc/dnsmasq-nova.conf
Troubleshoot
All-In-One node:
# list firewall rule
nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+ | IP Protocol | From Port | To Port | IP Range | Source Group | +-------------+-----------+---------+-----------+--------------+ | icmp | 8 | 0 | 0.0.0.0/0 | | | tcp | 22 | 22 | 0.0.0.0/0 | | +-------------+-----------+---------+-----------+--------------+
# Add by cmd
$ nova secgroup-add-rule default icmp 8 0 0.0.0.0/0
$ nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
# check iptables
iptables -L -nv -t nat
Chain nova-network-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
8 492 DNAT all -- * * 0.0.0.0/0 10.3.4.1 to:192.168.32.3
Chain nova-network-float-snat (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT all -- * * 192.168.32.3 192.168.32.3 to:10.3.4.1
3 252 SNAT all -- * eth0 192.168.32.3 0.0.0.0/0 to:10.3.4.1ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:c9:9c:27 brd ff:ff:ff:ff:ff:ff
inet 192.168.88.211/24 brd 192.168.88.255 scope global eth0
inet 10.3.4.1/32 scope global eth0
inet6 fe80::5054:ff:fec9:9c27/64 scope link
valid_lft forever preferred_lft forever
Disable firewall
To help debug networking issues with reaching VMs,
you can disable the firewall by setting the following option in
/etc/nova/nova.conf:
firewall_driver=nova.virt.firewall.NoopFirewallDriver
use tcpdump to identify if packets are being routed to the inbound interface on the compute host.
* If the packets are reaching the compute hosts but the connection is failing
reverse-path
sysctl -w net.ipv4.conf.ETH2.rp_filter=0
仍未有效:
# 應該是有 2 個 dnsmasq processes 的
killall dnsmasq
# 1 個 nova-network
service openstack-nova-network restart
instance to instance
/etc/nova/nova.conf
# defaults to true
allow_same_net_traffic=true
* On a flat network, this allows all instances from all projects unfiltered communication.
* With VLAN networking, this allows access between instances within the same project.
Command
flat ip
nova network-list
+--------------------------------------+-------------+-----------------+ | ID | Label | Cidr | +--------------------------------------+-------------+-----------------+ | 634b8d86-10f6-41d5-b35a-19dd0e1c68a5 | novanetwork | 192.168.32.0/22 | +--------------------------------------+-------------+-----------------+
nova network-show novanetwork
+---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | bridge | br100 | | bridge_interface | eth1 | | broadcast | 192.168.35.255 | | cidr | 192.168.32.0/22 | | cidr_v6 | - | | created_at | 2014-10-15T08:53:07.000000 | | deleted | 0 | | deleted_at | - | | dhcp_start | 192.168.32.2 | | dns1 | 8.8.4.4 | | dns2 | - | | gateway | 192.168.32.1 | | gateway_v6 | - | | host | openstack.local | | id | 634b8d86-10f6-41d5-b35a-19dd0e1c68a5 | | injected | False | | label | novanetwork | | multi_host | False | | netmask | 255.255.252.0 | | netmask_v6 | - | | priority | - | | project_id | - | | rxtx_base | - | | updated_at | 2014-10-15T10:33:40.000000 | | vlan | - | | vpn_private_address | - | | vpn_public_address | - | | vpn_public_port | - | +---------------------+--------------------------------------+
interface-attach
interface-detach
add-fixed-ip # Add new IP address on a network to server.
remove-fixed-ip # Remove an IP address from a server.
float ip
# 找可用的 IP
nmap -sP 192.168.88.221-230
# 查看有什麼 pool
nova floating-ip-pool-list
+------+ | name | +------+ | nova | +------+
# 查看有什麼 float ip
nova floating-ip-bulk-list
+----------------------------------+------------+---------------+------+-----------+ | project_id | address | instance_uuid | pool | interface | +----------------------------------+------------+---------------+------+-----------+ | - | 10.3.4.1 | - | nova | eth0 | | 7af17721fd3645d2a9510812dff57806 | 10.3.4.2 | - | nova | eth0 | | 7af17721fd3645d2a9510812dff57806 | 10.3.4.3 | - | nova | eth0 | | - | 10.3.4.4 | - | nova | eth0 | | - | 10.3.4.5 | - | nova | eth0 | | - | 10.3.4.6 | - | nova | eth0 |
# floating IP addresses that are allocated
# 有 project 拿了 ip 才有 output
nova floating-ip-list
+----------+-----------+----------+------+ | Ip | Server Id | Fixed Ip | Pool | +----------+-----------+----------+------+ | 10.3.4.2 | | - | nova | | 10.3.4.3 | | - | nova | +----------+-----------+----------+------+
floating-ip-bulk-create
# nova floating-ip-bulk-create [--pool <pool>] [--interface <interface>] <range>
# interface 不是 br100, 而是 bridge 去 br100 的 eth0
nova floating-ip-bulk-create --pool nova --interface eth0 192.168.88.221
沒有 output 的
nova floating-ip-bulk-delete
nova floating-ip-bulk-delete 10.3.4.0/24
沒有 output 的
# Allocate a floating IP (172.24.4.225 * ramdom) for the current tenant (keystonerc_admin)
floating-ip-create
# Associate a floating IP address to VM1 ( or ID )
nova floating-ip-associate VM1 172.24.4.225
# Disassociate a floating IP address from a server.
floating-ip-disassociate
# De-allocate a floating IP
floating-ip-delete
DHCP
cat /var/lib/nova/networks/nova-br100.conf
fa:16:3e:2a:86:2a,test21.novalocal,192.168.32.2
When nova-network was creating this instance, it chose an IP address for it from the pool of free fixed IP addresses
(network configuration of an instance is done in nova/network/manager.py, method allocate_for_instance).
The first available IP turned out to be 10.0.0.2 (availability of fixed and floating IPs is stored in the Nova database). Then, dnsmasq w
nova-network 的 network 與 iptables (FlatNetworking)
結構:
/ eth0|controller_node|eth1 # eth0 = float range
SW | # eth1 = fixed range
eth0|compute_node|eth1
* eth0 = "Promiscuous mode" # allows the interface to receive packets not targeted to this interface’s MAC address.
Traffic:
vm(eth0) --> (br100)compute(eth1) --> (eth1) controller (eth0) --> sw
* nova-network(vm 的 default gateway) responsible for forwarding traffic from the private network(fixed_range) to public (floating_range)
nova.conf:
fixed_range=192.168.32.0/22 <-- 自動 assign 到 vm 的
floating_range=10.3.4.0/22 <-- map 去 fixed_ip 的
Bridge
brctl show
bridge name bridge id STP enabled interfaces
br100 8000.525400572143 no eth1
vnet0Interface
ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:C9:9C:27
inet addr:192.168.88.211 Bcast:192.168.88.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fec9:9c27/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:610769 errors:0 dropped:0 overruns:0 frame:0
TX packets:311857 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:777008134 (741.0 MiB) TX bytes:29263466 (27.9 MiB)
eth1 Link encap:Ethernet HWaddr 52:54:00:57:21:43
inet6 addr: fe80::5054:ff:fe57:2143/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:468 (468.0 b) TX bytes:3674 (3.5 KiB)
Iptables:
INPUT:
-A INPUT -j nova-api-INPUT
-A INPUT -j nova-network-INPUT
-A INPUT -j nova-compute-INPUT
FORWARD:
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A FORWARD -j nova-network-FORWARD
-A FORWARD -j nova-compute-FORWARD
OUTPUT:
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-compute-OUTPUT
nova-filter-top
-A nova-filter-top -j nova-api-local <-- 無野
-A nova-filter-top -j nova-network-local <-- 無野
-A nova-filter-top -j nova-compute-local
nova-compute-local
-A nova-compute-local -d 192.168.32.2/32 -j nova-compute-inst-21
nova-compute-inst-21:
-A nova-compute-inst-21 -m state --state INVALID -j DROP
-A nova-compute-inst-21 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-compute-inst-21 -j nova-compute-provider
-A nova-compute-inst-21 -s 192.168.32.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A nova-compute-inst-21 -s 192.168.32.0/22 -j ACCEPT
-A nova-compute-inst-21 -p icmp -m icmp --icmp-type 8/0 -j ACCEPT
-A nova-compute-inst-21 -p tcp -m tcp --dport 22 -j ACCEPT
-A nova-compute-inst-21 -j nova-compute-sg-fallback
nova-compute-sg-fallback
-A nova-compute-sg-fallback -j DROP
nova-network-FORWARD:
-A nova-network-FORWARD -i br100 -j ACCEPT -A nova-network-FORWARD -o br100 -j ACCEPT
nova-network-INPUT:
Chain nova-network-INPUT (1 references)
pkts bytes target prot opt in out source destination
5 1540 ACCEPT udp -- br100 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- br100 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
2 152 ACCEPT udp -- br100 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- br100 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53nova-network-OUTPUT:
Chain nova-network-OUTPUT (1 references) pkts bytes target prot opt in out source destination
所以次序是:
FORWARD -> nova-filter-top -> nova-compute-local -> nova-compute-inst-21-> nova-compute-sg-fallback
NAT:
PREROUTING
-A PREROUTING -j nova-api-PREROUTING
-A PREROUTING -j nova-network-PREROUTING
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.88.211:8775 -A nova-network-PREROUTING -d 192.168.88.221/32 -j DNAT --to-destination 192.168.32.2
-A PREROUTING -j nova-compute-PREROUTING
POSTROUTING
-A POSTROUTING -j nova-api-POSTROUTING
-A POSTROUTING -j nova-network-POSTROUTING
-A nova-network-POSTROUTING -s 192.168.32.0/22 -d 192.168.88.211/32 -j ACCEPT -A nova-network-POSTROUTING -s 192.168.32.0/22 -d 192.168.32.0/22 -m conntrack ! --ctstate DNAT -j ACCEPT -A nova-network-POSTROUTING -s 192.168.32.2/32 -m conntrack --ctstate DNAT -j SNAT --to-source 192.168.88.221
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
# POSTROUTING --> nova-postrouting-bottom --> nova-network-snat --> nova-network-float-snat
-A nova-postrouting-bottom -j nova-network-snat -A nova-network-float-snat -s 192.168.32.2/32 -d 192.168.32.2/32 -j SNAT --to-source 192.168.88.221 -A nova-network-float-snat -s 192.168.32.2/32 -o eth0 -j SNAT --to-source 192.168.88.221
OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-compute-OUTPUT
-A nova-network-OUTPUT -d 192.168.88.221/32 -j DNAT --to-destination 192.168.32.2
nova-network FlatDHCPManager component
============================================ Rule for metadata host
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 192.168.1.90/32 -j ACCEPT
# ip addr add 169.254.169.254/32 scope link dev lo
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.90:8775
* If the nova-network and nova-api are not running on the same host,
you must define the metadata_host on the nova-network host to point to the nova-api host.
============================================ Rule for VMs to connect each other
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -m conntrack ! --ctstate DNAT -j ACCEPT
============================================ Rule for access outside of fixed subnet
-A nova-network-snat -s 10.0.0.0/8 -j SNAT --to-source 192.168.1.90
============================================ Rules for each network
Create a network and set the host:
# ./bin/nova-manage network create mynet 10.10.10.0/24
Boot a server:
nova boot --image a3fb743d-42df-49ba-b9c4-8042ebbd344e --flavor 1 myserver
After executing these commands, you have these rules:
Allow a forwarded traffic pass bridge so the IP on br100 can work as a gateway:
-A nova-network-FORWARD -i br100 -j ACCEPT
-A nova-network-FORWARD -o br100 -j ACCEPT
Allow DHCP and DNS traffic to come into the local dnsmasq process:
-A nova-network-INPUT -i br100 -p udp -m udp --dport 67 -j ACCEPT
-A nova-network-INPUT -i br100 -p tcp -m tcp --dport 67 -j ACCEPT
-A nova-network-INPUT -i br100 -p udp -m udp --dport 53 -j ACCEPT
-A nova-network-INPUT -i br100 -p tcp -m tcp --dport 53 -j ACCEPT
============================================ nova-api
On starting, nova-api creates a rule in the filter table to allow others to access the nova-api service.
-A nova-api-INPUT -d 192.168.1.90/32 -p tcp -m tcp --dport 8775 -j ACCEPT
============================================ Rules to allow forwarded traffic pass bridge
These rules on the nova-compute host allow VMs to connect with nova-network host and VMs on other compute hosts.
-A nova-compute-FORWARD -i br100 -j ACCEPT
-A nova-compute-FORWARD -o br100 -j ACCEPT
============================================ Create a floating IP in the default pool
# create floating range
nova-manage floating create --ip_range=192.168.1.232/30
# allocate a floating IP "X" from pool:
nova floating-ip-create
# FLAGS.public_interface is used to bind floating IPs. After you run the nova add-floating-ip command
# got an IP of "X". Now assign it to the instance that has an ID "Y"
nova add-floating-ip Y X
* Rules in NAT table for floating IPs
# allows the local process on nova-network to access VMs with floating IPs.
-A nova-network-OUTPUT -d X/32 -j DNAT --to-destination 10.10.10.2
# If a packet arrives at the nova-network host with the floating IP as target IP
-A nova-network-PREROUTING -d X/32 -j DNAT --to-destination 10.10.10.2
# Since all the traffic from VMs to outside of the fixed network is pointed to gateway, which is set by the nova-network's dnsmasq process
-A nova-network-float-snat -s 10.10.10.2/32 -j SNAT --to-source X
============================================ Ping VMs with floating IP
on the nova-compute host, a specific chain for each instance
(nova-compute creates one chain for each instance.)
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
# rule created under that instance's specific chain
-A nova-compute-inst-1 -p icmp -j ACCEPT
multi-host functionality
no single network controller acts as a single point of failure.
(This process configures each compute node for networking.)
# compute node
apt-get install nova-network nova-api-metadata
/etc/nova/nova.conf
[DEFAULT] ... network_manager=nova.network.manager.FlatDHCPManager firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver network_size=254 allow_same_net_traffic=False multi_host=True send_arp_for_ha=True share_dhcp_address=True force_dhcp_release=True flat_network_bridge=br100 flat_interface=eth1 public_interface=eth1
# Restart the network service:
service nova-network restart
Component
- nova.network.manager
- nova.network.linux_net
#### nova.network.manager
# If True, send a dhcp release on instance termination force_dhcp_release=True # Seconds after which a deallocated IP is disassociated #fixed_ip_disassociate_timeout=600
#### nova.network.linux_net
# Interface for public IP addresses public_interface=eth0 # uses specific DNS server for dnsmasq dns_server=8.8.8.8 # The IP address for the metadata API server metadata_host=192.168.88.211
Firewalls and default ports
8776 Block Storage (cinder) publicurl and adminurl
8774 Compute (nova) endpoints publicurl and adminurl
8773, 8775 Compute API (nova-api)
5900-5999 Compute ports for access to virtual machine consoles
6080 Compute VNC proxy for browsers ( openstack-nova-novncproxy)
35357 Identity service (keystone) administrative endpoint * adminurl
5000 Identity service public endpoint * publicurl
9292 Image Service (glance) API publicurl and adminurl
9191 Image Service registry
9696 Networking (neutron) publicurl and adminurl
8777 Telemetry (ceilometer) publicurl and adminurl
3260 iSCSI target OpenStack Block Storage. Required.
3306 MySQL database service <-- Block Storage, Networking, and Compute 都要 connect 它.
5672 Message Broker (AMQP traffic) <--
nova-network
If you restart nova-netork for instance, then all the iptables rules are flushed and recreated according to your network topology.
The iptables service doesn't need
