What if you want to setup a private network for a set of virtual machines running on dom0? What if you want this private network to still be able to access the WWW (when available) ?
I found myself in this predicament as I have a set of virtual machines (Windows, SLES, SLED etc) running on my Thinkpad T61p. The nature of my [mobile] work is such that there isn't always a LAN cable or a wifi connection available readily. Irregardless of my network environment, I need to perform testing and demonstration of software running on these virtual machines. Thus, I need a flexible setup with private networking for my domUs with Network Address Translation (NAT) for accessing the WWW when a physical LAN or Wifi becomes available.
Thanks to Till and Kai, my new German connections
, the following is how I did it on SLED 10 SP2* ...* - Note that official production support for Xen is for SLES only. I'm using SLED as a development & testing desktop and the following steps will work on SLES as well.
Attention: Linux commands in braces [ ] are executed as root
1) Stop Xen daemon with [ rcxend stop ]
2) Remove the default bridge networking by editing the config file /etc/xen/xend-config.sxp. Look for the following 2 lines and comment them out with hashes ##:
(network-script network-bridge)
(vif-script vif-bridge)
becomes
##(network-script network-bridge)
##(vif-script vif-bridge)
3) Setting up the bridge to physical network (eg eth0)
Create a network bridge br0 to the physical network device (ie eth0) by creating the file /etc/sysconfig/network/ifcfg-br0 with the following contents:
BRIDGE='yes'
STARTMODE='onboot'
BRIDGE_PORTS='eth0'
BOOTPROTO='dhcp'
BROADCAST=''
For br0 to work, it has to have the IP address. Therefore, change the physical network device (ie eth0) to not start the DHCP client routine by editing the /etc/sysconfig/network/ifcfg-eth-id-xxx file with the following setting:
BOOTPROTO='none'
4) Setting up a private network (eg. 192.168.1.x)
Create a private network bridge br1 by creating the file /etc/sysconfig/network/ifcfg-br1 with the following contents:
BRIDGE='yes'
STARTMODE='onboot'
IPADDR='192.168.1.1'
NETMASK='255.255.255.0'
5) Changing the firewall settings for br0 and br1.
Change the firewall to allow network traffic for br0 and br1 as an external and internal network device respectively. Edit the file /etc/sysconfig/SuSEfirewall2 and change the following settings as shown below:
FW_DEV_EXT="br0"
FW_DEV_INT="br1"
FW_ROUTE="yes"
FW_MASQUERADE="yes"
6) Restart networking and start Xen daemon
Execute the following in order:
[ SuSEconfig ]
[ rcnetwork restart ]
[ rcxend start ]
7) Edit each domUs config in /etc/xen/vm directory to include ,bridge=br1 as follows:
From:
vif=[ 'mac=00:16:3e:75:06:c3,model=rtl8139,type=ioemu', ]
To:
vif=[ 'mac=00:16:3e:75:06:c3,model=rtl8139,type=ioemu,bridge=br1', ]
Refresh this change for each domU via:
[ xm delete domUName ]
[ xm new domUName ]
8) Boot up your virtual machines (domUs) and setup IP address in the range of 192.168.1.xxx. Remember to set the default gateway to 192.168.1.1.
For Windows VMs: Control Panel, Network Connections, from network device, right-click properties, double-click Internet Protocol (TCP/IP), set a unique fixed IP within the range of 192.168.1.[2-254], subnet mask to 255.255.255.0 and your Default gateway to 192.168.1.1. Click OK and OK again to affect the change.
For SLES VMs (Linux): setting IP to 192.168.1.10 in example below:
[ ip addr add 192.168.1.10/24 dev eth0 ]
[ ip link set eth0 up ]
[ ip route add default via 192.168.1.1 ]
You should now be able to ping all your virtual machines (domUs) from dom0 and vice versa.
Have fun!
Hi,
ReplyDeleteI like this article. It is very clear and describes in detail. I can ping Dom 0 from other domains. However, I still cannot access the Internet. I cannot ping with IP or domain name. Could you give me some hints about how to deal with this problem?
Hi Tommy,
ReplyDeleteOne thing I did left out is that the domUs must have the same DNS as the dom0. Since we did not configure DHCP or DNS, you will need to manually create an IP and the DNS search within each domU.
Hope this will help. Thank you.
Hi Kam han,
ReplyDeleteThank you for quick response. I tried but still no luck. I used plain ip nor domain name to ping google but failed, so I think probably the problem is not due to DNS. By the way, I use DHCP to fetch the real ip and dns. Any ideas?
Hi Tommy,
ReplyDeleteRunning out of ideas... I did recall seeing in another setup where the domU can surf but just not ping google. Its not my machine and I did not have time to debug it about a month back.
Are you running SLES10 or Windows as your domU (not that it matters), try copying the /etc/resolv.conf values into the domU and see if that works.
Sorry, I can't help you further on this one. :)
Never mind. Thanks a lot for helping me on this issue. I use SLES11. I don't know why setting up network in xen is such a pain:-(. What is the "proper" way to set up network in DomU? Or should I use other linux distribution to make it work? Thank you.
ReplyDeletehii.. i need to know will this work for a "static ip" on the eth0..? if there is any configuration changes required can u let me know..??please...
ReplyDeleteHi Sri,
ReplyDeleteYes, you can have static IP on dom0 eth0. :)
hii.. i treid the above steps exactly, and when i restart network, the system goes out of network reach.. is there any extra configuration to make..??i have the same configuration..
ReplyDeleteReally interesting and good basic explanation. Anyway this way would not be good if either you use NetworkManager or you disabled SuSEFirewall at all. Is it right?
ReplyDeleteHi Sri,
ReplyDeletePlease define "out of network reach"...
I've looked at my blog entry and the only thing I have not included is the need to sync your /etc/resolv.conf.
If your host IP is static, can you surf the web and ping other hosts? If so, your domUs should be able to at least ping the other hosts as well. To surf, you will need to copy the host's /etc/resolv.conf entries into the domUs.
Good luck. :)
Hi Jan,
ReplyDeleteNot really understand your statement...
This setup is meant for those who uses NetworkManager and not the Traditional Networking (ifup/ifdown). Yes, if you disable the SuSEFirewall, the internal bridge will not be able to route the domU network packets out to the world.
If you were to use the default and supported Traditional networking (ifup/ifdown) with the default and supported Xen Bridged networking instead of NAT, it will work irregardless if SuSEFirewall is up or not. Having said that, call me paranoid but it always makes me nervous to not enable Firewall.
Hi Kam,
ReplyDeleteI did a mess. You are right, this setup is right for those who uses NetworkManager.
I personally don't use the SuSEfirewall as much as you, I prefer dedicated firewall.
Anyway, eventually I managed to create a similar configuration on SLE11 where netconfig is used and no firewall is enable (actually a mansquarade rule needs to be set from console). All is done by a dummy interface which seems to be not good for NetworkManager (I'll work it out).
Bridged can be well managed instead, and with the iptables rule rather than SuSEFirewall described here the job is done.
Hi Jan,
ReplyDeleteGlad to hear it worked for you. :)
I will be posting another blog entry later this week (hopefully) on this topic based on my recent encounter with a customer who uses NIC bonding with Xen.
Cheers!
Han
Hi Kam,
ReplyDeleteI had a question regarding this procedure for OpenSuSE 11.1
It seems to work for creating a private network, I can get the two domU's to communicate with each other just fine. However, they cannot seem to route to the outside following these instructions. Do you have an suggestions on what might be different in OpenSuSE 11.1, or what diagnostics I might run to help trouble shoot this problem? Thanks,
kfox
Hi kfox,
ReplyDeleteI've not done any Xen work on openSUSE... however, I find it very surprising if it would not work since both the openSUSE and SUSE Linux Enterprise are similar.
Since both domUs are up and can ping one another and your challenge is getting to the outside world (ie www) from domU, have you look at my comment about sync-ing /etc/resolv.conf on dom0 (on this same page) posted on 18 April 2009? See if that will work out for you.
Han (Kam is my last name) :)