Randomly can't connect to guest vm in libvirt

I cannot reliably trigger this, although if I spin up many vms at a time and then attempt to connect to some of them, I run into this condition:

$ ping 192.168.122.135
PING 192.168.122.135 (192.168.122.135) 56(84) bytes of data.
From 192.168.122.1 icmp_seq=1 Destination Host Unreachable
From 192.168.122.1 icmp_seq=2 Destination Host Unreachable
From 192.168.122.1 icmp_seq=3 Destination Host Unreachable

Note that this does not happen for all VMs that I create and start, only a handful of them (randomly).

The vm that has obtained the ip 192.168.122.135 has the following for its network in its domain xml:

<interface type='network'>
  <mac address='52:54:00:3d:72:ab'/>
  <source network='default'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <alias name='net0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>

And the default network is defined as (and yes, 22 vms are currently running):

<network connections='22'>
  <name>default</name>
  <uuid>69674b8b-f067-4513-b594-3e52360f391b</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

The output from ifconfig for vnet0 (referenced by the VM's network domain xml) and virbr0 (used by the default network as shown above):

$ sudo ifconfig vnet0
vnet0     Link encap:Ethernet  HWaddr fe:54:00:3d:72:ab
          inet6 addr: fe80::fc54:ff:fe3d:72ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:425 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1304 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:57503 (57.5 KB)  TX bytes:67257 (67.2 KB)

and

$ sudo ifconfig virbr0
virbr0    Link encap:Ethernet  HWaddr fe:54:00:08:e9:a4
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:882508 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2527165 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:93980992 (93.9 MB)  TX bytes:3047773583 (3.0 GB)

Below is the partial output from ip route list:

192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1

The route output above makes me think that it should be working. BUT ITS NOT. and it only fails sometimes, and works most of the time.

Why can't I connect to the guest (192.168.122.135) from the host??

I was originally using filters, but removing the filters from the VM's domain xml has no effect on this condition randomly showing up. If I spin up many VMs at the same time I can get it to happen to a lot of them. Some of the VMs work just fine though and allow me to connect.

Also, I am using ubuntu 14.04.3:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:        14.04
Codename:       trusty

With kernel 3.19.0-30-generic.

More info - virsh version:

$ virsh --version
1.2.2

libvirtd version:

$ libvirtd --version
libvirtd (libvirt) 1.2.2

Answers


I don't have enough reputation to comment... But I have a few suggestions on things you could try to further explore the problem.

Question: Does assigning an IP address in the 192.168.122.X subnet on vnet0 do anything? The route that is configured seems to suggest that your traffic will go to virbr0 since it has the 192.168.122.1 IP address. If you can't ping any other devices in that subnet, then I suspect that's the issue.

If that doesn't get you anywhere...

  • Packet trace on host / VM

    Try doing a packet dump on virbr0 and on the internal VM interface when this occurs. Ping the VM, and see what kind of traffic you see.

    sudo tcpdump -n -i virbr0 -v "icmp or arp"

    Depending on what you see there, will help narrow down the source of the problem. If you're not even getting your pings on that interface, then it's a routing issue on the host. If pings are going in, but the VM isn't seeing them, then it's a network/routing issue with the libvirt network.

    I recommend also doing the above with a working VM, so you have a reference to compare the traffic against.

  • Check ARP Cache

    Check your ARP cache on the host when this occurs. Does the mac address exist in the cache? Maybe it's getting mangled...

    To dump the arp cache:

    # arp

  • Check your libvirt logs

    If configured, libvirt will log to syslog using the 'libvirtd' tag. Check your configuration to be sure this is enabled. It seems unlikely it's a libvirt issue, but it wouldn't hurt to turn on the logging.

    To enable this setting

    # vi /etc/libvirt/libvirtd.conf

    Add the line

    log_outputs_"1:syslog:libvirtd"

    Restart libvirt

    # service libvirt-bin restart


I have the same network setting, and similar problem in a CentOS 7 host. Eventually, it turned out that the problem was guest VM's firewall setting blocked echo request and other external connection. After changing the firewall setting, the problem is solved.


I had similar issue. I just tried following command to check whether machine is installed properly or not.

lsmod | grep kvm

If it is showing kvm details then machine is installed properly.

After that to restart the services

service libvirtd restart

Also check gateway using the below command

netstat -rn

Need Your Help

AJAX not passing POST vars in IE7/IE8

javascript ajax internet-explorer post

Please Note: answers along the lines of 'use jquery' or 'use (insert wellknown framework)' is not helpful. Frameworks such as jquery includes alot of extra code which is not nessary at all for what...

libcurl multi interface single easy interface multiple requests

c++ http curl httpclient libcurl

I am looking to use libcurl for asynchronous HTTP requests. For that I am using multi interface provided by libcurl. My application will be having many requests coming periodically for which I am l...