NFD

Please attach pcap captures from send machine and receive machine.

Please run sender in GDB, and obtain a stacktrace after it crashes.

pcap traces show that all 4016 fragments are successfully sent, and all of them are received.

One possibility is that NFD EthernetFace cannot process fragments fast enough, so that some packets are dropped by the kernel.

Junxiao Shi wrote:

One possibility is that NFD EthernetFace cannot process fragments fast enough, so that some packets are dropped by the kernel.

Yep, I had the same thought. There's limited buffer space in the TPACKET implementation. If the application is unable to read the fragments fast enough, eventually the buffer will fill up and the kernel will start dropping incoming packets.

There's limited buffer space in the TPACKET implementation. If the application is unable to read the fragments fast enough, eventually the buffer will fill up and the kernel will start dropping incoming packets.

Is there a way to confirm whether this situation is happening, and to what extent?

Also, is the sender crash caused by writing packets too fast and filling up some buffer?

Junxiao Shi wrote:

Is there a way to confirm whether this situation is happening, and to what extent?

Yes, pcap_stats(3).

Also, is the sender crash caused by writing packets too fast and filling up some buffer?

I doubt it. In fact the stack trace shows that the crash is happening during a handleRead call... there are some too low values for this in the topmost 3 frames, suggesting a null pointer somewhere.

susmit, what does NFD print before crashing? Please attach a full log file of a crashing scenario.

You can also try increasing the ring buffer size (the libpcap default is 2M on Linux):

pcap_set_buffer_size(m_pcap.get(), 10*1024*1024); // e.g. 10 MB

Note that this call must happen before pcap_activate or it will fail.

Apply this patch http://gerrit.named-data.net/1692 and see if it reports any packet drop.

Davide Pesavento wrote:

Apply this patch http://gerrit.named-data.net/1692 and see if it reports any packet drop.

Make sure you build NFD in debug mode (./waf configure --debug)

This doesn't make any sense. The stack trace shows that the handleRead callback is invoked, and execution arrives at least as far as the async_read_some call before crashing. Thus there are two possibilities:

1. We received something (a frame) and processed it in processIncomingPacket
2. pcap_next_ex returned 0 (read timeout)

In both cases one or more log messages should have been printed, but I see nothing in the log file, so... what the hell?

Another thing:

1422555794.215244 INFO: [FaceTable] Removed face id=258 remote=ether://[01:00:5e:00:17:aa] local=dev://eno4

what happened here?

Davide Pesavento wrote:

1422555794.215244 INFO: [FaceTable] Removed face id=258 remote=ether://[01:00:5e:00:17:aa] local=dev://eno4

what happened here?

ah, it turns out that the failure reason passed to Face::fail is never logged anywhere. I'll submit a patch shortly.

Apply this patch http://gerrit.named-data.net/1692 and see if it reports any packet drop.

Make sure you build NFD in debug mode (./waf configure --debug)

Davide, something is printing the packets when I compile with --debug. I see a bunch of binary output on the terminal (Mostly As).

Well there's not much we can do then... NFD is single-threaded. Try increasing the buffer size as I said in note-9, and see at what point the packet loss stops (or is significantly reduced).

The loss is not reduced even after increasing the said buffer.

What size did you set it to? If you're sending two 5 MB packets, I expect you won't see any drops with a 20 MB buffer.

How many fragments (ethernet frames) are being sent per second?

Is the crash still reproducible with the latest NFD?

Davide Pesavento wrote:

the number of slots in the TPACKET ring buffer is way too small, which can cause frame drops if NFD is unable to handle incoming frames quickly enough.

This situation can be improved and packet loss alleviated by reducing the snaplen and/or increasing the buffer size.