Bug #1877
closed
EthernetFace creation fails on OS X with Boost 1.56.0
100%
Description
NFD built from the master on OS X 10.9.4 with Boost 1.56.0 dies when it tries to create an ethernet face.
The following assign() call fails:
https://github.com/named-data/NFD/blob/master/daemon/face/ethernet-face.cpp#L72
Note:
- This error has been duplicated on two similarly configured machines.
- Rolling back to Boost 1.55.0 eliminates the problem.
STEPS TO REPRODUCE
Install Boost 1.56.0 using Macports
Obtain, build and install ndn-cxx and nfd (configure --with-tests) from the master.
Run the following unit test (or try to run NFD).
$ ./build/unit-tests-daemon -t FaceEthernet
Running 3 test cases...
1408301259.014077 INFO: [EthernetFace] Creating ethernet face on en0: d0:e1:40:96:67:e2 <--> ff:ff:ff:ff:ff:ff
unknown location:0: fatal error in "MulticastFacesMap": std::runtime_error: assign: Invalid argument
../tests/daemon/face/ethernet.cpp:41: last checkpoint
1408301259.016865 INFO: [EthernetFace] Creating ethernet face on en0: d0:e1:40:96:67:e2 <--> ff:ff:ff:ff:ff:ff
unknown location:0: fatal error in "SendPacket": std::runtime_error: assign: Invalid argument
../tests/daemon/face/ethernet.cpp:41: last checkpoint
*** 2 failures detected in test suite "Master Test Suite"
Updated by Alex Afanasyev over 9 years ago
- Category set to Faces
- Status changed from New to In Progress
- Assignee set to Alex Afanasyev
- Target version set to v0.2
I confirm this. Here is gdb trace:
Catchpoint 1 (exception caught), throw location throw_exception.hpp:69, catch location unknown, exception type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >
0x00007fff8f642bec in __cxa_throw ()
(gdb) bt
#0 0x00007fff8f642bec in __cxa_throw ()
#1 0x000000010000c127 in boost::throw_exception<boost::system::system_error> (e=@0x7fff5fbfc0b8) at throw_exception.hpp:69
#2 0x000000010000c044 in boost::asio::detail::do_throw_error (err=@0x7fff5fbfc150, location=0x1005dd20b "assign") at throw_error.ipp:38
#3 0x000000010000bfc2 in boost::asio::detail::throw_error (err=@0x7fff5fbfc150, location=0x1005dd20b "assign") at throw_error.hpp:42
#4 0x00000001001e582b in boost::asio::posix::basic_descriptor<boost::asio::posix::stream_descriptor_service>::assign (this=0x103112290, native_descriptor=@0x7fff5fbfc3b8) at basic_descriptor.hpp:171
#5 0x00000001001e0ef3 in EthernetFace (this=0x10310d410, socket=@0x7fff5fbfc930, interface=@0x103110310, address=@0x7fff5fbfcfb8) at ../daemon/face/ethernet-face.cpp:72
#6 0x00000001001e086d in EthernetFace (this=0x10310d410, socket=@0x7fff5fbfc930, interface=@0x103110310, address=@0x7fff5fbfcfb8) at ../daemon/face/ethernet-face.cpp:90
#7 0x00000001001e999f in boost::make_shared<nfd::EthernetFace, boost::shared_ptr<boost::asio::posix::basic_stream_descriptor<boost::asio::posix::stream_descriptor_service> >, boost::shared_ptr<nfd::NetworkInterfaceInfo>, nfd::ethernet::Address> (a1=@0x7fff5fbfc930, a2=@0x103110310, a3=@0x7fff5fbfcfb8) at make_shared_object.hpp:880
#8 0x00000001001e813b in nfd::EthernetFactory::createMulticastFace (this=0x103112200, interface=@0x103110310, address=@0x7fff5fbfcfb8) at ../daemon/face/ethernet-factory.cpp:53
#9 0x000000010013404e in nfd::FaceManager::processSectionEther (this=0x103206590, configSection=@0x103110338, isDryRun=false, nicList=@0x7fff5fbfdb58) at ../daemon/mgmt/face-manager.cpp:706
#10 0x000000010012a6da in nfd::FaceManager::onConfig (this=0x103206590, configSection=@0x10310f2d8, isDryRun=false, filename=@0x7fff5fbff920) at ../daemon/mgmt/face-manager.cpp:198
#11 0x000000010015854e in boost::_mfi::mf3<void, nfd::FaceManager, boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>::operator() (this=0x7fff5fbfdfe0, p=0x103206590, a1=@0x10310f2d8, a2=false, a3=@0x7fff5fbff920) at mem_fn_template.hpp:393
#12 0x000000010015847b in boost::_bi::list4<boost::_bi::value<nfd::FaceManager*>, boost::arg<1>, boost::arg<2>, boost::arg<3> >::operator()<boost::_mfi::mf3<void, nfd::FaceManager, boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::_bi::list3<boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&> > (this=0x7fff5fbfdff0, f=@0x7fff5fbfdfe0, a=@0x7fff5fbfde38) at bind.hpp:457
#13 0x00000001001583c2 in boost::_bi::bind_t<void, boost::_mfi::mf3<void, nfd::FaceManager, boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::_bi::list4<boost::_bi::value<nfd::FaceManager*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >::operator()<boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const> (this=0x7fff5fbfdfe0, a1=@0x10310f2d8, a2=@0x7fff5fbfde9f, a3=@0x7fff5fbff920) at bind_template.hpp:116
#14 0x00000001001581a8 in boost::detail::function::void_function_obj_invoker3<boost::_bi::bind_t<void, boost::_mfi::mf3<void, nfd::FaceManager, boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::_bi::list4<boost::_bi::value<nfd::FaceManager*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, void, boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>::invoke (function_obj_ptr=@0x7fff5fbfdfe0, a0=@0x10310f2d8, a1=false, a2=@0x7fff5fbff920) at function_template.hpp:153
#15 0x00000001002cf397 in boost::function3<void, boost::property_tree::basic_ptree<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>::operator() (this=0x7fff5fbfdfd8, a0=@0x10310f2d8, a1=false, a2=@0x7fff5fbff920) at function_template.hpp:766
#16 0x00000001002cdd02 in nfd::ConfigFile::process (this=0x7fff5fbfef00, isDryRun=false, filename=@0x7fff5fbff920) at config-file.cpp:134
#17 0x00000001002cceed in nfd::ConfigFile::parse (this=0x7fff5fbfef00, input=@0x7fff5fbfe808, isDryRun=false, filename=@0x7fff5fbff920) at config-file.cpp:108
#18 0x00000001002cc67e in nfd::ConfigFile::parse (this=0x7fff5fbfef00, filename=@0x7fff5fbff920, isDryRun=false) at config-file.cpp:80
#19 0x0000000100022b57 in nfd::Nfd::initializeManagement (this=0x7fff5fbff920) at main.cpp:166
#20 0x0000000100009c2b in nfd::Nfd::initialize (this=0x7fff5fbff920) at main.cpp:80
#21 0x0000000100001c92 in main (argc=1, argv=0x7fff5fbffc08) at main.cpp:360
(gdb)
Updated by Alex Afanasyev over 9 years ago
The problem seem to be related to a recent change in kqueue implementation by Boost.Asio. Specifically, this commit https://github.com/boostorg/asio/commit/660e9fe30204e0ada0d1fd40c48015810d2647dc changes kqueue_reactor from setting "events" on demand (e.g., read events only when read operation requested, write when write requested) to always-on. And apparently, the write "event" doesn't exist/supported by the file descriptor returned from libpcap. Actually, lipcap documentation states that the returned file descriptor doesn't work with kqueue at all, so I wonder how did it work before.
Updated by Alex Afanasyev over 9 years ago
One work-around is to completely disable kqueues: #define BOOST_ASIO_DISABLE_KQUEUE 1
Simple solution is to disable globally. To disable only for ethernet, it would require some magic (ethernet-face should be excluded from precompiled headers...)
Updated by Davide Pesavento over 9 years ago
pcap_get_selectable_fd() documentation states that:
[...] poll() and kqueues work on that descriptor in Mac OS X 10.6 and later.
So I'm a little confused... although honestly I don't trust libpcap documentation that much.
Why doesn't asio fall back to poll()/select() if the fd is not usable with kqueue anyway?
Updated by Alex Afanasyev over 9 years ago
There are no detection mechanisms. Specific event processing mechanism is selected by the configuration (or something can be disabled). I'm trying now to figure out how to use select only for ethernet faces, keeping kevents for everything else.
Updated by Davide Pesavento over 9 years ago
Alex Afanasyev wrote:
I'm trying now to figure out how to use select only for ethernet faces, keeping kevents for everything else.
I don't think you can do that while keeping only one event loop.
Updated by Alex Afanasyev over 9 years ago
Oh... you're right... Then I don't see any way around except either disabling Ethernet on OSX with boost 1.56 or switching to select for the same combination of the platform.
Just in case, I have created bugreport in boost trac for that: https://svn.boost.org/trac/boost/ticket/10367
Updated by Alex Afanasyev over 9 years ago
- Status changed from In Progress to Feedback
- Assignee deleted (
Alex Afanasyev)
Updated by Junxiao Shi over 9 years ago
For this version only, we can disable EthernetFace, because it's broken anyway without NDNLP.
In a later version, switch to poll(2) so that EthernetFace can work.
Updated by Davide Pesavento over 9 years ago
It should be easy enough to define BOOST_ASIO_DISABLE_KQUEUE on OS X to force the usage of select(). Optionally, we can keep using kqueue if nfd is configured with --without-libpcap.
Updated by Alex Afanasyev over 9 years ago
Are there consequences of using select instead of kqeue? I saw some post on the Internet that mentioned that there is a limit on number of simultaneous connections (128).
I can confirm that with BOOST_ASIO_DISABLE_KQUEUE code started to work, but I haven't done any tests beyond just starting nfd.
Updated by Davide Pesavento over 9 years ago
select is slower than kqueue (and epoll on linux) with a large number of open connections, because it linearly iterates through all the file descriptors. There's also the problem of the maximum number of file descriptors (FD_SETSIZE, usually 1024), but that's a minor problem because you'll probably hit the per-process limit on the number of open handles anyway.
Updated by Alex Afanasyev over 9 years ago
I checked again and unfortunately, it is not trivial to change to select_reactor. We can, but we should do it in ndn-cxx library too, otherwise we going to get in huge trouble (this mostly related to scheduler, which uses the model selected in the library).
So, as a short-term, I see only disabling the ethernet face if boost is 1.56 on OSX and freebsd.
Updated by Davide Pesavento over 9 years ago
Right, then yes I agree to disable EthernetFace under those conditions.
How do we solve this properly in the longer term though? Using the select-based reactor is not a solution in my opinion, it doesn't scale...
Updated by Junxiao Shi over 9 years ago
- Subject changed from EthernetFace creation fails on Mac OS X with Boost 1.56.0 to EthernetFace creation fails on OS X with Boost 1.56.0
- Assignee set to Davide Pesavento
- Estimated time set to 1.00 h
For v0.2, we should disable EthernetFace on OSX with Boost 1.56 or above.
Updated by Junxiao Shi over 9 years ago
- Has duplicate Bug #1908: nfd-start fails to start daemon (best-route strategy patch refs/changes/66/1166/4) added
Updated by Alex Afanasyev over 9 years ago
- Status changed from Feedback to In Progress
- Assignee changed from Davide Pesavento to Alex Afanasyev
I plan to disable EthernetFace in the FaceManager with macros. Any objections? Alternative is to check for the specific version in waf, but I don't like that too much...
Updated by Junxiao Shi over 9 years ago
wscript should check version and print a warning message if EthernetFace is disabled due to incompatible version.
In the future we can add an option to force enable EthernetFace and causes Asio to use poll(2).
Updated by Davide Pesavento over 9 years ago
Junxiao Shi wrote:
wscriptshould check version and print a warning message if EthernetFace is disabled due to incompatible version.
I agree. Moreover, it's pointless to check for libpcap if the EthernetFace is not going to be compiled anyway.
Updated by Alex Afanasyev over 9 years ago
- Status changed from In Progress to Code review
- % Done changed from 0 to 100
Updated by Alex Afanasyev over 9 years ago
- Related to Task #1922: EthernetFace: workaround with kqueue and Boost 1.56.0 added
Updated by Alex Afanasyev over 9 years ago
- Status changed from Code review to Closed