NFD

One big part of self-learning is to verify Prefix Announcement attached to Data, and update FIB through NFD-RIB. The design is to have a separate thread in forwarding strategy, say "SL-RIB", and the strategy can dispatch the work to it with parameters, so that SL-RIB can verify PA and create command Interest and send it to NFD-RIB to update FIB.

Any comments on the design? The uncertain part to me is how to create a separate thread and how to communicate with NFD-RIB in details.

Remember that strategies are supposed to be stateless. Moreover, there can be multiple instances of the same strategy at any given time. Are you going to create a thread for each instance?

Remind me why the thread doing this work cannot be the RIB thread itself..?

Davide Pesavento wrote:

Remember that strategies are supposed to be stateless. Moreover, there can be multiple instances of the same strategy at any given time. Are you going to create a thread for each instance?

The thread will verify PA and communicate with RIB to update FIB, and each instance needs one thread to handle this, right? Logically, when self-learning strategy wants to deal with PA, it invokes a thread to handle it. No matter the thread updates FIB successfully or fails, the thread can manage itself correctly, such as being terminated correctly. I am not sure the details yet. Does it sound feasible?

Remind me why the thread doing this work cannot be the RIB thread itself..?

One argument is to keep the RIB thread clean and simple, whose job is only to manage FIB. Other parties like NLSR and self-learning should use RIB only to update FIB. For example, NLSR do announcement verification and route calculation itself.

Teng Liang wrote:

The thread will verify PA and communicate with RIB to update FIB, and each instance needs one thread to handle this, right? Logically, when self-learning strategy wants to deal with PA, it invokes a thread to handle it. No matter the thread updates FIB successfully or fails, the thread can manage itself correctly, such as being terminated correctly.

https://twitter.com/davidlohr/status/288786300067270656

I am not sure the details yet. Does it sound feasible?

Of course it's feasible, but you're assuming that using threads is simpler than not using them. And you're also ignoring the performance cost.

Davide Pesavento wrote:

Teng Liang wrote:

The thread will verify PA and communicate with RIB to update FIB, and each instance needs one thread to handle this, right? Logically, when self-learning strategy wants to deal with PA, it invokes a thread to handle it. No matter the thread updates FIB successfully or fails, the thread can manage itself correctly, such as being terminated correctly.

https://twitter.com/davidlohr/status/288786300067270656

I am not sure the details yet. Does it sound feasible?

Of course it's feasible, but you're assuming that using threads is simpler than not using them. And you're also ignoring the performance cost.

I am assuming we have to use threads. It would be great if we can avoid using threads. We don't want PA verification and RIB update to block forwarding, so we want to put these work on a different thread. I don't know if using a thread would have big performance cost or not. The more interesting question is that can we avoid using threads to achieve the goal (to verify PA and update RIB without blocking forwarding)? For the first version, we can ignore PA verification.

As discussed in today's NFD call, the async work can run on the RIB thread using its io-service. Based on some reading, here is what in my mind: in the strategy, the self-learning calls RIB io-service, like this ribIo->post( boost::bind(&SLRib::PAhandler, &SLRib_instance, PA)). In function SLRib::PAhandler, it verifies PA and invokes rib::beginApplyUpdate to update FIB. In NFD/daemon/main.cpp, ribIo is initialized, how can self-learning access it, making it global? SLRib is a class implemented in NFD/daemon/fw/Self-learning-rib.cpp. Any comments?

There are complaints about library bug in 4695,21, but it doesn't seem like so. I have a minimal example that shows Boost.Asio works as expected:

// g++ -std=c++14 -o x x.cpp $(pkg-config --cflags --libs libndn-cxx)

#include <condition_variable>
#include <cstdio>

#include <boost/asio.hpp>
#include <boost/thread.hpp>

static boost::thread_specific_ptr<boost::asio::io_service> g_io;
static boost::asio::io_service* g_ribIo;

boost::asio::io_service&
getGlobalIo()
{
  if (g_io.get() == nullptr) {
    g_io.reset(new boost::asio::io_service());
  }
  return *g_io;
}

int
main()
{
  std::mutex m;
  std::condition_variable cv;

  boost::thread ribThread([&] {
    {
      std::lock_guard<std::mutex> lock(m);
      g_ribIo = &getGlobalIo();
    }
    cv.notify_all();

    boost::asio::io_service::work work(*g_ribIo);
    g_ribIo->run();
  });

  {
    std::unique_lock<std::mutex> lock(m);
    cv.wait(lock, [&] { return g_ribIo != nullptr; });
  }

  g_ribIo->post([&] {
    printf("inside global io = %p\n", &getGlobalIo());
    printf("inside rib io = %p\n", g_ribIo);
  });

  printf("outside global io = %p\n", &getGlobalIo());
  printf("outside rib io = %p\n", g_ribIo);

  return 0;
}

$ ./x
outside global io = 0x1690090
outside rib io = 0x7f6fb80008c0
inside global io = 0x7f6fb80008c0
inside rib io = 0x7f6fb80008c0

The current strategy renews routes on each Data reception. An alternative design is to renew the route on every thousand Data packets reception (or every 5 mins if a Data packet is received). Either way needs to keep a counter or timer on each FIB entry, and the information belongs to strategyInfo. Therefore, this design is to make the FIB entry derived from StrategyInfoHost.

this design is to make the FIB entry derived from StrategyInfoHost.

Unnecessary. Just add a Measurements entry with same name as the FIB entry. ASF strategy is already doing that.

Self-learning code is updated with patch sets 23-27, according to the design doc. The code has been tested on NFDs using ndn-tools.

Regarding unit-test, the unit test for forwarding strategy uses the topology tester, which creates forwarder instances and links if needed. However, self-learning requires rib io_service existing on the rib thread along with the forwarder, which is not supported and is complex to implement. Therefore, Alex and I discussed that we can merge the code without unit-test for now. Nevertheless, the forwarding strategy has been tested through integration test.

A point of reference. If I have a multicast face and unicast face, the strategy will prefer the multicast face because its ID smaller and the second interest is NACKed

Example from logs:

Interest=/sl/test/ping/14835690534392738936?ndn.MaxSuffixComponents=1&ndn.MustBeFresh=true&ndn.Nonce=2728691870 from=265 to=259
1547158826.951094 DEBUG: [nfd.SelfLearningStrategy] broadcast discovery Interest=/sl/test/ping/14835690534392738936?ndn.MaxSuffixComponents=1&ndn.MustBeFresh=true&ndn.Nonce=2728691870 from=265 to=263
1547158826.953448 DEBUG: [nfd.SelfLearningStrategy] Nack for /sl/test/ping/14835690534392738936?ndn.MaxSuffixComponents=1&ndn.MustBeFresh=true&ndn.Nonce=2728691870 from=263: Duplicate

Maybe we can "broadcast" in reverse order of faces? This would prioritize anything that was recently created. Of course, no guarantees and not a long term solution, but potentially preferring unicast versus multicast.

Since the code of self-learning forwarding strategy version 1 is merged, should this issue and related ones be closed?