NFD

50 node simulation valgrind summary:

==9587== LEAK SUMMARY:
==9587==    definitely lost: 0 bytes in 0 blocks
==9587==    indirectly lost: 0 bytes in 0 blocks
==9587==      possibly lost: 2,263,514 bytes in 67,928 blocks
==9587==    still reachable: 1,474,943,776 bytes in 3,910,237 blocks
==9587==         suppressed: 0 bytes in 0 blocks
==9587==
==9587== For counts of detected and suppressed errors, rerun with: -v
==9587== ERROR SUMMARY: 37 errors from 37 contexts (suppressed: 0 from 0)

25 node simulation valgrind summary:

==9287== LEAK SUMMARY:
==9287==    definitely lost: 0 bytes in 0 blocks
==9287==    indirectly lost: 0 bytes in 0 blocks
==9287==      possibly lost: 400,259 bytes in 11,100 blocks
==9287==    still reachable: 437,147,930 bytes in 1,132,024 blocks
==9287==         suppressed: 0 bytes in 0 blocks
==9287==
==9287== For counts of detected and suppressed errors, rerun with: -v
==9287== ERROR SUMMARY: 31 errors from 31 contexts (suppressed: 0 from 0)

This issue also present in most up-to-date NFD code. It is not yet reported perhaps because in a standalone instance of NFD this is not a concern. Since I am running simulation with 25, 50 and 100+ nodes, each having its own NFD instance, the growth is very quick.

Vince and I debugged this issue little further and below are our findings:

• The issue is reproducible also on standalone NFD. Vince tried about 100 registration requests and there is a consistent increase in memory. This increase is present even if RibManager::sendSuccessResponse is not called.

• The memory grows even if we bypass the RibManager completely by using "nfdc add-nexthop" and this problem is present in the latest code of NFD since Vince tested this with the most up-to-date version.

• Another possibility we thought about was response messages getting cached in CS leading to increase in memory consumption by NFD. To rule this out, we set the CS size to 1 by calling 'ndnHelper.setCsSize(1);' before installing NDN L3 Stack on nodes, but yet we see memory growth.

• I also checked that default CS size is 100 packets. So even with this size, CS should not grow beyond 100 packets. So we do not think CS is causing this growth.

   42 StackHelper::StackHelper()
   43   : m_needSetDefaultRoutes(false)
   44   , m_maxCsSize(100)
   45   , m_isRibManagerDisabled(false)
   46   , m_isFaceManagerDisabled(false)
   47   , m_isStatusServerDisabled(false)
   48   , m_isStrategyChoiceManagerDisabled(false)
  

• It seems to be some internal pipeline issue, because when we performed either 1000 add-nexthop commands or 1000 registration commands for the same prefix, the memory increase was observed.

I investigate the stack trace from Valgrind report.

=================================
./NFD/rib/rib-manager.cpp

188   m_keyChain.sign(*responseData);
189   m_face.put(*responseData);

./NFD/daemon/mgmt/manager-base.cpp

98   m_keyChain.sign(*responseData);
99   m_face->put(*responseData);

Here each time it is allocating ~9K block of memory. I am not sure when this is released, but this is the topmost contributor for memory build up.
./ndn-cxx/src/encoding/encoder.cpp

27 Encoder::Encoder(size_t totalReserve/* = 8800*/, size_t reserveFromBack/* = 400*/)
28   : m_buffer(new Buffer(totalReserve))
29 {
30   m_begin = m_end = m_buffer->end() - (reserveFromBack < totalReserve ? reserveFromBack : 0);
31 }

This is another place in the encoder code, a new memory allocation takes place. This operates on the 9 KByte m_buffer allocated earlier.

83     Buffer* buf = new Buffer(size);
84     std::copy_backward(m_buffer->begin(), m_buffer->end(), buf->end());
85 

=================================
The other possibility is that dead-nonce-list is not getting cleared after the loop detection duration. Or perhaps the ndn::Block() is not released after the ndn::Name wireEncode() is done for the first time. This is the second most reason for memory build up.
Ref: https://github.com/named-data/NFD/blob/149e02cf7da50ac526f7d850c2ecd9570e943f9b/daemon/table/dead-nonce-list.cpp#L108

./NFD/daemon/table/dead-nonce-list.cpp

105 DeadNonceList::Entry
106 DeadNonceList::makeEntry(const Name& name, uint32_t nonce)
107 {
108   Block nameWire = name.wireEncode();

./ndn-cxx/src/encoding/block.cpp

344       m_subBlocks.push_back(Block(m_buffer,
345                                   type,
346                                   element_begin, element_end,
347                                   begin, element_end));
348 

If this is the problem, why is there still a memory increase when the same route is registered repeatedly? If the Name and route are not new, the Name from the request is not inserted into the RIB.

Few comments. Name::wireEncode would work only in #3495, as there is a call for Name::getFullName that reset the cached version of the wire for the name.

Allocation of 8800 block doesn't mean that the underlying buffer of all packets is 8800 bytes. We don't have (yet) the zero copy implementation and the buffer is being copied into the new buffer that has size of the actual interest or data packet.

However, this doesn't mean there is no problem. I agree that it is unintentional excessive memory use because getName() in RibManager, FibManager and other places does not "compact" the memory.

I have two alternative proposals:

• ::compact() method for Name to compress memory representation for this structure (as well as all other data structures)
• ::copy() method for Name that returns a compacted/copied data structure (can also work for other data structures)

Junxiao, I agree that 'nameWire' is not leaking memory. The subsequent part of the stack trace does the allocation when 'wireEncode()' is called for the first time. Alex's reference to #3495 confirms the same (i.e. Right now, you can simply call Name::wireEncode() which will make a separate memory allocation, releasing the previously referenced buffer). So question is: when does NFD releases this allocation? How long does it need to hold it?

Also, right now, I have configured only two name prefixes per node. So at any point in simulation there are not going to be '2 x N' prefixes in the network (where 'N' is # of nodes in topology). So I did not get what you are trying to say in #4.4. Once all the name prefixes are registered it should stabilize, right?

Analysis for the top most memory consumer. I am going through other other stack traces using same technique I used here.

src/ndnSIM/ndn-cxx/src/security/key-chain.cpp

697 void
698 KeyChain::signPacketWrapper(Data& data, const Signature& signature,
699                             const Name& keyName, DigestAlgorithm digestAlgorithm)
700 {
701   data.setSignature(signature);
702 
703   EncodingBuffer encoder;
704   std::cout << "1 Reference count of buffer: " << encoder.getBuffer().use_count() << std::endl;
705 
706   data.wireEncode(encoder, true);
707   std::cout << "2 Reference count of buffer: " << encoder.getBuffer().use_count() << std::endl;
708 
709   Block sigValue = pureSign(encoder.buf(), encoder.size(), keyName, digestAlgorithm);
710   std::cout << "3 Reference count of buffer: " << encoder.getBuffer().use_count() << std::endl;
711 
712   data.wireEncode(encoder, sigValue);
713   std::cout << "4 Reference count of buffer: " << encoder.getBuffer().use_count() << std::endl;
714 }

I am not sure if Alex mentioned this when he talked about compacting the buffer (in comment #6). But I have made some observations. EncodingBuffer, which derives Encoder allocates the 'share_ptr<Buffer> m_Buffer'. In above code, instance 'encoder' is a local so it should get deleted automatically; however, I see that there is no destructor defined for EncodingBuffer neither it is defined for Buffer class. IMO, we need to clear the vector and shrink_to_fit it? Correct?

Also, I added some debugs (see 'cout' in above code) and to print the reference count of share_ptr m_Buffer and I got this.

1 Reference count of buffer: 2
2 Reference count of buffer: 2
3 Reference count of buffer: 2
4 Reference count of buffer: 29

This indicates that by the time signPacketWrapper returns, the memory held by m_Buffer is not being released. IMO ideally this count should be 0. Since the reference count is non-zero, the memory is not getting deallocated. And even if this count becomes zero, I guess the resource will still not be cleaned since there is no destructor defined. Correct?

~/sandbox/ndnSIM/ns-3$ grep "4 Reference count of buffer: " 25node_traces/25-node-nlsr-simulation-log.txt  | wc -l
36941

For a 25 node topology test, there are close to 37K m_Buffer objects getting created and still being held. As per the above debugs, each of these 37K objects has reference count of 29 and above.

==== Adding on behalf of Alex. ====

Anil, I've pushed to the code review system a few patches that should reduce some memory use

library http://gerrit.named-data.net/#/c/2863/
NFD http://gerrit.named-data.net/#/c/2864/

Can you try to rerun some of your analysis with those patches? Just to see if we see smaller memory footprint or it says the same for some other reason.

--
Alex

Can someone please share the steps to measure memory consumption by RIB?
I will first revert the patch to get the reference and then test it again after the patch.

--

The patches quoted in note-11 is only about RIB. It does not solve #3619, and the observation in note-12 is mixing two issues.

Reply to note-14:

I do not know the steps. In general:

The test should observe memory allocations, and record an allocation only if a function in nfd::rib namespace is on the stack.
This shall observe whether note-11 patch fixes #3618, without being affected by #3619.

Below are the valgrind run AFTER Alex's fix. (Check below update #17) for readings BEFORE the fix).

Observations:

1. The size of "possibly lost memory" has INCREASED in this run compared to the run WITHOUT this fix.
2. The size of "still rechable" memory has DECREASED compared to the run WITHOUT the fix.

This indicates that while amount of memory held by process gone down, the potential leaks have increased.

1. The amount of memory used by "nfd::rib" is fairly consistent with or without Alex's fix. I am pasting one of the stack traces involving the RibManager.

==2770== 57,965,600 bytes in 6,587 blocks are possibly lost in loss record 1,218 of 1,218
==2770==    at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2770==    by 0x69A1353: allocate (new_allocator.h:104)
==2770==    by 0x69A1353: _M_allocate (stl_vector.h:168)
==2770==    by 0x69A1353: _M_create_storage (stl_vector.h:181)
==2770==    by 0x69A1353: _Vector_base (stl_vector.h:136)
==2770==    by 0x69A1353: vector (stl_vector.h:283)
==2770==    by 0x69A1353: ndn::Buffer::Buffer(unsigned long) (buffer.cpp:43)
==2770==    by 0x69A1F0A: ndn::encoding::Encoder::Encoder(unsigned long, unsigned long) (encoder.cpp:28)
==2770==    by 0x6A41E57: EncodingImpl (encoding-buffer.hpp:42)
==2770==    by 0x6A41E57: ndn::security::KeyChain::signPacketWrapper(ndn::Data&, ndn::Signature const&, ndn::Name const&, ndn::DigestAlgorithm) (key-chain.cpp:703)
==2770==    by 0x6A49755: void ndn::security::KeyChain::signImpl<ndn::Data>(ndn::Data&, ndn::security::SigningInfo const&) (key-chain.hpp:892)
==2770==    by 0x6B6384B: nfd::rib::RibManager::sendResponse(ndn::Name const&, ndn::mgmt::ControlResponse const&) (rib-manager.cpp:188)
==2770==    by 0x6B6589B: nfd::rib::RibManager::sendSuccessResponse(std::shared_ptr<ndn::Interest const> const&, ndn::nfd::ControlParameters const&) (rib-manager.cpp:505)
==2770==    by 0x6B664B0: nfd::rib::RibManager::registerEntry(std::shared_ptr<ndn::Interest const> const&, ndn::nfd::ControlParameters&) (rib-manager.cpp:287)
==2770==    by 0x6B63B26: operator() (functional:2471)
==2770==    by 0x6B63B26: nfd::rib::RibManager::onCommandValidated(std::shared_ptr<ndn::Interest const> const&) (rib-manager.cpp:254)
==2770==    by 0x6A98DDE: ndn::ValidatorConfig::checkPolicy(ndn::Interest const&, int, std::function<void (std::shared_ptr<ndn::Interest const> const&)> const&, std::function<void (std::shared_ptr<ndn::Interest const> const&, std::string const&)> const&, std::vector<std::shared_ptr<ndn::ValidationRequest>, std::allocator<std::shared_ptr<ndn::ValidationRequest> > >&) (validator-config.cpp:530)
==2770==    by 0x6AAE02A: ndn::Validator::validate(ndn::Interest const&, std::function<void (std::shared_ptr<ndn::Interest const> const&)> const&, std::function<void (std::shared_ptr<ndn::Interest const> const&, std::string const&)> const&, int) (validator.cpp:54)
==2770==    by 0x6B63272: validate (validator.hpp:101)
==2770==    by 0x6B63272: nfd::rib::RibManager::onLocalhostRequest(ndn::Interest const&) (rib-manager.cpp:216)
==2770==
==2770== LEAK SUMMARY:
==2770==    definitely lost: 0 bytes in 0 blocks
==2770==    indirectly lost: 0 bytes in 0 blocks
==2770==      possibly lost: 452,854,398 bytes in 1,688,776 blocks
==2770==    still reachable: 59,138 bytes in 152 blocks
==2770==         suppressed: 0 bytes in 0 blocks
==2770==
==2770== For counts of detected and suppressed errors, rerun with: -v

Below are the valgrind readings BEFORE Alex's fix.

==3531== 64,512,800 bytes in 7,331 blocks are still reachable in loss record 1,187 of 1,187
==3531==    at 0x4C2B0E0: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3531==    by 0x69A1323: allocate (new_allocator.h:104)
==3531==    by 0x69A1323: _M_allocate (stl_vector.h:168)
==3531==    by 0x69A1323: _M_create_storage (stl_vector.h:181)
==3531==    by 0x69A1323: _Vector_base (stl_vector.h:136)
==3531==    by 0x69A1323: vector (stl_vector.h:283)
==3531==    by 0x69A1323: ndn::Buffer::Buffer(unsigned long) (buffer.cpp:43)
==3531==    by 0x69A1EDA: ndn::encoding::Encoder::Encoder(unsigned long, unsigned long) (encoder.cpp:28)
==3531==    by 0x6A41E27: EncodingImpl (encoding-buffer.hpp:42)
==3531==    by 0x6A41E27: ndn::security::KeyChain::signPacketWrapper(ndn::Data&, ndn::Signature const&, ndn::Name const&, ndn::DigestAlgorithm) (key-chain.cpp:703)
==3531==    by 0x6A49725: void ndn::security::KeyChain::signImpl<ndn::Data>(ndn::Data&, ndn::security::SigningInfo const&) (key-chain.hpp:892)
==3531==    by 0x6B6365B: nfd::rib::RibManager::sendResponse(ndn::Name const&, ndn::mgmt::ControlResponse const&) (rib-manager.cpp:188)
==3531==    by 0x6B656AB: nfd::rib::RibManager::sendSuccessResponse(std::shared_ptr<ndn::Interest const> const&, ndn::nfd::ControlParameters const&) (rib-manager.cpp:505)
==3531==    by 0x6B662C0: nfd::rib::RibManager::registerEntry(std::shared_ptr<ndn::Interest const> const&, ndn::nfd::ControlParameters&) (rib-manager.cpp:287)
==3531==    by 0x6B63936: operator() (functional:2471)
==3531==    by 0x6B63936: nfd::rib::RibManager::onCommandValidated(std::shared_ptr<ndn::Interest const> const&) (rib-manager.cpp:254)
==3531==    by 0x6A98DAE: ndn::ValidatorConfig::checkPolicy(ndn::Interest const&, int, std::function<void (std::shared_ptr<ndn::Interest const> const&)> const&, std::function<void (std::shared_ptr<ndn::Interest const> const&, std::string const&)> const&, std::vector<std::shared_ptr<ndn::ValidationRequest>, std::allocator<std::shared_ptr<ndn::ValidationRequest> > >&) (validator-config.cpp:530)
==3531==    by 0x6AADFFA: ndn::Validator::validate(ndn::Interest const&, std::function<void (std::shared_ptr<ndn::Interest const> const&)> const&, std::function<void (std::shared_ptr<ndn::Interest const> const&, std::string const&)> const&, int) (validator.cpp:54)
==3531==    by 0x6B63082: validate (validator.hpp:101)
==3531==    by 0x6B63082: nfd::rib::RibManager::onLocalhostRequest(ndn::Interest const&) (rib-manager.cpp:216)
==3531==
==3531== LEAK SUMMARY:
==3531==    definitely lost: 0 bytes in 0 blocks
==3531==    indirectly lost: 0 bytes in 0 blocks
==3531==      possibly lost: 2,396,214 bytes in 72,078 blocks
==3531==    still reachable: 479,916,787 bytes in 1,680,083 blocks
==3531==         suppressed: 0 bytes in 0 blocks
==3531==
==3531== For counts of detected and suppressed errors, rerun with: -v
==3531== ERROR SUMMARY: 37 errors from 37 contexts (suppressed: 0 from 0)

@Junxiao, how do I bring in the fixes for ndn-cxx:cfaa1addf4592d44e119604abcec06c73ef113f2 and NFD:commit:9627e88d4b9a3e30090ad4d5240d58525ba0126f onto ndnSIM? I think without these two fixes, the leak/boating in ndnSIM will not go off. If I just patch in these two diffs on ndnSIM branch, will it help? I saw your readmine updates late so could not verify these so far. I will do it now.

/anil.

ndnSIM uses a slightly modified versions of ndn-cxx and NFD:

On Github (for public access):

• https://github.com/named-data-ndnSIM/ndn-cxx
• https://github.com/named-data-ndnSIM/NFD

On Gerrit (for submitting patches):

• https://gerrit.named-data.net/#/admin/projects/ndnSIM/ndn-cxx
• https://gerrit.named-data.net/#/admin/projects/ndnSIM/NFD

@Alex @Junxiao, which NFD branch the commit '9627e88d4b9a3e3009' is made into? I searched it in mainline NFD but did not find it. Am I missing something? (I could locate the ndn-cxx patch/diff)
@Alex, I need to try these first on my NFD clone under ndnSIM.

ubuntu:~/workspace/NFD$ git log > ~/nfd_log.txt
ubuntu:~/workspace/NFD$ cat ~/nfd_log.txt 

snip...
/snip

commit 336e4f71cf4e6e6d02329a69088eef4f21088e35
Author: Junxiao Shi <git@mail1.yoursunny.com>
Date:   Wed Jan 22 19:38:31 2014 -0700

doc: add pkg-config as a prerequisite

Change-Id: I820491e6441e0d459834a3f47a3ca5b5516cbc04

commit 2aa396272b9a6892b8c29f76efb66e81b443949a
Author: Alexander Afanasyev <alexander.afanasyev@ucla.edu>
Date:   Wed Jan 22 11:51:11 2014 -0800

Adding build system, README, COPYING, basic INSTALL.md, and skeleton for unit tests

Change-Id: I00a58106e43f6eaaec6eedf9fa7d217a22c19d2b

commit a14170028c5f6611add84362f18b476031590fc0
Author: Alexander Afanasyev <alexander.afanasyev@ucla.edu>
Date:   Tue Jan 21 20:29:55 2014 -0800

Initial commit
ubuntu:~/workspace/NFD$ 
ubuntu:~/workspace/NFD$ cat ~/nfd_log.txt | grep 9627e88d4b9a
ubuntu:~/workspace/NFD$ 

which NFD branch the commit '9627e88d4b9a3e3009' is made into?

This commit can be found at https://gerrit.named-data.net/#/c/3152/2 and it's not merged into any branch.

It's worth noting that this commit does not contain a fix for RIB memory growth issue, and applying it onto an older version of NFD will not fix the issue.
Any one or more commits since the older version of NFD used by ndnSIM to the current NFD may contain the fix, and all of them should be applied. This would take significant effort, and it's still ongoing in #3560.

Benchmark of https://gerrit.named-data.net/2864 patchset3:

vagrant@m0212:~/NFD$ /usr/bin/time -v build/pit-fib-benchmark 
Running 1 test case...

*** No errors detected
    Command being timed: "build/pit-fib-benchmark"
    User time (seconds): 26.22
    System time (seconds): 2.31
    Percent of CPU this job got: 99%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:28.67
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 3418672
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 84
    Minor (reclaiming a frame) page faults: 845481
    Voluntary context switches: 130
    Involuntary context switches: 216
    Swaps: 0
    File system inputs: 30096
    File system outputs: 8
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Comparing to master branch nfd:commit:2a0019431687ce44b060cc441f01bf7e24945c34 :

vagrant@m0212:~/NFD$ /usr/bin/time -v build/pit-fib-benchmark 
Running 1 test case...

*** No errors detected
    Command being timed: "build/pit-fib-benchmark"
    User time (seconds): 21.95
    System time (seconds): 2.42
    Percent of CPU this job got: 99%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.60
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 3366316
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 28
    Minor (reclaiming a frame) page faults: 836082
    Voluntary context switches: 52
    Involuntary context switches: 428
    Swaps: 0
    File system inputs: 4968
    File system outputs: 8
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

This test is almost irrelevant to the patch. The primary concern in the patch is about FIB/RIB managers (minus change in name tree that may be useless/wrong). The tests doesn't exercise any of the registration commands.