Project

General

Profile

Actions

Bug #3619

closed

NameTree entry not released

Added by Junxiao Shi almost 8 years ago. Updated almost 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Tables
Target version:
Start date:
05/10/2016
Due date:
% Done:

100%

Estimated time:
3.00 h

Description

It has been observed that the NameTree entry counter on some testbed nodes keeps growing.
The growth of this counter is inconsistent with anticipated level of traffic, and it does not decrease until NFD is restarted.

This indicates that either some table entries are not deleted, or NameTree entries are not released when table entries are deleted.


Files

22f7bf5f-ad59-4b9f-ae23-7be0095014aa.png (14.1 KB) 22f7bf5f-ad59-4b9f-ae23-7be0095014aa.png Junxiao Shi, 05/10/2016 04:29 PM
nameTree-spurs.json (197 KB) nameTree-spurs.json Alex Afanasyev, 05/18/2016 05:45 PM

Related issues 1 (0 open1 closed)

Related to NFD - Bug #3618: Memory bloating in RIBRejectedWeiwei Liu05/09/2016

Actions
Actions #1

Updated by Junxiao Shi almost 8 years ago

  • Related to Bug #3618: Memory bloating in RIB added
Actions #2

Updated by Junxiao Shi almost 8 years ago

Cacti graph from CAIDA node shows this problem.

Actions #3

Updated by John DeHart almost 8 years ago

And for anyone that wants to monitor this on some nodes, here is the link
to the cacti monitoring graphs:

http://ndndemo.arl.wustl.edu/cacti/index.php

You can log in as guest with password ndnTest

Once you are logged in you can click on the tab in the upper right that looks like a waveform.
The dropdown menus at the top then allow you to select specific sets of graphs based
on type or node. For example, if you select "NFD Counters" from the Template menu
you will get the graphs show the NameTree and PIT counters for all nodes.
If you click on one of the graphs it will show you that nodes data over different sample periods.

One interesting thing I notice is that some nodes show the NameTree counter increasing and some don't.

ARIZONA, REMAP, UCI and UCLA appear relatively flat.

CAIDA, CSU, MEMPHIS and WU appear to grow.

I am currently only monitoring 8 nodes. I do plan to add more when I get time.

Actions #4

Updated by Alex Afanasyev almost 8 years ago

Here is some analysis of NameTree that I grabbed on spurs. I made it in json format, so it can be further analyzed, if needed.

It seems that there are a lot of entries that do not have anything associated to them, as we expected.

Actions #5

Updated by Alex Afanasyev almost 8 years ago

In case one wants to play with the debugger, I've created Vagrant environment with all the tools I've used: https://github.com/cawka/nfd-debug-environment

Actions #6

Updated by Junxiao Shi almost 8 years ago

  • Assignee set to Junxiao Shi

note-4 attachment suggests PIT is leaking NameTree entries.

$ grep nFIB nameTree-spurs.json | grep '"nFIB": 0, "nPIT": 0, "nMsmnts": 0, "nSC": 0, "nParent": 1, "nChildren": 0' | sed -e 's/.*"Prefix": "//' -e 's/"},//' | sort
/ndn/edu/ucla/ping/3048590203459139533
/ndn/edu/ucla/ping/3048590203459139541
/ndn/NLSR/sync/recovery/3807018970de2105a1cd0bde747f67b3bdbfe4058591bd964b9c1c6ed5a9a3ba
/ndn/NLSR/sync/recovery/3dc501c008d5dd9718b71a0428eda562f3c4d0032fb81752f1ce60ea8f1062fd
/ndn/nTorrent/NTORRENT/fooM/bar1.txt/%FE%00/%8D%CF%EF%0E%3ER%E1%C5%B2V%8A%89%12%99%9AWj6JE%9B%A0%EF3b%F2%B1%D6%01%FF%25Z
/ndn/nTorrent/NTORRENT/fooM/bar1.txt/%FE%00/%FE%00/%91%A0r%DCY%BF%8Cj%B3%C8f%20%CC%84%B8%02%29%07%1D%5E%06%BA%7Co%C1%A7%8C%84%FA%28%5E%AE
/ndn/nTorrent/NTORRENT/fooT/torrent-file/H8kq%B4%3F%B7%BA%20%03%8E%B5%80%05%5EN%F1GY%8B%D2%C7%B2%F7%3A%DD%0B%1A%E7d%0A%82

(see full output)

All these names appear to be PIT entry names.
And the problem isn't limited to "Interest with full name", because ping Interests usually do not carry implicit digests.

At this moment I cannot determine whether this problem is introduced by #3363, because Cacti does not have logs before its merge.

I'm assigning this to myself since I changed PIT design in #3363.
My plan is to review the code logic of PIT-NameTree attachment.

Actions #7

Updated by Junxiao Shi almost 8 years ago

  • Status changed from New to In Progress
Actions #8

Updated by Junxiao Shi almost 8 years ago

I don't see obvious error in Pit::findOrInsert and Pit::erase implementation.

I try to reproduce this bug in a VM, but nfd-status -v somehow isn't working; I'm investigating.

Actions #9

Updated by Junxiao Shi almost 8 years ago

I think I've found the problem.

std::pair<shared_ptr<pit::Entry>, bool>
Pit::findOrInsert(const Interest& interest, bool allowInsert)
{
  (omit)
  shared_ptr<name_tree::Entry> nte = m_nameTree.lookup(isEndWithDigest ? name.getPrefix(-1) : name);

  (omit)

  if (!allowInsert) {
    return {nullptr, true};
  }

  auto entry = make_shared<pit::Entry>(interest);
  (omit)
  return {entry, true};
}

When this function is invoked with allowInsert=false, and the name tree entry doesn't already exist, return {nullptr, true}; is executed but the name tree entry created during m_nameTree.lookup is leaked.


I'm currently waiting for @Weiwei to complete #3608 or reassign it to me, to avoid conflicts in pit.cpp.
After that, I'll work on a patch.

Actions #10

Updated by Alex Afanasyev almost 8 years ago

Can we have a similar problem from other tables, e.g., Measurements?

Actions #11

Updated by Junxiao Shi almost 8 years ago

Reply to note-10:

There's no evidence indicating so from note-4.

Actions #12

Updated by Junxiao Shi almost 8 years ago

  • Status changed from In Progress to Code review
  • % Done changed from 0 to 100
  • Estimated time set to 3.00 h
Actions #13

Updated by Anil Jangam almost 8 years ago

Which commit of NFD this function (Pit::findOrInsert) was introduced in pit.cpp for the first time? I do not find this function in NFD under ndnSIM. What other code is not merged into NFD in ndnSIM?

https://github.com/named-data-ndnSIM/NFD/blob/bd8eea71137bc382fe1fb8225334b926a5484527/daemon/table/pit.cpp
Actions #14

Updated by Junxiao Shi almost 8 years ago

  • Status changed from Code review to Closed
Actions #15

Updated by Junxiao Shi almost 8 years ago

Reply to note-13:

The fix for PIT leaking NameTree entry is commit:28797798044d0b80c8b883a821645632db9552e5 (it has been merged), as referenced in this Redmine issue.

You may try ndn-cxx ndn-cxx:commit:4f512fb24e8105e2b394ae7528f6236905b50ab4 (it has been merged) and NFD http://gerrit.named-data.net/#/c/2864/2.
This would include fixed for both #3618 and #3619.
We are interested to know whether there's any memory leak elsewhere.

Actions #16

Updated by Anil Jangam almost 8 years ago

Agreed. The Name Tree fix you are talking about is in master NFD branch.

What I am saying that the same is not available in ndnSIM's NFD fork. In fact the "pit.cpp" file is about 12 commits behind the master NFD. How do we go about this? I have already taken ndn-cxx ndn-cxx:4f512fb24e8105e2b394ae7528f6236905b50ab4 (it has been merged) and NFD http://gerrit.named-data.net/#/c/2864/2. Not sure how to take in 28797798044d0b80c8b883a821645632db9552e5.

Actions #17

Updated by Junxiao Shi almost 8 years ago

Reply to note-16:

You'll need to run your scenario without ndnSIM (eg. in Mini-NDN).

Alternatively, wait for the next ndnSIM release that incorporates NFD 0.4.0 or above, then apply all commits after that.

Actions #18

Updated by Anil Jangam almost 8 years ago

I can not use Mini-NDN for my tests.
By when the next ndnSIM release with NFD 0.4.0 will be released?

Actions #19

Updated by Junxiao Shi almost 8 years ago

Reply to note-18:

I have posted this question to ndnSIM mailing list.
http://www.lists.cs.ucla.edu/pipermail/ndnsim/2016-May/002750.html

Please wait for an answer from ndnSIM developers.

Actions

Also available in: Atom PDF