Project

General

Profile

Task #4556

ndncatchunks: stop using Selectors

Added by Junxiao Shi over 1 year ago. Updated 5 months ago.

Status:
Closed
Priority:
High
Start date:
Due date:
% Done:

100%

Estimated time:
3.00 h

Description

ndncatchunks currently uses deprecated ChildSelector, Exclude, MinSuffixComponents, and MaxSuffixComponents in pipelines and version discovery procedures. These uses shall be eliminated.


Related issues

Blocked by NDN Specifications - Feature #4706: RDR packet format specificationClosed

Blocked by ndn-cxx - Feature #4707: RDR reference implementationClosed

Blocked by ndn-tools - Task #4832: ndncatchunks: change version discovery logicClosed

History

#1 Updated by Davide Pesavento about 1 year ago

  • Tracker changed from Feature to Task
  • Subject changed from catchunks: stop using ChildSelector and Exclude to ndncatchunks: stop using ChildSelector and Exclude
  • Priority changed from Normal to High

#2 Updated by Klaus Schneider about 1 year ago

Is there an obvious way to do the version discovery without those selectors?

#3 Updated by Junxiao Shi about 1 year ago

Is there an obvious way to do the version discovery without those selectors?

You may use RDR protocol for version discovery (blocked by #4706 and #4707), or just require the caller to specify a version number.

#4 Updated by Davide Pesavento about 1 year ago

just require the caller to specify a version number

We already have a fixed-version mode for version discovery that doesn't actually discover anything but requires the user to provide the exact version number. But that's not a valid solution. This task is about replacing the existing "iterative" method with something else that still performs a real version discovery, starting from a name without any version number.

#5 Updated by Chavoosh Ghasemi about 1 year ago

  • Assignee set to Chavoosh Ghasemi

#6 Updated by Chavoosh Ghasemi about 1 year ago

This task is about replacing the existing "iterative" method with something else that still performs a real version discovery, starting from a name without any version number.

Apparently, right now we look at the very first 3 components (as both MaxSuffixComponents and MinSuffixComponents are set to 3) of the name prefix and use the right-most child to find the version number.

First of all, please let me know what the problem is with this solution??
Second, what do you mean with "starting from a name without any version number"??

#7 Updated by Davide Pesavento about 1 year ago

Chavoosh Ghasemi wrote:

First of all, please let me know what the problem is with this solution??

MinSuffixComponents, MaxSuffixComponents, ChildSelector are deprecated and no longer exist in packet format v0.3

Second, what do you mean with "starting from a name without any version number"??

Meaning that the user provides a name prefix that doesn't already contain a version number component.

#8 Updated by Chavoosh Ghasemi about 1 year ago

We can set the name of the very first discovery Interest as below:

<prefix>/VDISCOVERY

Upon receiving this Interest by the Producer it will reply by a Data packet that its name includes the version number (probably with no content).
Does this solution make sense to you?

#9 Updated by Davide Pesavento about 1 year ago

Don't reinvent the wheel. See #4396 and #4706.

#10 Updated by Junxiao Shi about 1 year ago

Reply to note-8: This is exactly same idea as the RDR protocol (see note-3). Therefore, please work on #4706 to define the protocol, and then implement the protocol under this task.

#11 Updated by Chavoosh Ghasemi about 1 year ago

Cool, I just read RDR paper and as you mentioned it is exactly as same as my idea.
Anyways, to implement RDR, issues #4706, #4707, and #4577 should be solved. That sounds a lot of work! Any plan that how to manage them?

P.S. I am just wondering why RDR paper is out but its code is not.

#12 Updated by Chavoosh Ghasemi about 1 year ago

According to my meeting with Beichuan, this task can be divided into 2 different steps:
1- Making the FixedVersionDiscovery the default discovery version, so applications can work with packet format v0.3 and disabling IterativeVersionDiscovery
2- Finding a solution for real version discovery (e.g. RDR)

I am going to start with the first step and push the changes. If you think we need a separate redmine issue for this one, let me know.

#13 Updated by Davide Pesavento 12 months ago

  • Blocked by Feature #4706: RDR packet format specification added

#14 Updated by Spyros Mastorakis 12 months ago

Chavoosh Ghasemi wrote:

According to my meeting with Beichuan, this task can be divided into 2 different steps:
1- Making the FixedVersionDiscovery the default discovery version, so applications can work with packet format v0.3 and disabling IterativeVersionDiscovery
2- Finding a solution for real version discovery (e.g. RDR)

I am going to start with the first step and push the changes. If you think we need a separate redmine issue for this one, let me know.

About #2: each consumer can do discovery by sending out a discovery interest with name //metadata (with the mustBeFresh and MustBePrefix fields enabled). When the producer receives this Interest, it should send back the version of the latest file in the content of the metadata packet with name //metadata/. The name component is a monotonically increasing integer (starting from 0) that is appended to each metadata packet generated by the producer, so that it has a unique name. The freshness period of the metadata packet should be relatively small (e.g., 10ms should be a good starting point).

After the discovery phase, the consumer can start fetching the segments of the file.

#15 Updated by Junxiao Shi 11 months ago

I would suggest to get rid of "modular" version discovery altogether and use the following logic instead: if the user-provided name contains a version number, use that one and start fetching immediately; otherwise use RDR to obtain the latest version and then fetch the discovered version.

I agree with this solution because it reduces complexity.

#16 Updated by Chavoosh Ghasemi 11 months ago

I would suggest to get rid of "modular" version discovery altogether and use the following logic instead: if the user-provided name contains a version number, use that one and start fetching immediately; otherwise use RDR to obtain the latest version and then fetch the discovered version.

I agree with this solution because it reduces complexity.

How does this help improve the content retrieval process?

#17 Updated by Junxiao Shi 7 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

#18 Updated by Junxiao Shi 7 months ago

  • Status changed from Closed to In Progress
  • % Done changed from 100 to 30

#19 Updated by Junxiao Shi 7 months ago

  • Blocks Task #4832: ndncatchunks: change version discovery logic added

#20 Updated by Junxiao Shi 7 months ago

20190213 NFD call recognizes that the API of PipelineInterests::run cannot effectively support RDR version discovery method.

The existing iterative version discovery method would retrieve a data segment during version discovery.
Under this assumption, PipelineInterests::run method accepts this data segment: its name is used to derive versioned prefix, and its payload is stored so that the same segment does not need to be re-fetched.

RDR verson discovery method can only learn the versioned prefix from the metadata packet, but does not have a data segment.
To support RDR, PipelineInterests class should be generalized: PipelineInterests::run should accept versioned prefix instead of a data segment.
This does mean that the segment found during iterative or fixed version discovery would be re-fetched, but it would be supplied by the local node CS right away and would not generate excess network traffic.

#21 Updated by Chavoosh Ghasemi 7 months ago

To pass the version number to pipeline (instead of data packet), the following steps can be considered:

  • Change
    signal::Signal<DiscoverVersion, const Data&> onDiscoverySuccess in DiscoveryVersion class to
    signal::Signal<DiscoverVersion, const uint64_t> onDiscoverySuccess
    so it will accept version number and not data packet.

  • Change
    run(const Data& data, DataCallback onData, FailureCallback onFailure) in PipelineInterests class to
    run(const uint64_t version, DataCallback onData, FailureCallback onFailure)

  • Update Consumer class according to above changes

#22 Updated by Junxiao Shi 7 months ago

great, so we broke the default version discovery and the latest release of ndn-tools contains a broken-by-default chunks tool.
it would be a good idea to add a "unit" test that integrates consumer and producer with each other to ensure the pair of tools actually works together. If a unit test for this is infeasible, an integration test would still be better than nothing. But this doesn't have to be in this commit.

I thought it's in IntegrationTests but it's not. Yes, this should be structured as a unit test but it's out of scope of this issue.

#23 Updated by Davide Pesavento 7 months ago

  • % Done changed from 30 to 80

#24 Updated by Davide Pesavento 7 months ago

#25 Updated by Chavoosh Ghasemi 6 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 80 to 100

#26 Updated by Davide Pesavento 6 months ago

  • Status changed from Closed to In Progress
  • % Done changed from 100 to 90

This is not finished. There are still some uses of Exclude in the codebase. Those should be eliminated while you implement #4832.

#27 Updated by Davide Pesavento 6 months ago

  • Blocks deleted (Task #4832: ndncatchunks: change version discovery logic)

#28 Updated by Davide Pesavento 6 months ago

  • Blocked by Task #4832: ndncatchunks: change version discovery logic added

#29 Updated by Davide Pesavento 6 months ago

  • Subject changed from ndncatchunks: stop using ChildSelector and Exclude to ndncatchunks: stop using

#30 Updated by Davide Pesavento 6 months ago

  • Subject changed from ndncatchunks: stop using to ndncatchunks: stop using Selectors
  • Description updated (diff)

#31 Updated by Davide Pesavento 6 months ago

  • Description updated (diff)

#32 Updated by Davide Pesavento 5 months ago

  • Status changed from In Progress to Code review
  • % Done changed from 90 to 100

https://gerrit.named-data.net/c/ndn-tools/+/5386

With this change, ndn-tools (with tests) finally compiles with zero deprecation warnings.

#33 Updated by Davide Pesavento 5 months ago

  • Status changed from Code review to Closed

Also available in: Atom PDF