https://redmine.named-data.net/https://redmine.named-data.net/favicon.ico?14759811232017-10-27T13:52:14ZNDN project issue tracking systemndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=209732017-10-27T13:52:14ZAnonymous
<ul><li><strong>Blocked by</strong> <i><a class="issue tracker-2 status-5 priority-2 priority-default closed child" href="/issues/4289">Feature #4289</a>: ndncatchunks: React to congestion marks</i> added</li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=209772017-10-27T14:37:00ZDavide Pesavento
<ul><li><strong>Tracker</strong> changed from <i>Task</i> to <i>Feature</i></li></ul><p>I'm guessing this API should go in ndn-cxx (or a separate library on top of ndn-cxx).</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=209812017-10-27T16:17:12ZAnonymous
<ul></ul><p>Davide Pesavento wrote:</p>
<blockquote>
<p>I'm guessing this API should go in ndn-cxx (or a separate library on top of ndn-cxx).</p>
</blockquote>
<p>Makes sense.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=218112018-01-10T14:13:30ZJunxiao Shi
<ul><li><strong>Project</strong> changed from <i>NFD</i> to <i>ndn-cxx</i></li><li><strong>Category</strong> set to <i>Base</i></li><li><strong>Target version</strong> set to <i>v0.7</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=222922018-02-12T22:22:09ZDavide Pesavento
<ul><li><strong>Subject</strong> changed from <i>Create Consumer API for Congestion Control</i> to <i>Design Consumer API for Congestion Control</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224042018-02-17T18:43:38ZAnonymous
<ul><li><strong>Subject</strong> changed from <i>Design Consumer API for Congestion Control</i> to <i>Design Simple Consumer API for Congestion Control</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/22404/diff?detail_id=19597">diff</a>)</li><li><strong>Assignee</strong> set to <i>Eric Newberry</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224052018-02-17T18:50:55ZAnonymous
<ul></ul><p>I'm thinking about the following functionality abstraction: </p>
<p>At the producer:</p>
<ul>
<li>putObject(data, Name): Splits up the data into chunks, signs it, and puts it into the content repository.</li>
</ul>
<p>At the consumer:</p>
<ul>
<li>data getObject(Name): Reliably retrieves the data under a given name. This includes window adaptation, congestion control, and retransmissions. </li>
</ul>
<p>This functionality is currently provided by the putchunks and catchunks programs. However, the API should allow other applications to use these functions. </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224062018-02-17T19:05:34ZAnonymous
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/22406/diff?detail_id=19599">diff</a>)</li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224082018-02-17T19:21:42ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Should the <code>putObject</code> method above be blocking or non-blocking?</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224102018-02-17T19:33:07ZDavide Pesavento
<ul></ul><p>Klaus Schneider wrote:</p>
<blockquote>
<p>putObject(data, Name): Splits up the data into chunks, signs it, and puts it into the content repository.</p>
</blockquote>
<ul>
<li>What is "data"?</li>
<li>Is "content repository" an in-memory structure?</li>
<li>What happens if <code>putObject</code> is called more than once with the same Name? Does it create a new version? Throws an error? Something else?</li>
<li>What happens to "old" data? Is it kept in the "content repository" forever?</li>
</ul>
<blockquote>
<p>data getObject(Name): Reliably retrieves the data under a given name. This includes window adaptation, congestion control, and retransmissions. </p>
</blockquote>
<ul>
<li>If the data cannot be retrieved, does <code>getObject</code> keep retrying <em>ad infinitum</em>?</li>
</ul>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224112018-02-17T19:47:05ZAnonymous
<ul></ul><p>These are all good questions. Let's start with an easy one:</p>
<blockquote>
<ul>
<li>If the data cannot be retrieved, does <code>getObject</code> keep retrying <em>ad infinitum</em>?</li>
</ul>
</blockquote>
<p>No, there should be a maximum-retry threshold, after which the API gives up and returns an error code. </p>
<p>In catchunks this threshold is 3 by default, which I found often to be too low (at least with congestion marks disabled): Catchunks often returned an error, even though the file could have been transferred successfully with a couple more retransmissions.</p>
<p>Davide Pesavento wrote:</p>
<blockquote>
<p>Klaus Schneider wrote:</p>
<blockquote>
<p>putObject(data, Name): Splits up the data into chunks, signs it, and puts it into the content repository.</p>
</blockquote>
<ul>
<li>What is "data"?</li>
</ul>
</blockquote>
<p>The number of bytes that make up a content object. Can be a file on a disk, but doesn't have to be. </p>
<blockquote>
<ul>
<li>Is "content repository" an in-memory structure?</li>
</ul>
</blockquote>
<p>Intuitively, I thought of a permanent on-disk storage place, the way the term "repository" is used in the original CCN paper. However, I realize that putchunks serves the content from memory.</p>
<blockquote>
<ul>
<li>What happens if <code>putObject</code> is called more than once with the same Name? Does it create a new version? Throws an error? Something else?</li>
</ul>
</blockquote>
<p>Throwing an error would be the simpler solution. But if that's not satisfactory, we need some kind of version management.</p>
<blockquote>
<ul>
<li>What happens to "old" data? Is it kept in the "content repository" forever?</li>
</ul>
</blockquote>
<p>Yes? </p>
<p>I mean realistically, there should be some way to remove/overwrite it once the repository is full. But I'm not sure if we have to worry about this right now. </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224122018-02-17T19:49:14ZAnonymous
<ul></ul><p>Eric Newberry wrote:</p>
<blockquote>
<p>Should the <code>putObject</code> method above be blocking or non-blocking?</p>
</blockquote>
<p>Can we support both?</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224162018-02-17T23:18:14ZDavide Pesavento
<ul></ul><p>Klaus Schneider wrote:</p>
<blockquote>
<blockquote>
<ul>
<li>If the data cannot be retrieved, does <code>getObject</code> keep retrying <em>ad infinitum</em>?</li>
</ul>
</blockquote>
<p>No, there should be a maximum-retry threshold, after which the API gives up and returns an error code. </p>
<p>In catchunks this threshold is 3 by default, which I found often to be too low (at least with congestion marks disabled): Catchunks often returned an error, even though the file could have been transferred successfully with a couple more retransmissions.</p>
</blockquote>
<p>Maybe a global timeout ("how long is the user willing to wait?") makes more sense than a max no. of retries.</p>
<blockquote>
<blockquote>
<ul>
<li>Is "content repository" an in-memory structure?</li>
</ul>
</blockquote>
<p>Intuitively, I thought of a permanent on-disk storage place, the way the term "repository" is used in the original CCN paper. However, I realize that putchunks serves the content from memory.</p>
</blockquote>
<p>You haven't answered the question. If it's in memory, we already have <code>InMemoryStorage</code> that we can use. An on-disk repository is more complicated.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224172018-02-18T00:02:04ZAnonymous
<ul></ul><p>Davide Pesavento wrote:</p>
<blockquote>
<p>Klaus Schneider wrote:</p>
<blockquote>
<blockquote>
<ul>
<li>If the data cannot be retrieved, does <code>getObject</code> keep retrying <em>ad infinitum</em>?</li>
</ul>
</blockquote>
<p>No, there should be a maximum-retry threshold, after which the API gives up and returns an error code. </p>
<p>In catchunks this threshold is 3 by default, which I found often to be too low (at least with congestion marks disabled): Catchunks often returned an error, even though the file could have been transferred successfully with a couple more retransmissions.</p>
</blockquote>
<p>Maybe a global timeout ("how long is the user willing to wait?") makes more sense than a max no. of retries.</p>
</blockquote>
<p>Although that creates a couple of new questions: How is the waiting time measured? From the first request of the file? Or is the timer reset for each request?</p>
<p>Also, doesn't the tolerance ("how long is the user willing to wait?") depend on the file size?</p>
<p>I think it's easier to just increase the threshold a bit. + The congestion marks should get rid of the problem anyways.</p>
<blockquote>
<blockquote>
<blockquote>
<ul>
<li>Is "content repository" an in-memory structure?</li>
</ul>
</blockquote>
<p>Intuitively, I thought of a permanent on-disk storage place, the way the term "repository" is used in the original CCN paper. However, I realize that putchunks serves the content from memory.</p>
</blockquote>
<p>You haven't answered the question. If it's in memory, we already have <code>InMemoryStorage</code> that we can use. An on-disk repository is more complicated.</p>
</blockquote>
<p>Yeah, I would say let's start with the simpler solution.</p>
<p>I vaguely remember running experiments with a permanent repository. But that might have been with the ccnx-0.8.2 software, ccnputfile and ccngetfile. </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224282018-02-18T12:36:53ZDavide Pesavento
<ul></ul><p>Klaus Schneider wrote:</p>
<blockquote>
<p>Davide Pesavento wrote:</p>
<blockquote>
<p>Maybe a global timeout ("how long is the user willing to wait?") makes more sense than a max no. of retries.</p>
</blockquote>
<p>Although that creates a couple of new questions: How is the waiting time measured? From the first request of the file? Or is the timer reset for each request?</p>
<p>Also, doesn't the tolerance ("how long is the user willing to wait?") depend on the file size?</p>
</blockquote>
<p>I realized I haven't really explained what I meant. As long as the transfer progresses, there's no timeout. The countdown starts when the transfer stalls (no data packets are received anymore). This model matches what I, as a user, would expect from a simple "file download" application, and I suppose current web browsers also operate in a similar manner.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224292018-02-18T12:46:43ZAnonymous
<ul></ul><p>Davide Pesavento wrote:</p>
<blockquote>
<p>Klaus Schneider wrote:</p>
<blockquote>
<p>Davide Pesavento wrote:</p>
<blockquote>
<p>Maybe a global timeout ("how long is the user willing to wait?") makes more sense than a max no. of retries.</p>
</blockquote>
<p>Although that creates a couple of new questions: How is the waiting time measured? From the first request of the file? Or is the timer reset for each request?</p>
<p>Also, doesn't the tolerance ("how long is the user willing to wait?") depend on the file size?</p>
</blockquote>
<p>I realized I haven't really explained what I meant. As long as the transfer progresses, there's no timeout. The countdown starts when the transfer stalls (no data packets are received anymore). This model matches what I, as a user, would expect from a simple "file download" application, and I suppose current web browsers also operate in a similar manner.</p>
</blockquote>
<p>Yeah, this sounds good. So how long should this timeout be? 1-2 seconds? </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224302018-02-18T12:59:08ZDavide Pesavento
<ul></ul><p>Klaus Schneider wrote:</p>
<blockquote>
<p>Yeah, this sounds good. So how long should this timeout be? 1-2 seconds?</p>
</blockquote>
<p>It should be provided by the application. But the default should be much longer than that... I was thinking 30-60 seconds.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224312018-02-18T13:05:34ZAnonymous
<ul></ul><p>Davide Pesavento wrote:</p>
<blockquote>
<p>Klaus Schneider wrote:</p>
<blockquote>
<p>Yeah, this sounds good. So how long should this timeout be? 1-2 seconds?</p>
</blockquote>
<p>It should be provided by the application. But the default should be much longer than that... I was thinking 30-60 seconds.</p>
</blockquote>
<p>Sounds too long to me.</p>
<p>Unless we're dealing with mobility/intermittent connectivity, receiving no packets whatsoever for 2 seconds is already an extraordinarily rare event. </p>
<p>I don't think the default should be optimized for that case. But I do agree that the application should be able to choose that value.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224322018-02-18T13:15:16ZDavide Pesavento
<ul></ul><p>In 2018, I consider it a bug to not take mobility or imperfect connection conditions into account, regardless of the application. I don't want my download to fail because my wifi card is roaming to a different AP, or because my phone switched from wifi to cellular or vice versa.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224332018-02-18T13:28:27ZAnonymous
<ul></ul><p>Fair enough, but even those things take less than 5 seconds. </p>
<p>On the other hand, if my connection goes down or some link fails, I don't want to wait 30-60s for the application to tell me. </p>
<p>Maybe we resolve both needs by having the application print a message after X seconds that says "currently experiencing connectivity problems", just like Bluejeans does sometimes (at least on my connection). </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224342018-02-18T13:38:05ZJunxiao Shi
<ul></ul><p>Two parts are missing in note-7 design:</p>
<ul>
<li>At consumer: how to input the trust schema to the retriever?</li>
<li>At producer: how to publish a continuous stream as opposed to an object with a finite ending? It’s fine to pause reading from the stream when consumer isn’t making progress, similar to how ‘netcat’ works.</li>
</ul>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=224382018-02-18T18:06:06ZAnonymous
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<p>Two parts are missing in note-7 design:</p>
<ul>
<li>At consumer: how to input the trust schema to the retriever?</li>
</ul>
</blockquote>
<p>No idea. Don't we have security people to figure this out? :) </p>
<p>As far as I know, ndncatchunks doesn't perform any authenticity/integrity checks on the data either. </p>
<blockquote>
<ul>
<li>At producer: how to publish a continuous stream as opposed to an object with a finite ending? It’s fine to pause reading from the stream when consumer isn’t making progress, similar to how ‘netcat’ works.</li>
</ul>
</blockquote>
<p>Maybe we can focus on finite objects first, and worry about continuous streams later? </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=225942018-03-03T17:45:55ZEric Newberryenewberry@cs.ucla.edu
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=225982018-03-03T18:20:00ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Klaus Schneider wrote:</p>
<blockquote>
<p>Eric Newberry wrote:</p>
<blockquote>
<p>Should the <code>putObject</code> method above be blocking or non-blocking?</p>
</blockquote>
<p>Can we support both?</p>
</blockquote>
<p>To me, it only makes sense that <code>putObject</code> would split up the content into chunks and then serve it asynchronously, returning after it sets up the <code>InterestCallback</code>.</p>
<p>For <code>getObject</code>, I think we can only support asynchronous operation because <code>expressInterest</code> is asynchronous.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=226002018-03-03T18:46:21ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Eric Newberry wrote:</p>
<blockquote>
<p>Klaus Schneider wrote:</p>
<blockquote>
<p>Eric Newberry wrote:</p>
<blockquote>
<p>Should the <code>putObject</code> method above be blocking or non-blocking?</p>
</blockquote>
<p>Can we support both?</p>
</blockquote>
<p>To me, it only makes sense that <code>putObject</code> would split up the content into chunks and then serve it asynchronously, returning after it sets up the <code>InterestCallback</code>.</p>
<p>For <code>getObject</code>, I think we can only support asynchronous operation because <code>expressInterest</code> is asynchronous.</p>
</blockquote>
<p>Disregard both. I forgot about the mechanics of <code>Face::processEvents</code>.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=227632018-03-24T12:09:39ZAnonymous
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/22763/diff?detail_id=19897">diff</a>)</li></ul><p>The Cisco guys presented some update on their socket API on the last icnrg meeting: <a href="https://datatracker.ietf.org/meeting/interim-2018-icnrg-01/materials/slides-interim-2018-icnrg-01-sessa-hicn-socket-library-for-http-luca-muscariello-00">https://datatracker.ietf.org/meeting/interim-2018-icnrg-01/materials/slides-interim-2018-icnrg-01-sessa-hicn-socket-library-for-http-luca-muscariello-00</a></p>
<p>Might be interesting to figure out the differences to our API design. </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=227642018-03-24T12:12:04ZAnonymous
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229012018-04-05T21:01:21ZAnonymous
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Code review</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229022018-04-05T21:01:46ZAnonymous
<ul></ul><p>Link to gerrit: <a href="https://gerrit.named-data.net/c/4651/">https://gerrit.named-data.net/c/4651/</a></p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229322018-04-09T06:52:33ZJunxiao Shi
<ul></ul><blockquote>
<blockquote>
<p>At consumer: how to input the trust schema to the retriever?</p>
</blockquote>
<p>No idea. Don't we have security people to figure this out? :)</p>
</blockquote>
<p>It needs a <code>Validator</code> at least. You can report validation errors to the callers, although caller's processing logic is tricky (see <a class="issue tracker-2 status-1 priority-2 priority-default" title="Feature: NotificationSubscriber: Validator (New)" href="https://redmine.named-data.net/issues/3663">#3663</a>).</p>
<blockquote>
<blockquote>
<p>At producer: how to publish a continuous stream as opposed to an object with a finite ending? It’s fine to pause reading from the stream when consumer isn’t making progress, similar to how ‘netcat’ works.</p>
</blockquote>
<p>Maybe we can focus on finite objects first, and worry about continuous streams later?</p>
</blockquote>
<p>Continuous streams are more important (see <a class="issue tracker-3 status-5 priority-2 priority-default closed" title="Task: Provide canonical example for latest data retrieval without selectors in real-time applications (Closed)" href="https://redmine.named-data.net/issues/4396">#4396</a>). I don't see how "finite object publishing" is related to congestion control. It is just a regular file server.</p>
<blockquote>
<p><a href="https://gerrit.named-data.net/c/4651/">https://gerrit.named-data.net/c/4651/</a></p>
</blockquote>
<p>In patchset5 the code is in <code>src/util/</code> as two new classes.<br>
I disagree with having producer side at all.<br>
Consumer side should be integrated into <code>SegmentFetcher</code> that essentially does the same. You can move it to <code>src/socket/</code> if it's going to be more complicated.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229412018-04-09T13:35:23ZAnonymous
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<blockquote>
<blockquote>
<p>At consumer: how to input the trust schema to the retriever?</p>
</blockquote>
<p>No idea. Don't we have security people to figure this out? :)</p>
</blockquote>
<p>It needs a <code>Validator</code> at least. You can report validation errors to the callers, although caller's processing logic is tricky (see <a class="issue tracker-2 status-1 priority-2 priority-default" title="Feature: NotificationSubscriber: Validator (New)" href="https://redmine.named-data.net/issues/3663">#3663</a>).</p>
</blockquote>
<p>Yeah, we can think about that. </p>
<blockquote>
<blockquote>
<blockquote>
<p>At producer: how to publish a continuous stream as opposed to an object with a finite ending? It’s fine to pause reading from the stream when consumer isn’t making progress, similar to how ‘netcat’ works.</p>
</blockquote>
<p>Maybe we can focus on finite objects first, and worry about continuous streams later?</p>
</blockquote>
<p>Continuous streams are more important (see <a class="issue tracker-3 status-5 priority-2 priority-default closed" title="Task: Provide canonical example for latest data retrieval without selectors in real-time applications (Closed)" href="https://redmine.named-data.net/issues/4396">#4396</a>). </p>
</blockquote>
<p><a class="issue tracker-3 status-5 priority-2 priority-default closed" title="Task: Provide canonical example for latest data retrieval without selectors in real-time applications (Closed)" href="https://redmine.named-data.net/issues/4396">#4396</a> talks about "real-time applications". How does it show that those are more important than all the other applications? </p>
<blockquote>
<p>I don't see how "finite object publishing" is related to congestion control. </p>
</blockquote>
<p>Finite objects that consist of multiple chunks need congestion control to avoid overloading any links/routers inside the network. I really don't see any difference to streams in this regard. </p>
<blockquote>
<p>It is just a regular file server.</p>
</blockquote>
<p>It's like developing a regular file server <em>without being able to use the TCP protocol</em>. The transport-layer part is likely the larger contribution of this API. </p>
<blockquote>
<blockquote>
<p><a href="https://gerrit.named-data.net/c/4651/">https://gerrit.named-data.net/c/4651/</a></p>
</blockquote>
<p>In patchset5 the code is in <code>src/util/</code> as two new classes.<br>
I disagree with having producer side at all.</p>
</blockquote>
<p>How can you fetch content without having a producer? </p>
<blockquote>
<p>Consumer side should be integrated into <code>SegmentFetcher</code> that essentially does the same. You can move it to <code>src/socket/</code> if it's going to be more complicated.</p>
</blockquote>
<p>Can you give me a link to "SegmentFetcher"? </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229452018-04-09T22:43:21ZJunxiao Shi
<ul></ul><blockquote>
<blockquote>
<p>I don't see how "finite object publishing" is related to congestion control. </p>
</blockquote>
<p>Finite objects that consist of multiple chunks need congestion control to avoid overloading any links/routers inside the network. I really don't see any difference to streams in this regard. </p>
</blockquote>
<p>In 4651,5 <code>CongestionProducer</code> I see nothing related to congestion control. Its Interest processing behavior has no difference from a standard repository.</p>
<blockquote>
<blockquote>
<p>I disagree with having producer side at all.</p>
</blockquote>
<p>How can you fetch content without having a producer? </p>
</blockquote>
<p>Use a standard repository such as <a href="https://github.com/3rd-ndn-hackathon/repo-sql/" class="external">repo-sql</a> or <a href="https://github.com/remap/ndnfs-port/" class="external">NDNFS</a>.</p>
<blockquote>
<blockquote>
<p>Consumer side should be integrated into <code>SegmentFetcher</code> that essentially does the same. You can move it to <code>src/socket/</code> if it's going to be more complicated.</p>
</blockquote>
<p>Can you give me a link to "SegmentFetcher"?</p>
</blockquote>
<p>See <code>src/util/segment-fetcher.hpp</code>.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229462018-04-10T00:28:41ZAnonymous
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<blockquote>
<blockquote>
<p>I don't see how "finite object publishing" is related to congestion control. </p>
</blockquote>
<p>Finite objects that consist of multiple chunks need congestion control to avoid overloading any links/routers inside the network. I really don't see any difference to streams in this regard. </p>
</blockquote>
<p>In 4651,5 <code>CongestionProducer</code> I see nothing related to congestion control. </p>
</blockquote>
<p>Well, that's because the congestion control part is implemented on the consumer side. You can find it in the CongestionConsumer class.</p>
<p>I still don't see how this has anything to do with the difference between streams and finite objects. </p>
<blockquote>
<p>Its Interest processing behavior has no difference from a standard repository.</p>
</blockquote>
<p>Two things the producer does is to decide 1) how to segment a content object into chunks and 2) how to name those chunks. </p>
<p>This may be the same as a "standard repository", but then the behavior of the standard repository needs to be part of the API. For example, the consumer needs to know the naming convention. </p>
<blockquote>
<blockquote>
<blockquote>
<p>I disagree with having producer side at all.</p>
</blockquote>
<p>How can you fetch content without having a producer? </p>
</blockquote>
<p>Use a standard repository such as <a href="https://github.com/3rd-ndn-hackathon/repo-sql/" class="external">repo-sql</a> or <a href="https://github.com/remap/ndnfs-port/" class="external">NDNFS</a>.</p>
</blockquote>
<p>See my comment above. </p>
<blockquote>
<blockquote>
<blockquote>
<p>Consumer side should be integrated into <code>SegmentFetcher</code> that essentially does the same. You can move it to <code>src/socket/</code> if it's going to be more complicated.</p>
</blockquote>
<p>Can you give me a link to "SegmentFetcher"?</p>
</blockquote>
<p>See <code>src/util/segment-fetcher.hpp</code>.</p>
</blockquote>
<p>Our API seems to be higher-level than the SegementFetcher. That is, it provides a "fetchObject()" rather than a "fetchNextSegment()" function. </p>
<p>But I agree that there seems to be a lot of overlap. Maybe Eric can look into it. </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229472018-04-10T00:37:48ZJunxiao Shi
<ul></ul><blockquote>
<p>Two things the producer does is to decide 1) how to segment a content object into chunks and 2) how to name those chunks. </p>
<p>This may be the same as a "standard repository", but then the behavior of the standard repository needs to be part of the API. For example, the consumer needs to know the naming convention. </p>
<blockquote>
<p>Use a standard repository such as <a href="https://github.com/3rd-ndn-hackathon/repo-sql/" class="external">repo-sql</a> or <a href="https://github.com/remap/ndnfs-port/" class="external">NDNFS</a>.</p>
</blockquote>
</blockquote>
<p><a href="https://github.com/named-data/repo-ng/blob/93eb49ba064574369627e769e2c7885673293d54/tools/ndnputfile.cpp" class="external"><code>ndnputfile</code></a> inserts a file (i.e. finite object) into a repo-ng instance; there's no equivalent in repo-sql but you can send a pull request or I can add one in the future.<br>
NDNFS natively supports files via FUSE.<br>
They all use the same naming convention that is compatible with ndnputchunks.</p>
<blockquote>
<p>Our API seems to be higher-level than the SegementFetcher. That is, it provides a "fetchObject()" rather than a "fetchNextSegment()" function. </p>
</blockquote>
<p>The main API of <code>SegmentFetcher</code> is <code>SegmentFetcher::fetch</code> static function, which retrieves a finite object, not an individual segment. <code>fetchNextSegment</code> is a private function.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229482018-04-10T01:11:39ZAnonymous
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<blockquote>
<p>Two things the producer does is to decide 1) how to segment a content object into chunks and 2) how to name those chunks. </p>
<p>This may be the same as a "standard repository", but then the behavior of the standard repository needs to be part of the API. For example, the consumer needs to know the naming convention. </p>
<blockquote>
<p>Use a standard repository such as <a href="https://github.com/3rd-ndn-hackathon/repo-sql/" class="external">repo-sql</a> or <a href="https://github.com/remap/ndnfs-port/" class="external">NDNFS</a>.</p>
</blockquote>
</blockquote>
<p><a href="https://github.com/named-data/repo-ng/blob/93eb49ba064574369627e769e2c7885673293d54/tools/ndnputfile.cpp" class="external"><code>ndnputfile</code></a> inserts a file (i.e. finite object) into a repo-ng instance; there's no equivalent in repo-sql but you can send a pull request or I can add one in the future.<br>
NDNFS natively supports files via FUSE.<br>
They all use the same naming convention that is compatible with ndnputchunks.</p>
</blockquote>
<p>So the existing repos and putchunks all use the same unwritten naming conventions. </p>
<p>I think it's useful to make these conventions explicit and part of the API. </p>
<blockquote>
<blockquote>
<p>Our API seems to be higher-level than the SegementFetcher. That is, it provides a "fetchObject()" rather than a "fetchNextSegment()" function. </p>
</blockquote>
<p>The main API of <code>SegmentFetcher</code> is <code>SegmentFetcher::fetch</code> static function, which retrieves a finite object, not an individual segment. <code>fetchNextSegment</code> is a private function.</p>
</blockquote>
<p>Oh, I must have overlooked that. We should see how much of the code we can reuse.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229612018-04-11T17:13:54ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<blockquote>
<blockquote>
<p>At producer: how to publish a continuous stream as opposed to an object with a finite ending? It’s fine to pause reading from the stream when consumer isn’t making progress, similar to how ‘netcat’ works.</p>
</blockquote>
<p>Maybe we can focus on finite objects first, and worry about continuous streams later?</p>
</blockquote>
<p>Continuous streams are more important (see <a class="issue tracker-3 status-5 priority-2 priority-default closed" title="Task: Provide canonical example for latest data retrieval without selectors in real-time applications (Closed)" href="https://redmine.named-data.net/issues/4396">#4396</a>). I don't see how "finite object publishing" is related to congestion control. It is just a regular file server.</p>
</blockquote>
<p>I disagree - both are important. Congestion control is related to "finite object pushing" or any other transfer because the amount of data present may congest some links, no matter if it is a continuous stream or a finite object. This is especially true in the latter case when the object is large. However, in this particular scenario, we just want to make an API for finite objects to demonstrate congestion control and allow it to be easily adopted by NDN applications.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229622018-04-11T17:16:45ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Klaus Schneider wrote:</p>
<blockquote>
<p>Junxiao Shi wrote:</p>
<blockquote>
<blockquote>
<p>Two things the producer does is to decide 1) how to segment a content object into chunks and 2) how to name those chunks. </p>
<p>This may be the same as a "standard repository", but then the behavior of the standard repository needs to be part of the API. For example, the consumer needs to know the naming convention. </p>
<blockquote>
<p>Use a standard repository such as <a href="https://github.com/3rd-ndn-hackathon/repo-sql/" class="external">repo-sql</a> or <a href="https://github.com/remap/ndnfs-port/" class="external">NDNFS</a>.</p>
</blockquote>
</blockquote>
<p><a href="https://github.com/named-data/repo-ng/blob/93eb49ba064574369627e769e2c7885673293d54/tools/ndnputfile.cpp" class="external"><code>ndnputfile</code></a> inserts a file (i.e. finite object) into a repo-ng instance; there's no equivalent in repo-sql but you can send a pull request or I can add one in the future.<br>
NDNFS natively supports files via FUSE.<br>
They all use the same naming convention that is compatible with ndnputchunks.</p>
</blockquote>
<p>So the existing repos and putchunks all use the same unwritten naming conventions. </p>
<p>I think it's useful to make these conventions explicit and part of the API. </p>
</blockquote>
<p>I 100% agree. Nothing should be "assumed" in a protocol without being explicitly written down.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229632018-04-11T17:17:37ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>I think it would be good to see whether congestion control can be merged with the SegmentFetcher API, instead of as a separate component. I'll evaluate this and report back.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229692018-04-11T17:27:46ZDavide Pesavento
<ul></ul><blockquote>
<blockquote>
<blockquote>
<p>They all use the same naming convention that is compatible with ndnputchunks.</p>
</blockquote>
<p>So the existing repos and putchunks all use the same unwritten naming conventions. </p>
<p>I think it's useful to make these conventions explicit and part of the API. </p>
</blockquote>
<p>I 100% agree. Nothing should be "assumed" in a protocol without being explicitly written down.</p>
</blockquote>
<p>Are you talking about <a href="https://named-data.net/wp-content/uploads/2014/08/ndn-tr-22-ndn-memo-naming-conventions.pdf">https://named-data.net/wp-content/uploads/2014/08/ndn-tr-22-ndn-memo-naming-conventions.pdf</a> ?</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=229762018-04-12T15:36:18ZAnonymous
<ul></ul><p>I was thinking about getting rid of the version numbers in the name, in order to simplify the API.</p>
<p>For catchunks the version numbers are a major pain (requires a discovery mechanism and a timeout in the beginning) + they seem to be barely used by anyone. </p>
<p>In fact, I'm not sure if ndnputchunks even supports uploading the same content with a newer version. </p>
<p>If someone wants a new version, they can just change the name slightly, or add their own version prefix for the name. </p>
<p>Any comments or use cases that I'm missing here? </p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=230022018-04-13T22:06:22ZJunxiao Shi
<ul><li><strong>Status</strong> changed from <i>Code review</i> to <i>Feedback</i></li></ul><blockquote>
<p>I was thinking about getting rid of the version numbers in the name, in order to simplify the API.<br>
For catchunks the version numbers are a major pain (requires a discovery mechanism and a timeout in the beginning) + they seem to be barely used by anyone. </p>
</blockquote>
<p>Versioning is independent from congestion control. You can keep <code>SegmentFetcher</code>’s very simple version discovery (send ChildSelector=1 and take whatever version is returned) for now, and focus on how to add congestion control to the retrieval process after the version is determined.<br>
In <a class="issue tracker-3 status-5 priority-3 priority-high3 closed" title="Task: SegmentFetcher: eliminate selector usage (Closed)" href="https://redmine.named-data.net/issues/4555">#4555</a> we can consider remove or change <code>SegmentFetcher</code>’s version discovery procedure.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=231882018-05-07T15:02:59ZJunxiao Shi
<ul></ul><blockquote>
<p>Do not change SegmentFetcher::fetch function. Complete <a class="issue tracker-3 status-5 priority-2 priority-default closed" title="Task: Make SegmentFetcher pure Signal-based (Closed)" href="https://redmine.named-data.net/issues/4464">#4464</a> first before adding more features.</p>
</blockquote>
<p>Additional explanation:<br>
Parameters to the congestion control algorithm should be provided as getter/setter on the <code>SegmentFetcher</code> instance. You may either define multiple getters and setters, or combine several parameters into a POD struct and set all of them at once.<br>
Statistics of the congestion control algorithms can be collected internally, and exported via getter. If collection of some statistics would cause high performance overhead, they should be disabled initially and enabled via a function call.</p>
<p>Regarding <a class="issue tracker-3 status-5 priority-3 priority-high3 closed" title="Task: SegmentFetcher: eliminate selector usage (Closed)" href="https://redmine.named-data.net/issues/4555">#4555</a>: There’s no blocking relation. However, this issue must not introduce new selector usage.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=231902018-05-07T15:03:15ZJunxiao Shi
<ul><li><strong>Blocked by</strong> <i><a class="issue tracker-3 status-5 priority-2 priority-default closed" href="/issues/4464">Task #4464</a>: Make SegmentFetcher pure Signal-based</i> added</li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=233762018-06-01T21:28:52ZEric Newberryenewberry@cs.ucla.edu
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Code review</i></li><li><strong>% Done</strong> changed from <i>10</i> to <i>100</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=234892018-06-18T00:36:46ZEric Newberryenewberry@cs.ucla.edu
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-3 status-1 priority-2 priority-default" href="/issues/4637">Task #4637</a>: ndncatchunks: use SegmentFetcher</i> added</li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235292018-06-25T09:15:03ZEric Newberryenewberry@cs.ucla.edu
<ul><li><strong>Status</strong> changed from <i>Code review</i> to <i>Closed</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235332018-06-25T10:00:45ZAnonymous
<ul></ul><p>Where is the congestion control API documented?</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235342018-06-25T13:41:09ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Jeff Thompson wrote:</p>
<blockquote>
<p>Where is the congestion control API documented?</p>
</blockquote>
<p>Are you looking for end-user docs (e.g., how to use the API) or docs on the internal design of the fetching API?</p>
<p>For the former, docs should be available in-line and on Doxygen (<code>util::SegmentFetcher</code>).</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235352018-06-25T13:56:31ZDavide Pesavento
<ul></ul><p>Jeff Thompson wrote:</p>
<blockquote>
<p>Where is the congestion control API documented?</p>
</blockquote>
<p>We actually did not design any new API in this task. The code that was merged simply implements congestion control in the existing <code>SegmentFetcher</code> component. The issue title should be changed to avoid confusion.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235362018-06-25T13:59:13ZDavide Pesavento
<ul></ul><p>Also, I'm not sure what "congestion control API" means. Congestion control is not an API.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235372018-06-25T14:00:21ZDavide Pesavento
<ul><li><strong>Subject</strong> changed from <i>Design Simple Consumer API for Congestion Control</i> to <i>Implement congestion control in SegmentFetcher</i></li><li><strong>Category</strong> changed from <i>Base</i> to <i>Utils</i></li></ul> ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235382018-06-25T14:27:08ZEric Newberryenewberry@cs.ucla.edu
<ul></ul><p>Davide Pesavento wrote:</p>
<blockquote>
<p>Jeff Thompson wrote:</p>
<blockquote>
<p>Where is the congestion control API documented?</p>
</blockquote>
<p>We actually did not design any new API in this task. The code that was merged simply implements congestion control in the existing <code>SegmentFetcher</code> component. The issue title should be changed to avoid confusion.</p>
</blockquote>
<p>However, the SegmentFetcher API was changed recently in <a class="issue tracker-3 status-5 priority-2 priority-default closed" title="Task: Make SegmentFetcher pure Signal-based (Closed)" href="https://redmine.named-data.net/issues/4464">#4464</a>.</p>
ndn-cxx - Feature #4364: Implement congestion control in SegmentFetcherhttps://redmine.named-data.net/issues/4364?journal_id=235422018-06-25T19:54:26ZDavide Pesavento
<ul></ul><p>Yes, but that had nothing to do with congestion control.</p>