https://redmine.named-data.net/https://redmine.named-data.net/favicon.ico?14759811232018-08-07T13:29:56ZNDN project issue tracking systemndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=237922018-08-07T13:29:56ZDavide Pesavento
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/23792/diff?detail_id=20858">diff</a>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=237942018-08-07T13:44:50ZJunxiao Shi
<ul><li><strong>Subject</strong> changed from <i>Provide a segment publisher to complement segment fetcher</i> to <i>SegmentPublisher: segmentation utility</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/23794/diff?detail_id=20863">diff</a>)</li><li><strong>Target version</strong> set to <i>v0.7</i></li><li><strong>Estimated time</strong> set to <i>6.00 h</i></li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=237952018-08-07T14:00:45ZJunxiao Shi
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-1 priority-2 priority-default" href="/issues/4699">Feature #4699</a>: Dispatcher: use SegmentPublisher in StatusDatasetContext</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=238032018-08-07T15:15:10ZDavide Pesavento
<ul></ul><blockquote>
<p>It should support both a finite object and an infinite stream.</p>
</blockquote>
<p>Can you elaborate on the "infinite stream"? Are you simply talking about the case when the content is read from a <code>std::istream</code>? Or are you referring to continuous data production? I think you're mixing two things here: (i) whether the ADU is a stream or a quantized piece of content with well-defined boundaries, and (ii) whether data production is continuous or instantaneous; these two dimensions are orthogonal to each other.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=238072018-08-07T22:13:02ZDavide Pesavento
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-3 status-5 priority-2 priority-default closed" href="/issues/4702">Task #4702</a>: ndnputchunks: use Segmenter class</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=241682018-10-18T17:10:02ZAshlesh Gawande
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-5 priority-2 priority-default closed" href="/issues/4760">Feature #4760</a>: Use SegmentPublisher utility from PSync to publish LSA data</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=241722018-10-18T17:11:15ZAshlesh Gawande
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-5 priority-3 priority-high3 closed" href="/issues/4662">Feature #4662</a>: Segment hello and sync data in partial sync</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=241762018-10-18T17:11:59ZAshlesh Gawande
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-3 status-5 priority-2 priority-default closed" href="/issues/4716">Task #4716</a>: Segment sync data in full sync</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=242492018-11-01T14:04:21ZAshlesh Gawande
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-2 status-5 priority-3 priority-high3 closed" href="/issues/4662">Feature #4662</a>: Segment hello and sync data in partial sync</i>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=242512018-11-01T14:04:27ZAshlesh Gawande
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-3 status-5 priority-2 priority-default closed" href="/issues/4716">Task #4716</a>: Segment sync data in full sync</i>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=244452018-12-17T11:52:44ZAshlesh Gawande
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-2 status-5 priority-2 priority-default closed" href="/issues/4760">Feature #4760</a>: Use SegmentPublisher utility from PSync to publish LSA data</i>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=244482018-12-17T12:18:20ZAshlesh Gawande
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-5 priority-2 priority-default closed" href="/issues/4791">Feature #4791</a>: Use SegmentPublisher utility from ndn-cxx to publish LSA data</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=249652019-03-08T16:43:30ZJunxiao Shi
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-2 status-7 priority-2 priority-default closed" href="/issues/1354">Feature #1354</a>: SegmentedEncodingBuffer</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=257472019-08-27T20:53:53ZDavide Pesavento
<ul><li><strong>Target version</strong> deleted (<del><i>v0.7</i></del>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=259592019-10-18T13:18:03ZJunxiao Shi
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-2 priority-2 priority-default" href="/issues/5004">Feature #5004</a>: CertificateBundle encoding and decoding</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=259622019-10-18T13:31:03ZJunxiao Shi
<ul></ul><blockquote>
<p>Can you elaborate on the "infinite stream"? Are you simply talking about the case when the content is read from a <code>std::istream</code>? Or are you referring to continuous data production?</p>
</blockquote>
<p>I'm referring to continuous data production.</p>
<p>For reference:<br>
Recently I'm attempting to implement <code>SegmentPublisher</code> in <a href="https://github.com/yoursunny/NDNts" class="external">NDNts</a>. It has two parts: <a href="https://github.com/yoursunny/NDNts/blob/cfaf04672bd097a6e42b915cb99ac7ea5b39a932/packages/segmented-object/src/chunker.ts" class="external">chunker</a> slices the input into segment payloads no larger than a specified size, and <a href="https://github.com/yoursunny/NDNts/blob/cfaf04672bd097a6e42b915cb99ac7ea5b39a932/packages/segmented-object/src/serve.ts" class="external">serve</a> turns the chunks into Data packets for available.<br>
The basic logic is pull-based: when the producer receives an Interest for a segment not yet generated, it attempts to read more input via the chunker, until reaching the requested segment number.<br>
A difficulty I'm running into is, since the input stream is non-seekable, the producer has to keep all generated segments in memory, because it should be able to satisfy an Interest asking for an older segment. <code>ndnputchunks</code> has the same problem <a class="issue tracker-1 status-1 priority-2 priority-default" title="Bug: ndnputchunks consumes a large amount of RAM (New)" href="https://redmine.named-data.net/issues/4866">#4866</a>.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264202020-03-13T03:46:15ZAshlesh Gawande
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<blockquote>
<p>Can you elaborate on the "infinite stream"? Are you simply talking about the case when the content is read from a <code>std::istream</code>? Or are you referring to continuous data production?</p>
</blockquote>
<p>I'm referring to continuous data production.</p>
<p>For reference:<br>
Recently I'm attempting to implement <code>SegmentPublisher</code> in <a href="https://github.com/yoursunny/NDNts" class="external">NDNts</a>. It has two parts: <a href="https://github.com/yoursunny/NDNts/blob/cfaf04672bd097a6e42b915cb99ac7ea5b39a932/packages/segmented-object/src/chunker.ts" class="external">chunker</a> slices the input into segment payloads no larger than a specified size, and <a href="https://github.com/yoursunny/NDNts/blob/cfaf04672bd097a6e42b915cb99ac7ea5b39a932/packages/segmented-object/src/serve.ts" class="external">serve</a> turns the chunks into Data packets for available.<br>
The basic logic is pull-based: when the producer receives an Interest for a segment not yet generated, it attempts to read more input via the chunker, until reaching the requested segment number.<br>
A difficulty I'm running into is, since the input stream is non-seekable, the producer has to keep all generated segments in memory, because it should be able to satisfy an Interest asking for an older segment. <code>ndnputchunks</code> has the same problem <a class="issue tracker-1 status-1 priority-2 priority-default" title="Bug: ndnputchunks consumes a large amount of RAM (New)" href="https://redmine.named-data.net/issues/4866">#4866</a>.</p>
</blockquote>
<p>Can supporting continuous data stream be a separate issue - seems more complicated (can't seem to find time to research further)? This can be just a simple segment and store a given buffer (instantaneous) utility (a more generalized version of PSync's).</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264212020-03-13T04:33:05ZJunxiao Shi
<ul></ul><p>You don't have to solve the infinite buffering issue. Other than that, keep reading from a stream and generate a segment when you have a pre-configured amount of payload. It's simple enough to fit in this issue.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264222020-03-13T10:18:34ZJeremy Clark
<ul></ul><p>For the fetcher side of this change, should the current behavior where the fetcher retrieves all segments up to the last block, writes them to a buffer, and passes the buffer to the calling process still be supported?</p>
<p>If the stream is continuous, the m_receivedSegments hashmap can't keep storing new segments. </p>
<p>Or is it possible the fetcher can determine whether the object is finite or infinite such as through the existence of a final block ID in the first segment and react accordingly?</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264282020-03-15T18:10:34ZAshlesh Gawande
<ul></ul><p>Okay, so I think this SegmentPublisher should basically be a Segmenter and just be responsible for Segmenting given data and store it in desired storage type. Publishing would then be the responsibility of the app. </p>
<pre><code class="c++ syntaxhl" data-language="c++"><span class="cp">#include</span> <span class="cpf"><ndn-cxx/face.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/security/v2/key-chain.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-fifo.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-persistent.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-lru.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-lfu.hpp></span><span class="cp">
</span>
<span class="k">namespace</span> <span class="n">ndn</span> <span class="p">{</span>
<span class="k">namespace</span> <span class="n">util</span> <span class="p">{</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">IMS</span><span class="p">></span>
<span class="k">class</span> <span class="nc">Segmenter</span>
<span class="p">{</span>
<span class="k">static_assert</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">is_base_of</span><span class="o"><</span><span class="n">ndn</span><span class="o">::</span><span class="n">InMemoryStorage</span><span class="p">,</span> <span class="n">IMS</span><span class="o">>::</span><span class="n">value</span> <span class="o">==</span> <span class="nb">true</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="k">struct</span> <span class="nc">Options</span>
<span class="p">{</span>
<span class="n">security</span><span class="o">::</span><span class="n">SigningInfo</span> <span class="n">signingInfo</span><span class="p">;</span>
<span class="n">time</span><span class="o">::</span><span class="n">milliseconds</span> <span class="n">freshnessPeriod</span><span class="p">{</span><span class="mi">10000</span><span class="p">};</span>
<span class="kt">size_t</span> <span class="n">maxSegmentSize</span> <span class="o">=</span> <span class="n">MAX_NDN_PACKET_SIZE</span> <span class="o">>></span> <span class="mi">1</span><span class="p">;</span>
<span class="c1">// optionally return the corresponding segment to application so that it can answer the interest:</span>
<span class="n">optional</span><span class="o"><</span><span class="kt">uint64_t</span><span class="o">></span> <span class="n">seg</span><span class="p">;</span>
<span class="p">};</span>
<span class="cm">/*
Always use ioService in IMS as it seems like without it data is not marked stale
(this could be leading to some problems that we have seen in PSync)
I think Apps can use longer freshness instead of never marking the data stale ?
*/</span>
<span class="cm">/*
Persistent does not have a limit and hence no limit constructor
*/</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span> <span class="o">=</span> <span class="n">IMS</span><span class="p">,</span>
<span class="k">typename</span> <span class="o">=</span> <span class="k">typename</span> <span class="nc">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">InMemoryStoragePersistent</span><span class="p">></span><span class="o">::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span><span class="o">></span>
<span class="n">Segmenter</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="n">KeyChain</span><span class="o">&</span> <span class="n">keyChain</span><span class="p">)</span>
<span class="o">:</span> <span class="n">m_keyChain</span><span class="p">(</span><span class="n">keyChain</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ims</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span><span class="p">(</span><span class="n">ioService</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span> <span class="o">=</span> <span class="n">IMS</span><span class="p">,</span>
<span class="k">typename</span> <span class="o">=</span> <span class="k">typename</span> <span class="nc">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><!</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">InMemoryStoragePersistent</span><span class="p">></span><span class="o">::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span><span class="o">></span>
<span class="n">Segmenter</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="n">KeyChain</span><span class="o">&</span> <span class="n">keyChain</span><span class="p">,</span>
<span class="kt">size_t</span> <span class="n">limit</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">numeric_limits</span><span class="o"><</span><span class="kt">size_t</span><span class="o">>::</span><span class="n">max</span><span class="p">())</span>
<span class="o">:</span> <span class="n">m_keyChain</span><span class="p">(</span><span class="n">keyChain</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ims</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span><span class="p">(</span><span class="n">ioService</span><span class="p">,</span> <span class="n">limit</span><span class="p">);</span>
<span class="p">}</span>
<span class="cm">/*
Not sure whether Options should go here or in the constructor or both?
It makes more sense here, as applications maybe serving different kinds of data
*/</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="k">const</span> <span class="kt">uint8_t</span><span class="o">*</span> <span class="n">buffer</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">bufferSize</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">Block</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="cm">/*
Not sure if this needs to be asynchronous? putchunk's is synchronous.
*/</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">is</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="n">std</span><span class="o">::</span><span class="n">unique_ptr</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span> <span class="n">ims</span><span class="p">;</span>
<span class="nl">private:</span>
<span class="n">KeyChain</span><span class="o">&</span> <span class="n">m_keyChain</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}</span> <span class="c1">// namespace util</span>
<span class="p">}</span> <span class="c1">// namespace ndn</span>
</code></pre>
<p>Applications should have the Segmenter as a member.<br>
Applications can then segment and store that data using Segmenter and serve it by directly accessing the public ims.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264352020-03-16T19:14:20ZJunxiao Shi
<ul></ul><blockquote>
<p>Okay, so I think this SegmentPublisher should basically be a Segmenter and just be responsible for Segmenting given data and store it in desired storage type.</p>
</blockquote>
<p>Yes.</p>
<blockquote>
<pre><code class="c++ syntaxhl" data-language="c++"> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="k">const</span> <span class="kt">uint8_t</span><span class="o">*</span> <span class="n">buffer</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">bufferSize</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">Block</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">is</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
</code></pre></blockquote>
<p>What's the return value of these functions? Data packets have been inserted into the IMS, and do not need to be returned.</p>
<blockquote>
<pre><code class="c++ syntaxhl" data-language="c++"> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">is</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
</code></pre></blockquote>
<p>This function needs to be async. <code>ndnputchunks</code> assumes the stream is finite, which is not necessarily the case. It breaks if you pipe a infinite stream (e.g. video from webcam) into <code>ndnputchunks</code>.</p>
<blockquote>
<p>Publishing would then be the responsibility of the app. <br>
Applications should have the Segmenter as a member.<br>
Applications can then segment and store that data using Segmenter and serve it by directly accessing the public ims.</p>
</blockquote>
<p>This functionality should be provided in another class, also under this Feature.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264362020-03-16T19:16:05ZJunxiao Shi
<ul></ul><p>For reference, I just put a similar feature in <a href="https://github.com/yoursunny/NDNts/tree/01fc4f7d9a107689e1a416e262a15bd79cf30ba8/packages/segmented-object" class="external"><code>@ndn/segmented-object</code> package</a>. Its main components are:</p>
<ul>
<li><strong>ChunkSource</strong> takes an input that is either a buffer, a (potentially infinite) stream, or an iterable of buffers, and returns an iterable of chunks that are sized properly for segments.</li>
<li><strong>DataProducer</strong> turns these chunks into Data packets. It handles Data naming, setting ContentType and FreshnessPeriod, etc. In the future it will also handle encryption.</li>
<li><strong>SequentialDataProducer</strong> adds Data buffering functionality, which is needed only if the underlying <strong>ChunkSource</strong> is not "seekable": a buffer is seekable because you can read an earlier chunk again, while a stream is not seekable. It has <code>bufferBehind</code> option that determines how many earlier segments (in regards to latest arriving Interest) are to be kept in the buffer. The caveat is that, if <code>bufferBehind</code> is not infinity, some Interests requesting early segments may not be answered when there are multiple consumers.</li>
<li><strong>serve</strong> function handles prefix registration.</li>
<li><strong>serveVersioned</strong> function can automatically insert a version component, enabling basic version discovery through CanBePrefix. In the future it will also publish RDR metadata packet.</li>
</ul>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264472020-03-18T03:38:14ZAshlesh Gawande
<ul></ul><p>Thank you for the comments.</p>
<p>The return value is the data corresponding to the given segment (named seg in Options for now) after segmentation.<br>
If an app (or the new Publisher class) receives an interest, and wants to segment data for it, it can tell the Segmenter to return the segment the interest wants. Or is it better to let it do the segmentation and then let the app (or Publisher class) find corresponding data using ims in such an on-demand segmentation?</p>
<p>For the Publisher class - is there any functionality overlap with the Dispatcher?</p>
<p>Also, a question on the ndnts feature - is there a major difference between having a bufferBehind option and say using a Fifo IMS with a reasonable limit (as dictated by the app) such that old data is kicked out on its own as the stream continues?</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264482020-03-18T06:17:44ZAshlesh Gawande
<ul></ul><p>Updated class, not sure what's the best way to make the function async:</p>
<pre><code class="c++ syntaxhl" data-language="c++"><span class="cp">#include</span> <span class="cpf"><ndn-cxx/face.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/security/v2/key-chain.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-fifo.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-persistent.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-lru.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-lfu.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/util/scheduler.hpp></span><span class="cp">
</span>
<span class="k">namespace</span> <span class="n">ndn</span> <span class="p">{</span>
<span class="k">namespace</span> <span class="n">util</span> <span class="p">{</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">IMS</span><span class="p">></span>
<span class="k">class</span> <span class="nc">Segmenter</span>
<span class="p">{</span>
<span class="k">static_assert</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">is_base_of</span><span class="o"><</span><span class="n">InMemoryStorage</span><span class="p">,</span> <span class="n">IMS</span><span class="o">>::</span><span class="n">value</span> <span class="o">==</span> <span class="nb">true</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="k">struct</span> <span class="nc">Options</span>
<span class="p">{</span>
<span class="n">security</span><span class="o">::</span><span class="n">SigningInfo</span> <span class="n">signingInfo</span><span class="p">;</span>
<span class="n">time</span><span class="o">::</span><span class="n">milliseconds</span> <span class="n">freshnessPeriod</span><span class="p">{</span><span class="mi">10000</span><span class="p">};</span>
<span class="kt">size_t</span> <span class="n">maxSegmentSize</span> <span class="o">=</span> <span class="n">MAX_NDN_PACKET_SIZE</span> <span class="o">>></span> <span class="mi">1</span><span class="p">;</span>
<span class="n">optional</span><span class="o"><</span><span class="kt">uint64_t</span><span class="o">></span> <span class="n">segment</span><span class="p">;</span> <span class="c1">// return the requested segment to app so that it can answer the interest</span>
<span class="kt">bool</span> <span class="n">removeStale</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span> <span class="c1">// Schedule removal of data from IMS after freshnessPeriod if true</span>
<span class="p">};</span>
<span class="cm">/*
Always use ioService in IMS as it seems like without it data is not marked stale
I think Apps can use longer freshness instead of never marking the data stale?
*/</span>
<span class="cm">/*
Persistent does not have a limit
*/</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span> <span class="o">=</span> <span class="n">IMS</span><span class="p">,</span>
<span class="k">typename</span> <span class="o">=</span> <span class="k">typename</span> <span class="nc">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">InMemoryStoragePersistent</span><span class="p">></span><span class="o">::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span><span class="o">></span>
<span class="n">Segmenter</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="n">KeyChain</span><span class="o">&</span> <span class="n">keyChain</span><span class="p">)</span>
<span class="o">:</span> <span class="n">m_scheduler</span><span class="p">(</span><span class="n">ioService</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_keyChain</span><span class="p">(</span><span class="n">keyChain</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ims</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span><span class="p">(</span><span class="n">ioService</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span> <span class="o">=</span> <span class="n">IMS</span><span class="p">,</span>
<span class="k">typename</span> <span class="o">=</span> <span class="k">typename</span> <span class="nc">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><!</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">InMemoryStoragePersistent</span><span class="p">></span><span class="o">::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span><span class="o">></span>
<span class="n">Segmenter</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="n">KeyChain</span><span class="o">&</span> <span class="n">keyChain</span><span class="p">,</span>
<span class="kt">size_t</span> <span class="n">limit</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">numeric_limits</span><span class="o"><</span><span class="kt">size_t</span><span class="o">>::</span><span class="n">max</span><span class="p">())</span>
<span class="o">:</span> <span class="n">m_scheduler</span><span class="p">(</span><span class="n">ioService</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_keyChain</span><span class="p">(</span><span class="n">keyChain</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ims</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span><span class="p">(</span><span class="n">ioService</span><span class="p">,</span> <span class="n">limit</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="k">const</span> <span class="kt">uint8_t</span><span class="o">*</span> <span class="n">buffer</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">bufferSize</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">Block</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="cm">/*
If infinite, then I was thinking to spawn a detached std::thread to insert from the stream continously.
-> I guess no locks are needed assuming that application will only use
this function, use it once (maybe we could enforce this), and not insert anything in the IMS directly.
Or is it better to put a lock when inserting?
-> Are there any alternatives to using std::thread to be considered?
*/</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">is</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">infinite</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="n">std</span><span class="o">::</span><span class="n">unique_ptr</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span> <span class="n">ims</span><span class="p">;</span>
<span class="nl">private:</span>
<span class="n">Scheduler</span> <span class="n">m_scheduler</span><span class="p">;</span>
<span class="n">KeyChain</span><span class="o">&</span> <span class="n">m_keyChain</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}</span> <span class="c1">// namespace util</span>
<span class="p">}</span> <span class="c1">// namespace ndn</span>
</code></pre> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264742020-03-22T07:23:02ZJunxiao Shi
<ul></ul><p>To store a (potentially infinite) stream, using a secondary thread would open up a lot of thread safety issues. Instead:</p>
<ul>
<li>Maintain a list of in-progress stream store operations in the segmenter.</li>
<li>Triggered by the associated <code>boost::asio::io_context</code>, read from stream and save into IMS.
<ul>
<li>In Arduino this would be called in <code>loop()</code> function. You'll have to find the Asio equivalent.</li>
</ul></li>
<li>The <code>store</code> function should return a <em>handle</em> of the in-progress stream store operation, which has
<ul>
<li>Function to cancel the operation: stop storing more segments and close the stream)</li>
<li>Function to know progress, i.e. how many segments have been stored.</li>
<li>Signals to know progress: triggered after storing each segment, and triggered when the operation has completed and stream is closed.</li>
<li>Signal to receive errors.</li>
</ul></li>
<li>An in-progress stream store operation should be removed from the aforementioned list when it's completed, canceled, or errored.</li>
</ul>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=264782020-03-23T19:06:46ZAshlesh Gawande
<ul></ul><p>Thanks, not sure on two points:</p>
<ol>
<li>How to use std::istream with posix::stream_descriptor (Currently passing file descriptor instead)?</li>
<li>Should we async_read_some or async_read?</li>
</ol>
<p>Updated:</p>
<pre><code class="c++ syntaxhl" data-language="c++"><span class="cp">#include</span> <span class="cpf"><ndn-cxx/face.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/security/v2/key-chain.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-fifo.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-persistent.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-lru.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/ims/in-memory-storage-lfu.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/util/scheduler.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/util/signal.hpp></span><span class="cp">
#include</span> <span class="cpf"><ndn-cxx/detail/cancel-handle.hpp></span><span class="cp">
</span>
<span class="cp">#include</span> <span class="cpf"><boost/asio/posix/stream_descriptor.hpp></span><span class="cp">
</span>
<span class="k">namespace</span> <span class="n">ndn</span> <span class="p">{</span>
<span class="k">namespace</span> <span class="n">util</span> <span class="p">{</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">IMS</span><span class="p">></span>
<span class="k">class</span> <span class="nc">Segmenter</span>
<span class="p">{</span>
<span class="k">static_assert</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">is_base_of</span><span class="o"><</span><span class="n">InMemoryStorage</span><span class="p">,</span> <span class="n">IMS</span><span class="o">>::</span><span class="n">value</span> <span class="o">==</span> <span class="nb">true</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="k">struct</span> <span class="nc">Options</span>
<span class="p">{</span>
<span class="n">security</span><span class="o">::</span><span class="n">SigningInfo</span> <span class="n">signingInfo</span><span class="p">;</span>
<span class="n">time</span><span class="o">::</span><span class="n">milliseconds</span> <span class="n">freshnessPeriod</span><span class="p">{</span><span class="mi">10000</span><span class="p">};</span>
<span class="kt">size_t</span> <span class="n">maxSegmentSize</span> <span class="o">=</span> <span class="n">MAX_NDN_PACKET_SIZE</span> <span class="o">>></span> <span class="mi">1</span><span class="p">;</span>
<span class="n">optional</span><span class="o"><</span><span class="kt">uint64_t</span><span class="o">></span> <span class="n">segment</span><span class="p">;</span> <span class="c1">// return the requested segment to app so that it can answer the interest</span>
<span class="kt">bool</span> <span class="n">removeStale</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span> <span class="c1">// Schedule removal of data from IMS after freshnessPeriod if true</span>
<span class="p">};</span>
<span class="k">class</span> <span class="nc">StreamHandle</span> <span class="o">:</span> <span class="k">public</span> <span class="n">ndn</span><span class="o">::</span><span class="n">detail</span><span class="o">::</span><span class="n">CancelHandle</span>
<span class="p">{</span>
<span class="nl">public:</span>
<span class="n">StreamHandle</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="kt">int</span> <span class="n">fd</span><span class="p">,</span> <span class="k">const</span> <span class="n">Segmenter</span><span class="o">::</span><span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">)</span>
<span class="o">:</span> <span class="n">CancelHandle</span><span class="p">([</span><span class="k">this</span><span class="p">]</span> <span class="p">()</span> <span class="p">{</span> <span class="n">stop</span><span class="p">();</span> <span class="p">})</span>
<span class="p">,</span> <span class="n">m_socket</span><span class="p">(</span><span class="n">ioService</span><span class="p">,</span> <span class="n">fd</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_options</span><span class="p">(</span><span class="n">options</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">start</span><span class="p">();</span>
<span class="p">}</span>
<span class="cm">/*
Start async_read_some to read up to maxSegmentSize
-> If call back has an error, emit onError and stop.
-> If no more bytes to read, emit onComplete and stop
-> If bytes read, insert into IMS, increment m_numSegmentsStored,
emit onSegmentInserted, call start again
*/</span>
<span class="kt">void</span>
<span class="n">start</span><span class="p">();</span>
<span class="cm">/*
Emit onError and cancel async operations, close, and release fd
*/</span>
<span class="kt">void</span>
<span class="n">stop</span><span class="p">();</span>
<span class="kt">size_t</span>
<span class="n">getNumSegmentsStored</span><span class="p">();</span>
<span class="nl">public:</span>
<span class="n">Signal</span><span class="o"><</span><span class="n">StreamHandle</span><span class="p">,</span> <span class="n">Data</span><span class="o">></span> <span class="n">onSegmentInserted</span><span class="p">;</span>
<span class="n">Signal</span><span class="o"><</span><span class="n">StreamHandle</span><span class="p">,</span> <span class="n">boost</span><span class="o">::</span><span class="n">system</span><span class="o">::</span><span class="n">error_code</span><span class="o">></span> <span class="n">onError</span><span class="p">;</span>
<span class="n">Signal</span><span class="o"><</span><span class="n">StreamHandle</span><span class="o">></span> <span class="n">onComplete</span><span class="p">;</span>
<span class="nl">private:</span>
<span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">posix</span><span class="o">::</span><span class="n">stream_descriptor</span> <span class="n">m_socket</span><span class="p">;</span>
<span class="n">Options</span><span class="o">&</span> <span class="n">m_options</span><span class="p">;</span>
<span class="kt">size_t</span> <span class="n">m_numSegmentsStored</span><span class="p">;</span>
<span class="p">};</span>
<span class="cm">/*
Persistent does not have a limit
*/</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span> <span class="o">=</span> <span class="n">IMS</span><span class="p">,</span>
<span class="k">typename</span> <span class="o">=</span> <span class="k">typename</span> <span class="nc">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">InMemoryStoragePersistent</span><span class="p">></span><span class="o">::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span><span class="o">></span>
<span class="n">Segmenter</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="n">KeyChain</span><span class="o">&</span> <span class="n">keyChain</span><span class="p">)</span>
<span class="o">:</span> <span class="n">m_scheduler</span><span class="p">(</span><span class="n">ioService</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_keyChain</span><span class="p">(</span><span class="n">keyChain</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_ioService</span><span class="p">(</span><span class="n">ioService</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ims</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span><span class="p">(</span><span class="n">ioService</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span> <span class="o">=</span> <span class="n">IMS</span><span class="p">,</span>
<span class="k">typename</span> <span class="o">=</span> <span class="k">typename</span> <span class="nc">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><!</span><span class="n">std</span><span class="o">::</span><span class="n">is_same</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">InMemoryStoragePersistent</span><span class="p">></span><span class="o">::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span><span class="o">></span>
<span class="n">Segmenter</span><span class="p">(</span><span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">ioService</span><span class="p">,</span> <span class="n">KeyChain</span><span class="o">&</span> <span class="n">keyChain</span><span class="p">,</span>
<span class="kt">size_t</span> <span class="n">limit</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">numeric_limits</span><span class="o"><</span><span class="kt">size_t</span><span class="o">>::</span><span class="n">max</span><span class="p">())</span>
<span class="o">:</span> <span class="n">m_scheduler</span><span class="p">(</span><span class="n">ioService</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_keyChain</span><span class="p">(</span><span class="n">keyChain</span><span class="p">)</span>
<span class="p">,</span> <span class="n">m_ioService</span><span class="p">(</span><span class="n">ioService</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">ims</span> <span class="o">=</span> <span class="n">std</span><span class="o">::</span><span class="n">make_unique</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span><span class="p">(</span><span class="n">ioService</span><span class="p">,</span> <span class="n">limit</span><span class="p">);</span>
<span class="p">}</span>
<span class="cm">/*
* Return the Data corresponding to options.segment
*/</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="k">const</span> <span class="kt">uint8_t</span><span class="o">*</span> <span class="n">buffer</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">bufferSize</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">Data</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="n">Block</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="cm">/*
* Create Handle for stream
* -> Insert into m_streams, capture iterator
* -> onError or onComplete callback, remove the handler from m_streams
* -> Return the handler
*
* Takes in a file descriptor to pass to boost::asio::posix::stream_descriptor
* (This change because I am not sure how to pass std::istream to stream_descriptor)
*/</span>
<span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">StreamHandle</span><span class="o">>&</span>
<span class="n">store</span><span class="p">(</span><span class="k">const</span> <span class="n">Name</span><span class="o">&</span> <span class="n">dataName</span><span class="p">,</span> <span class="kt">int</span> <span class="n">fd</span><span class="p">,</span> <span class="k">const</span> <span class="n">Options</span><span class="o">&</span> <span class="n">options</span><span class="p">);</span>
<span class="nl">public:</span>
<span class="n">std</span><span class="o">::</span><span class="n">unique_ptr</span><span class="o"><</span><span class="n">IMS</span><span class="o">></span> <span class="n">ims</span><span class="p">;</span>
<span class="nl">private:</span>
<span class="n">Scheduler</span> <span class="n">m_scheduler</span><span class="p">;</span>
<span class="n">KeyChain</span><span class="o">&</span> <span class="n">m_keyChain</span><span class="p">;</span>
<span class="n">boost</span><span class="o">::</span><span class="n">asio</span><span class="o">::</span><span class="n">io_service</span><span class="o">&</span> <span class="n">m_ioService</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="n">set</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">shared_ptr</span><span class="o"><</span><span class="n">StreamHandle</span><span class="o">>></span> <span class="n">m_streams</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">}</span> <span class="c1">// namespace util</span>
<span class="p">}</span> <span class="c1">// namespace ndn</span>
</code></pre> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=266382020-04-29T04:13:25ZAshlesh Gawande
<ul></ul><p>After discussion with Jeremy, seems easier to just take in a reference to IMS instead.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=266732020-05-18T13:46:56ZDavide Pesavento
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Target version</strong> set to <i>0.8.0</i></li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=266742020-05-18T14:06:34ZDavide Pesavento
<ul></ul><p>Junxiao Shi wrote:</p>
<blockquote>
<p>To store a (potentially infinite) stream, using a secondary thread would open up a lot of thread safety issues. Instead:</p>
<ul>
<li>Maintain a list of in-progress stream store operations in the segmenter.</li>
<li>Triggered by the associated <code>boost::asio::io_context</code>, read from stream and save into IMS. [...]</li>
<li>The <code>store</code> function should return a <em>handle</em> of the in-progress stream store operation, which has [...]</li>
<li>An in-progress stream store operation should be removed from the aforementioned list when it's completed, canceled, or errored.</li>
</ul>
</blockquote>
<p>This makes no sense at all, on a various different levels.<br>
First of all, as usual, you're needlessly complicating the issue and vastly overengineering the solution. We can easily solve 80% of the current use cases with 20% of the work. Let's do that instead of conjecturing and trying to solve theoretical problems on which we don't have enough application-driven experience.<br>
Secondly, this approach is technically flawed. Asio does not guarantee that async operations will behave correctly on file descriptors that refer to regular files (as opposed to sockets). And even if it does happen to work on a particular platform, (1) it's not portable and (2) file I/O is actually performed immediately, not asynchronously. Asio is a networking library, don't try to use it in other use cases.</p>
<p>I strongly recommend to keep it simple and use a <code>std::istream</code>. Optionally you can support an <em>eager</em> mode, where the entire stream until EOF is consumed and Data packets are produced immediately, and a <em>lazy</em> mode, where no Data packets are initially produced and the app has to explicitly call another function to generate the next <code>n</code> segments up to a given segment number.</p>
<p>The <code>istream</code> interface is obviously in addition to the interface(s) taking a buffer (or array of buffers) that have already been proposed.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=266762020-05-18T15:00:25ZJunxiao Shi
<ul></ul><p>I never said the stream is a socket. It could be (a wrapper of) <code>std::istream</code>.<br>
I don't know enough about Asio. It's not meant to be operated like a socket, but <code>loop()</code> is simply a function that is invoked frequently via a timer.</p>
<p>The equivalent functionality in NDNts is here:<br>
<a href="https://github.com/yoursunny/NDNts/tree/b431e05dcff54b21b272c9b8a637cc74bdca81ad/packages/segmented-object/src/serve/chunk-source">https://github.com/yoursunny/NDNts/tree/b431e05dcff54b21b272c9b8a637cc74bdca81ad/packages/segmented-object/src/serve/chunk-source</a><br>
Start looking at <code>BufferChunkSource</code> (input from a byte array) and <code>StreamChunkSource</code> (input from an infinite stream) and <code>FileChunkSource</code> (input from a file with known size), as well as their base classes.<br>
They all implement <code>ChunkSource</code> interface. If <code>getChunk</code> is implemented, it's lazy; otherwise, <code>listChunks</code> will be used, it's eager.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=267022020-05-26T18:17:47ZAshlesh Gawande
<ul></ul><p>What ndn-cxx applications require SegmentPublisher to be implemented at this point of time?</p>
<ul>
<li>For <a class="issue tracker-2 status-1 priority-2 priority-default" title="Feature: Dispatcher: use SegmentPublisher in StatusDatasetContext (New)" href="https://redmine.named-data.net/issues/4699">#4699</a>, <code>StatusDatasetContext</code> can be implemented with Segmenter instead of replacing.</li>
<li>NLSR, PSync would be using the Segmenter directly.</li>
<li>ndnputchunks can also just replace custom in-memory storage solution with the Segmenter.</li>
</ul>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=267032020-05-27T06:20:42ZJunxiao Shi
<ul></ul><p>All three applications should use <code>SegmentPublisher</code> to replace their custom buffering behavior, not just <code>Segmenter</code>. More importantly, this allows new developers to more easily create new applications.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=269062020-06-29T23:44:25ZDavide Pesavento
<ul><li><strong>Tags</strong> set to <i>API</i></li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=273752021-07-07T15:39:21ZDavide Pesavento
<ul><li><strong>Target version</strong> changed from <i>0.8.0</i> to <i>0.9.0</i></li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=273822021-08-05T18:01:28ZDavide Pesavento
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/27382/diff?detail_id=24171">diff</a>)</li><li><strong>Assignee</strong> deleted (<del><i>Ashlesh Gawande</i></del>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=277602022-11-04T20:20:24ZDavide Pesavento
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-3 priority-high3 closed" href="/issues/5247">Feature #5247</a>: Segmenter: simple content segmentation helper</i> added</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=277642022-11-04T20:22:42ZDavide Pesavento
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-3 status-5 priority-2 priority-default closed" href="/issues/4702">Task #4702</a>: ndnputchunks: use Segmenter class</i>)</li></ul> ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=277682022-11-04T20:50:12ZDavide Pesavento
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>New</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Target version</strong> deleted (<del><i>0.9.0</i></del>)</li></ul><p>I'm splitting this task and implementing the simpler cases (finite object, eager creation of all Data packets in one shot) in <a class="issue tracker-2 status-5 priority-3 priority-high3 closed" title="Feature: Segmenter: simple content segmentation helper (Closed)" href="https://redmine.named-data.net/issues/5247">#5247</a>.</p>
ndn-cxx - Feature #4696: SegmentPublisher: segmentation utilityhttps://redmine.named-data.net/issues/4696?journal_id=280022023-08-14T11:02:05ZJunxiao Shi
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-2 status-5 priority-2 priority-default closed" href="/issues/4791">Feature #4791</a>: Use SegmentPublisher utility from ndn-cxx to publish LSA data</i>)</li></ul>