NFD

What is the motivation for silent reject?

Each ControlCommand may define its own reply format when authentication fails, and common choices are included in enum RejectReply, including silently dropping the request.

What is the motivation for silent reject?

Silently dropping the request is useful to conceal the existence of a service from unauthorized requesters.

Why is makeAcceptAllAuthorization() a public ManagementDispatch method? I would expect this to be entirely taken care of by the validator.

Why is makeAcceptAllAuthorization() a public ManagementDispatch method? I would expect this to be entirely taken care of by the validator.

makeAcceptAllAuthorization() is a convenient method that returns an Authorization which accepts everything.
It's useful in test case, or outside of NFD.

ManagementValidator::validate is an implementation for Authorization that is normally used inside NFD.

Note that ManagementValidator class is unrelated to ndn::Validator in ndn-cxx.
It has access to ControlParameters, while ndn::Validator doesn't have such information.

Clarification: ManagementDispatcher has a namespace of ndn. Should it live in the cxx library or is this a typo?

It's intentional. See the last goal: basic mechanisms can be reused inside and outside NFD.

Looking at the dispatcher, I noticed there are a couple of cases where the "add" operation requires a ReplyEncrypt callback while the Handler type requires Reply.

• addControlCommand and ControlCommandHandler
• addStatusDataset and StatusDatasetHandler

Is this intentional or should ReplyEncrypt type always be used? I'm not sure I understand the Reply type if null encryption is provided.

Also, StatusDatasetHandlers expect an End type, but addStatusDataset does take one as an argument. Should there be an End argument?

ReplyEncrypt differs from Reply.

ReplyEncrypt type represents an algorithm that is used to encrypt replies. A ReplyEncrypt instance is reused across all management requests.

Reply type represents an action that makes use of the algorithm provided by ReplyEncrypt. A Reply instance is specific to one request.

End type represents an action that represents all relies has been passed to Reply. An End instance is specific to one request.

In impl, Reply and End are typically bound functions that know about the request, so that ControlCommandHandler and StatusDatasetHandler don't need to pass the request as argument to Reply and End.

Can you please clarify what End does in the implementation? As you said, I see it being called after all replies have been sent in provided handleListRequest, but the comments don't describe its function (I suppose it doesn't matter from the caller's perspective). Is it a placeholder no-op?

Can you please clarify what End does in the implementation? As you said, I see it being called after all replies have been sent in provided handleListRequest, but the comments don't describe its function (I suppose it doesn't matter from the caller's perspective). Is it a placeholder no-op?

End allows a StatusDatasetHandler to tell dispatcher that all segments have been sent.

A possible implementation is:

1. temporary store all Data packets passed to Reply
2. upon End call, if Reply is called zero time, generate one Data packet and send to requester
3. upon End call, if Reply is called one or more times, set FinalBlockId on all Data packets, and send all Data packets

I'm thinking about a different design: Reply passed to StatusDatasetHandler shouldn't accept Data. It should accept WireEncodable concept.

This seems more useful than note-3. What do you think?

Junxiao Shi wrote:

End allows a StatusDatasetHandler to tell dispatcher that all segments have been sent.

A possible implementation is:

1. temporary store all Data packets passed to Reply
2. upon End call, if Reply is called zero time, generate one Data packet and send to requester
3. upon End call, if Reply is called one or more times, set FinalBlockId on all Data packets, and send all Data packets

My thinking was that the SegmentStream in mentioned in note 3's dataset handler example should've taken care of FinalBlockIds. I'm not sure if there is a SegmentStream class, but core/SegmentPublisher does (though only on the last Data).

I'm concerned about scope creep. I feel the dispatcher should just dispatch and this is probably what caused my confusion about End. Why should the dispatcher care about the handling being finished or not? It has already dispatched the Interest.

I'm thinking about a different design: Reply passed to StatusDatasetHandler shouldn't accept Data. It should accept WireEncodable concept.

This seems more useful than note-3. What do you think?

I don't think the dispatcher should be involved with packetization unless it's generating an error response. This sounds like the dispatcher is absorbing core/SegmentPublisher and its derived classes.

I don't think the dispatcher should be involved with packetization unless it's generating an error response. This sounds like the dispatcher is absorbing core/SegmentPublisher and its derived classes.

Yes, ManagementDispatcher is the only entrypoint of management system. It is absorbing SegmentPublisher. SegmentPublisher may be used internally by ManagementDispatcher.

It's necessary to let ManagementDispatcher handle packetization because encryption can change the length of packets.

Similarly, ManagementDispatcher is also absorbing NotificationPublisher.

Junxiao Shi wrote:
It's necessary to let ManagementDispatcher handle packetization because encryption can change the length of packets.

That's a good point. I'm assuming this means dataset results need to be encrypted before packetization (as opposed to chunking conservatively).

So should both Reply and ReplyEncrypt take a WireEncodable concept? Your comment in note-14 sounds like you're thinking of having different Reply types for commands and datasets.

No. Reply and ReplyEncrypt cannot take WireEncodable because it's a concept and the only way to accept a concept is to make a template. Instead,

  /** \brief represents an asynchronous callback to be invoked upon completion of ReplyEncrypt
   */
  typedef std::function<void(bool hasError, const std::vector<shared_ptr<Data>>& result)> ReplyEncryptCallback;

  /** \brief indicates whether segment number should be appended to prefix when constructing a Data packet in ReplyEncrypt
   */
  enum ReplySegmentNumber {
    /** \brief segment number is always added
     */
    RSN_REQUIRED,
    /** \brief segment number is added if encrypted blocks cannot fit in one Data packet
     */
    RSN_OPTIONAL,
    /** \brief segment number cannot be added; error if encrypted blocks cannot fit in one Data packet
     */
    RSN_FORBIDDEN
  };

  /** \brief a function to process reply Data
   *
   *  Procedure:
   *  1. concatenate blocks into a single buffer
   *  2. encrypt this buffer for the requester identified by identity
   *  3. slice the buffer into one or more Data packets, where Name is prefix plus segment number (if needed according to rsn)
   *  4. set FinalBlockId on at least the last Data packet (if segment number is added)
   *  5. sign these Data packets and invoke cb
   */
  typedef std::function<void(const Name& prefix, const Interest& interest, const Name& identity,
                             ReplySegmentNumber rsn, const std::vector<Block>& blocks,
                             ReplyEncryptCallback cb)> ReplyEncrypt;

  /** \brief a function to be called to send reply
   *  \param block an unencrypted Block, which will be appended to payload of reply Data before encryption
   */
  typedef std::function<void(Block block)> Reply;

It's necessary to let ReplyEncrypt handle encryption + segmentation + signing, because the result of encryption affects how segmentation is performed, and parameters used during encryption may need to appear in packet headers (such as in MetaInfo).

The Reply passed to ControlCommandHandler should expect to be called once with ControlResponse block.
After that, the block will be passed to ReplyEncrypt with RSN_FORBIDDEN.

The Reply passed to StatusDatasetHandler should expected to be called zero or more times with any type of block, followed by exactly one call to End.
After that, all collected blocks will be passed to ReplyEncrypt with RSN_REQUIRED.

What should happen if hasError is true? My initial thought was a 5xx server error should be generated and ReplyEncrypt'd, but that could potentially result in an infinite loop (until stack exhaustion). Is it ok for a 5xx response to be in the clear or should the dispatcher just do nothing?

What should happen if hasError is true?

Log an error and don't respond.

ReplyEncrypt would return error only if RSN_FORBIDDEN is specified but block is too long.
This condition is met only if a ControlResponse is too long, which violates the design assumption of ControlCommand.

Junxiao Shi wrote:

  /** \brief a function to process reply Data
   *
   *  Procedure:
   *  1. concatenate blocks into a single buffer
   *  2. encrypt this buffer for the requester identified by identity
   *  3. slice the buffer into one or more Data packets, where Name is prefix plus segment number (if needed according to rsn)
   *  4. set FinalBlockId on at least the last Data packet (if segment number is added)
   *  5. sign these Data packets and invoke cb
   */
  typedef std::function<void(const Name& prefix, const Interest& interest, const Name& identity,
                             ReplySegmentNumber rsn, const std::vector<Block>& blocks,
                             ReplyEncryptCallback cb)> ReplyEncrypt;

It just occurred to me that ReplyEncrypt is now responsible for appending a version (if any) to the response name. Also, the initial segment number is increasing in the case of notification streams. I could just add appropriate parameters to ReplyEncrypt, but this is getting very complex. Any suggestions?

typedef std::function<void(Block notification)> PostNotification;

PostNotification needs an Interest to call ReplyEncrypt. However, addNotificationStream returns a PostNotification, so the Interest cannot be provided at dispatch time like control command and datasets. I think the Interest has to be provided by the PostNotification's caller.

There's no way for PostNotification's caller to provide an Interest. A notification is generated whenever something happens, regardless of whether there's an Interest requesting such notification.

According to Notification subscriber operations, the Interest can be assumed to be: Name=ndn:/localhost/nfd/faces/events, ChildSelector=rightmost.

The implementation of accept all authorization basically needs to determine the identity of the command's signer and pass that name to the accept continuation. Does the library provide any abstractions for extracting the identity of a signed Interest?

Questions about library shall be sent to ndn-lib mailing list. I have entered the question on your behalf http://www.lists.cs.ucla.edu/pipermail/ndn-lib/2015-February/000269.html. Please wait for an answer.

Thanks for relaying the question. I've been thinking about the problem #24 a bit more and it has caused me to wonder about the dispatcher passing an identity everywhere. Clearly, identity is an important question for authorization: we need an accept/reject decision. However, does the dispatcher really care about the Interest sender's identity beyond that?

For example, why does ReplyEncrypt need the sender's identity to be provided by the dispatcher? I don't think we can do per identity encryption -- response packetization is performed later by the dispatcher and NFD management protocols do not define a way to name responses/datasets/whatever in an identity specific manner.

response packetization is performed later by the dispatcher

No. See note-18.

NFD management protocols do not define a way to name responses/datasets/whatever in an identity specific manner.

ControlCommand is identity specific.

StatusDataset and Notification are currently not identity specific and therefore cannot be encrypted in identity specific manner.

The implementation of accept all authorization basically needs to determine the identity of the command's signer and pass that name to the accept continuation. Does the library provide any abstractions for extracting the identity of a signed Interest?

As answered in http://www.lists.cs.ucla.edu/pipermail/ndn-lib/2015-March/000286.html, use NDN RegEx.

I have two questions about the design:

1/ ndn::ManagementDispatcher uses ndn::Face (to support usage outside NFD, I guess), while NFD::Forwarder uses nfd::Face. When a ndn::ManagementDispatcher instance is going to be used inside NFD, I should “connect” such a ndn::Face (or instance of a derived class) to a nfd::Face (or instance of a derived class). Can I do something like this:

 1) connect ndn::Transport::m_receiveCallback with nfd::Face::onReceiveInterest and nfd::Face::onReceiveData
 2) redirect ndn::Transport::send() to nfd::Face::sendInterest() and nfd::Face::sendData()

If there are smarter methods, could you please give me some hints?

2/ ndn::ManagementDispatcher::EncryptReply should also handler segmentation. But I only find one SegmentPublisher class, which is under the nfd namespace.

Shall I use nfd::SegmentPublisher directly, or move that class to ndn namespace, or implement the core functionality of that class in ndn::ManagementDispatcher?

After doing implementation, I have further questions about the design:

1/ As we intend to reuse the basic mechanism outside NFD, why the dispatcher should know nfd-control-parameters?
I think only nfd-managers should know the detail about nfd-control-parameters, and can extract these parameters from the Interest. What the dispatcher needs to do is passing the Interest to the authorization and command handler. Then nfd-authorizations and nfd-handlers can extract the parameters if required. And this process can be done in the nfd-manager-base class.

2/ AcceptContinuation has a identity parameter which is passed to the ReplyEncrypt finally. But why RejectContinuation has not such a parameter? Does RejectContinuation require to send out replies? Should the reject reply also be encrypted?

3/ Why the ReplyEncrypt needs an identity parameter? for signing Data?

4/ Why the ReplyEncrypt should be responsible to segmenting and signing? Can we have the dispatcher deals with these issues, leaving the ReplyEncrypt process encryption only (take blocks as input and outputs an encrypted encoding buffer). Then the ReplyEncryptCallBack provided by the dispatcher takes the encoding buffer as input and deals with the rest issues.

5/ Why the add- functions need a std::vector parameter as the relative prefix (e.g. /face/events), but not a real prefix (e.g. /localhost/nfd/face/events)?
Since the dispatcher mechanism may be used outside NFD, we can not guess the top-level prefixes as /localhost/nfd or /localhop/nfd. I think we can ask the manager to provide the real prefix. But the top-level prefixes for nfd-managers (/localhost/nfd, /localhop/nfd) can be provided by the nfd-manager-base class.

6/ What's the intentions of the prefix parameter and the identity parameter of ControlCommandHandler and StatusDatasetHandler?

7/ Also, what's the intentions of the prefix parameter of Authorization and ReplyEncrypt?
you indicates that the prefix parameter of Authorization is a top-level prefix which is used in authorization. But the Authorization is provided by the manager when it calls add- functions, it should already know the top-level prefix.

Thanks for your answers.

I still have the following questions:

1. Does RejectContinuation require to send out replies? Should the reject reply also be encrypted?
   RejectContinuation does take the identity parameter but the ReplyEncrypt does.

2. Should RejectContinuation take another parameter describes the reason of rejection? I think it's more reasonable to reply both code and reason to the requester.

3. yes it's a question that whether should we perform segmentation before or after encryption. In my opinion, ReplyEncrypt should only do encryption according to some algorithm. Thus, the encryption and corresponding decryption can keep "cleaner" logic that will not be affected by the detail of segmentation and collecting segment packets. To this point, I prefer to perform segmentation after encryption.

4. I know that the dispatcher should only register the top level prefixes to the RIB/FIB. But if we only provide the relative prefix when adding command handler or data status handler, we must keep the relative prefix unique (even with different top level prefixes), because we do not know this relative prefix belongs to which top-level prefix at that time.

5. The Authorization is provided by the manager, I think the manager already knows the top-level prefix so it can directly supplies this parameter.

Answer to note-33:

1. RejectContinuation sends a reply only if act != RejectReply::SILENT.
   See note-31 answer to note-30 item 2 on why RejectContinuation does not take identity parameter.
2. It's intentional for RejectContinuation to not take a parameter to show why authorization has been rejected. Exposing the detailed reason to the client is a security risk.
   For debug purpose, it's okay for the authorization procedure to write the detailed reason to the server log.
3. ReplyEncrypt procedure has defined that encryption should normally happen before segmentation.
4. Yes relPrefixes MUST be unique, and the Dispatcher MAY check this.
5. No, it's necessary to pass the top-level prefix to Authorization. RibManager can register rib/register handler which would be used by all top-level prefixes; this registration is done in one call so there's only one Authorization function. But this Authorization function must use different procedures per top-level prefix.

1 RejectContinuation sends a reply only if act != RejectReply::SILENT.

this only answers the first half of my question. the next half is whether this reply should be encrypted.

3 ReplyEncrypt procedure has defined that encryption should normally happen before segmentation.

I know you have described this before. But my opinion is not only doing encryption before segmentation but also keeping ReplyEncrypt doing encryption only. Other issues like segmentation and signing data are processed by the dispatcher as it's the dispatcher that sends out the reply finally.

Answer to note-35:

1 No, the reply after RejectContinuation is not encrypted.
Encryption algorithm and key are selected after knowing the identity of requester, but the identity isn't known if authorization is rejected.

3 No.
(a) The encryption algorithm may involve segmentation, eg. use a digest over the previous segment as the IV of the current segment.
(b) The encryption algorithm may need to add fields into MetaInfo in order to assist the request in selecting the correct decryption key.

20150612 conference call reviewed revision ad9d9d7104e9ea4e93c1d8c1d398c2cd6d94da93.

We decided:

• We don't see the necessity in encrypting the responses, and don't have a clear understand of how reply encryption could work (especially, how to select an encryption key). Encrypting only the response but not the request is not useful. Thus, in this design, encryption should be dropped completely.
• Authorization shouldn't return identity because there's no universal definition of this concept. Instead, Authorization may return a string that represents the requester, whose meaning is up to the Authorization function.
• Relative prefix should be represented with PartialName (#1962 note-8) instead of vector of Components.

Alex also asked the following:

• It's not obvious to use XCallback cb as asynchronous return.
  • A: This convention is commonly used in JavaScript for asynchronous return, such as NodeJS.
• Why does StatusDatasetHandler needs two functions StatusDatasetAppend and StatusDatasetEnd?
  • A: When enough octets are collected to fill a Data packet, it can be sent without waiting for the whole dataset to be generated.

https://gist.github.com/yoursunny/e3c520a590af86a0bb37/9526c4f9efd396cf8ae81c0acd5ac065a065cc22

updated for note-37

https://gist.github.com/yoursunny/e3c520a590af86a0bb37/582b30991fe08d0e54dfead1cd711cf1a3014c45

This revision addresses Yanbiao's question asked on 20150615 conference call: Dispatcher needs a KeyChain in order to sign reply Data.

Questions:

line_43, line_71, line_92: Bothe Authorization, ValidateParameters and ControlCommandHandler are provided by some manager who wants to register some command handler, and are executed outside the dispatcher. Why Authorization takes a base class pointer (const ControlParameters*) as a parameter but others take a base class reference (const ControlParameters&)? What's the different intentions here?

line_57: ControlParameters is used by Authorization, ValidateParameters and ControlCommandHandler, but who will supply this parameter? I think the dispatcher should extract the ControlParameters from the Interest and then supply this parameter to whoever requires it. However, for localhop or localhost requests (in the format "/localhost or localhop/nfd/moudler/verb/parameters"), we can easily extract the ControlParameters from the Interest.

But how about other cases (the basic mechanism of dispatcher may be used outside nfd)?. There is no general rule to do such a extracting. So, I think there should be some function (like extractFromInterest) in the base class ControlParameters that will be implemented in the derived class for different request formats.

line_63: Why this is "virtual Block wireEncode() const = 0" but not "virtual const Block& wireEncode() const = 0"? Is it a typo or there is some special purpose?

line_75: Shall we define a base class for ControlResponse here, making the ndn::nfd::ControlResponse a derived class?

line_102: Why a Rvalue reference is used here? I think most objects have a wireEncode function that returns a Lvalue reference that can be directly used for appending the status dataset.

line_159: The top-prefix is used for the dispatcher to receive interested Interest, while the relPrefix is used as a key to retrieve the corresponding handler. My question here is how to get the relPrefix from the Interest. Currently, I use LPM to determine the relPrefix. Shall we define some explicit rules, such as putting the relPrefix as one component of the Interest name or restricting the number of its components.

line_207: should the status dataset always be segmented even though all information can fit into one Data packet?

line_217: How to get the next sequence number? Currently, I maintain a variable of last used sequence number for each relPrefix.

Answer to note-41 regarding revision 582b30991fe08d0e54dfead1cd711cf1a3014c45:

Why Authorization takes a base class pointer but others take a base class reference?

Authorization is used by both ControlCommand and StatusDataset.

ControlParameters is only applicable to ControlCommand.
When Authorization is invoked for a StatusDataset request, params is nullptr.

How to extract ControlParameters?

ControlCommand spec requires parameters to appear in the NameComponent at a specific position in the Name; this rule is (assumed to be) universal to all use cases.

addControlCommand function template has a template parameter CP, which gives the type of the parameters.

CP fulfills DefaultConstructible and WireDecodable concept, so that no separate extractFromInterest function is needed.

Why ControlParameters::wireEncode returns Block instead of const Block&?

Returning a const reference requires a ControlParameter subclass to store the Block as a member field.

Returning a value eliminates this requirement, and therefore allows different choices.

Shall we define a base class for ControlResponse here, making the ndn::nfd::ControlResponse a derived class?

There's no necessity for now.

ndn::nfd::ControlResponse can be a typedef of ndn::mgmt::ControlResponse.

Why an r-value reference is accepted in StatusDatasetAppend?

StatusDatasetAppend may choose to store the block into an STL container, and join them together at a later time.

Accepting an r-value reference allows the block to be moved into the container without copying.

how to get the relPrefix from the Interest?

relPrefix = interest.getName().getSubName(topPrefix.size(), 2);

Note that this is incompatible with ForwarderStatus revision 4.
In fact, ForwarderStatus is not even a StatusDataset.
This module needs to be handled outside of the dispatcher.

I'll look into changing that protocol so that it becomes a dataset and has a two-component relPrefix.

Should the status dataset always be segmented even though all information can fit into one Data packet?

Yes. See StatusDataset protocol.

How to get the next sequence number?

Remember the sequence number somewhere.

Why Authorization takes a base class pointer but others take a base class reference?

Authorization is used by both ControlCommand and StatusDataset.
ControlParameters is only applicable to ControlCommand. When Authorization is invoked for a StatusDataset request, params is nullptr.

My question is not why Authorization takes a pointer, but why there are differences. According to your comments, can I deduce that it's better to use reference, but have to use pointer for authorization for the above reason.

How to extract ControlParameters?

ControlCommand spec requires parameters to appear in the NameComponent at a specific position in the Name; this rule is (assumed to be) universal to all use cases.

Yes, I know this specification and did utilize it in previous and current implementations. My concern is that this
specification is a NFD specification, why should we force other apps (especially that comes in future) obey this NFD-specified rule. Since we do have a ndn::nfd::ControlParameters class, why not specify an interface in the base class to extract parameters from the Interest and implement it in ndn::nfd::ControlParameters? Firstly, it's easy to do this and I do not see any big harmful. Secondly, in this way, we can not only support current NFD-specified format but also other approved formats (in the future). If other app or a future app want to use the basic mechanism of the dispatcher, what's required is just implementing the extract interface in some derived class.

addControlCommand function template has a template parameter CP, which gives the type of the parameters.
CP fulfills DefaultConstructible and WireDecodable concept, so that no separate extractFromInterest function is needed.

Yes, all CPs derived from the ndn::nfd::ControlParameters know how to extract parameters from Interest based on the NFD-specified format. But my question is why we should force others obey the NFD-specified rule when we use the basic mechanism outside NFD.

Why ControlParameters::wireEncode returns Block instead of const Block&?

Returning a const reference requires a ControlParameter subclass to store the Block as a member field.
Returning a value eliminates this requirement, and therefore allows different choices.

Yes, but what is used in ndn::nfd::ControlParameters is const reference. My concern is not which one is better but why they are different. If we do not modify one of them, we must implement a similar interface that returns value in ndn::nfd::ControlParameters to prevent it from being a abstract class.

Shall we define a base class for ControlResponse here, making the ndn::nfd::ControlResponse a derived class?

There's no necessity for now.
ndn::nfd::ControlResponse can be a typedef of ndn::mgmt::ControlResponse.

Except the benefit for future use, there are two points in my mind for the necessity.

1. Since the dispatcher will only use part of the features of ndn::nfd::ControlResponse, I think it's better to have a separate class here and make ndn::nfd::ControlResponse inherits from this class.
2. If we make a typedef, I will have the concern again: why we restrict the dispatcher to some NFD specifications.

Why an r-value reference is accepted in StatusDatasetAppend?

StatusDatasetAppend may choose to store the block into an STL container, and join them together at a later time.
Accepting an r-value reference allows the block to be moved into the container without copying.

I think it's better to have the StatusDatasetAppend send out a packet whenever it's proper, rather than waiting all blocks ready and then join them together. For this purpose, I fulfill all received blocks into an EncodingBuffer, and generate a Data packet when it will exceed the size limit if adding the next block. In this way, I do not see the directly benefit of using r-value. Besides, if we chose r-value here, we can not directly use all wireEncode functions, which in my mind is a direct disadvantage.

how to get the relPrefix from the Interest?

relPrefix = interest.getName().getSubName(topPrefix.size(), 2);"

Where specifies that the relPrefix have 2 components only (I mean without any NFD specification)? My question is whether should we have such a rule but not how to do if we already have such a rule. If we should make it a rule that relPrefix can only have 2 components. I think we should specify this rule in the definition of dispatcher (e.g. dispatcher.hpp).

How to get the next sequence number?

Remember the sequence number somewhere.

Maybe my question was not clear. Actually, my question is should we have separate sequence number for different relPrefixes. I told you that I remember a sequence number for each relPrefix in my current implementation, and asked you whether this is correct.

Answer to note-43:

What's difference between const ControlParameter* and const ControlParameters&?

A parameter is pointer type when it can be null.

A parameter is reference type when it must not be null; nullptr is not a valid value for a reference type.

Why not specify an interface in the ndn::mgmt::ControlParameters class to extract parameters from the Interest and implement it in ndn::nfd::ControlParameters?

There's no use case.

NFD, former NRD, repo-ng, and NLSR all places the encoded parameters at the same position.

We could adopt the proposal in this question when a use case appears.

If ndn::mgmt::ControlParameters::wireEncode returns Block, how to prevent ndn::nfd::ControlParameters from being abstract?

Change ndn::nfd::ControlParameters::wireEncode to return Block.

Shall we define a base class for ControlResponse here, making the ndn::nfd::ControlResponse a derived class?

No. There's no use case.

ndn::mgmt::ControlResponse::setBody accepts a block, not ndn::nfd::ControlParameters, so that other applications can use it as well.

When a use case arises, we can change this part.

Why an r-value reference is accepted in StatusDatasetAppend? We cannot use wireEncode functions.

See note-42 for the benefits for accepting r-value.

To use a wireEncode function that returns a l-value reference, write as:

append(Block(something.wireEncode()));

How to get the relPrefix from the Interest?

note-42 answer extracts 2 components, but this appears to be incompatible with NLSR Management.

One solution is not to extract an exact relPrefix from the Interest, but try a few different lengths:

PartialName relName = interest.getName().getSubName(topPrefix.size());
for (size_t relPrefixSize = 1; relPrefixSize < relName.size(); ++relPrefixSize) {
  PartialName relPrefix = relName.getPrefix(relPrefixSize);
  auto i = dispatchTable.find(relPrefix);
  if (i != dispatchTable.end()) {
    ..
    break;
  }
}

This is effectively a "shortest prefix match".

It's unnecessary to use longest prefix match, because we can reasonably assume there's no overlapping in relPrefixes.

Is there one sequence number per notification stream, or one sequence number for all notification streams?

From Notification protocol:

The sequence numbers of notifications in the same stream should be consecutive and increasing.

To satisfy the "consecutive" requirement, there should be one sequence number per notification stream.

What's difference between const ControlParameter* and const ControlParameters&?

A parameter is pointer type when it can be null.

A parameter is reference type when it must not be null; nullptr is not a valid value for a reference type.

I need to improve my expression. My question is not WHAT is the difference between pointer and reference, but WHY there are differences between Authorization, ValidateParameters and ControlCommandHandler when they require ControlParameters. Can I can deduce that reference is a better choice in the above functions but we have to use pointer in the Authorization?

If ndn::mgmt::ControlParameters::wireEncode returns Block, how to prevent ndn::nfd::ControlParameters from being abstract?

Change ndn::nfd::ControlParameters::wireEncode to return Block.

Yes I know we can do this. But if you think the original design of ndn::nfd::ControlParamters is not proper now (should return Block instead of a reference), I think it's better to set up a new task to handler this improvement (from reference to value). So that all other apps utilizes ndn::nfd::ControlParamters will know this modification and may change their codes if required.

How to get the relPrefix from the Interest?

note-42 answer extracts 2 components, but this appears to be incompatible with NLSR Management.

One solution is not to extract an exact relPrefix from the Interest, but try a few different lengths:

PartialName relName = interest.getName().getSubName(topPrefix.size());
for (size_t relPrefixSize = 1; relPrefixSize < relName.size(); ++relPrefixSize) {
  PartialName relPrefix = relName.getPrefix(relPrefixSize);
  auto i = dispatchTable.find(relPrefix);
  if (i != dispatchTable.end()) {
    ..
    break;
  }
}

This is effectively a "shortest prefix match".

It's unnecessary to use longest prefix match, because we can reasonably assume there's no overlapping in relPrefixes.

My concern is not how to do if we have above rule and assumption, but whether we should make the above rule (2 components) and assumption (no overlapping). And once we chose to have that rule and assumption, we should clarify both of them in the definition of the dispatcher.

Answer to note-45:

Why there are differences between Authorization, ValidateParameters and ControlCommandHandler when they require ControlParameters?

As explained in note-44, there's a semantics difference between a pointer and a reference.

Authorization accepts const ControlParameters* to indicate that it can take nullptr;
ValidateParameters and ControlCommandHandler accpet const ControlParameter& to indicate that they require the argument to be not null.

Does changing ndn::nfd::ControlParameters::wireEncode to return Block needs a new task?

No. It's unlikely for anything to break.

Should we make the "2 components" rule and "no overlapping" assumption?

No for the "2 components" rule, because ForwarderStatus does not conform to this rule.
See note-44 for an algorithm that does not need this rule.

Yes for "no overlapping" assumption, because there's no use case that violates this assumption.

one more question:

registerTopPrefix function must return the RegisteredPrefixId*, so I have to call some function of the Face that calls Face::Impl::registerPrefix, which will call nfd::Controller::start to register the prefix finally. As a result, we must have the RibManager works before we can register any top-level prefix to the dispatcher. But if we can not register a top-level prefix to the dispatcher, how can RibManager receive the registration commands?

If you just want to allow the registered top-level prefix be unregistered later. Why not use InterestFilterId* instead of RegisteredPrefixId*. We can call Face::setInterestFilter to just insert a record into Face::Impl::m_interestFilterTable without any FIB/RIB registration, and can use its return value to unset the filter later.

Problem in note-49 is solved in revison c6f76cada262079388578840814866d71874b1f4.

The solution is to expose processInterest, so that an alternate prefix registration method (FIB in NFD Management, setInterestFilter in NFD RIB) can be used instead of registerTopPrefix which calls registerPrefix.

However, I suddenly realized another serious problem: the design doesn't fulfill the goal of "each Interest is dispatched only once", because an Interest is dispatched once in Face's InterestFilter table, and again in the dispatcher.

I'll think about changing the design so that every Interest is dispatched only once, in the InterestFilter table.

Yes, I thought about this issue before. In my previous implementation, I just put the top-level prefix plus the relPrefix into the interest-filter table so that every interest will only be dispatched once.

But there may be another issue. I guess the top-level prefixes may not be too many, but there may be a large number of relPrefixfes (each StatusDatasetHandler/CommandHandler will require an unique relPrefix). I'm afraid that inserting too much things in the interest-filter table will affect the efficiency.

Besides, I did not see the interest-table support "shortest prefix matching"? But we decided to do "shortest prefix matching" to determine the relPrefix.

I'm afraid that inserting too much things in the interest-filter table will affect the efficiency.

Efficiency is not a concern of Dispatcher. If there's a bottleneck, it will affect every application, and should be improved in Face.

I did not see the interest-filter table support "shortest prefix matching"

"shortest prefix matching" is a simplification. Any matching algorithm will work, because relPrefixes are non-overlapping.

https://gist.github.com/yoursunny/e3c520a590af86a0bb37/9a1ccee0ed8a221c2d635b959c4fc44d661ab4d4 fulfills note-50.

Some questions about the design shown in note-53.

1. the precondition for addControlCommand, addStatusDataset and addNotificationStream is that there is no top-level prefix added. So, once a top-level prefix has been added, we can not add any control command, status dataset and notification stream any more? Why we should we setup such a restriction?

2. you said that this design will work only for only one top-level prefix. it seems that there is assumption that this design is for exactly one top-level prefix. But line 101 is for the scenario that there are two or more top-level prefixes.

3. line 182 confuses me a lot. I can not guess your intention there by saying "\note In most cases, \p registerTopPrefix should be used instead". Do you mean the addTopPrefix instead? Even though, processInterest is called to dispatch the incoming interest, why addTopPrefix (it should be involved to configure the dispatcher) will be involved here.

4. According to the new design, the incoming interest can not be dispatched to the right handler until the top-level prefix is added (only on that time, some prefix(es) will be registered to the FIB (top-level prefix) or the interest-filter table (full prefixes)). In another word, even after some module calls addControlComamnd to register a command handler, the registered handler can not receive any interest until a top-level prefix is registered.

5. Why top-level prefix is related to addNotificationStream. Top-level prefix is used to help dispatcher the incoming interest to the right handler (command handler or status dataset handler). But for notification stream, when some module calls addNotificationStream, it will a get a postNotification function back, and will then call this function to publish notification when required. In this process, no interest is dispatched.

6. What's the challenges or difficulties of designing a mechanism to support two or more top-level prefixes now?

Answer to note-54:

1. This precondition can simplify implementation.

* Having this precondition allows the following simplification:

    * `addControlCommand`, `addStatusDataset`, and `addNotificationStream` only need to record the relPrefix internally.
    * `addTopPrefix` is the only place that invokes `face.registerPrefix` and `face.setInterestFilter`

* Not having this precondition causes the following complication:

    * `addControlCommand` and `addStatusDataset` would have to enumerate existing top-level prefixes, and invoke `face.setInterestFilter`.

* This precondition is reasonable, because what ControlCommands, StatusDatasets, and NotificationStreams a program have is hard coded, and there's no use case for runtime changes.

1. "This design works well only when there's exactly one top-level prefix." applies to NotificationStream only.
   See also answer 5.

2. processInterest function should be deleted. InterestFilter installed into the face should dispatch directly to a bound function of a relevant ControlCommand or StatusDataset.

3. Yes.

4. NotificationStream needs exactly one top-level prefix in order to construct the Name of a notification packet.

   For example, topPrefix "ndn:/localhost/nfd" + relPrefix "faces/events" + sequence number 7 gives:
   
   ndn:/localhost/nfd/faces/events/%FE%07

   When there are multiple top-level prefix added, the dispatcher is allowed to pick any top-level prefix; a simple implementation is to pick the first top-level prefix in whatever data structure the dispatcher uses.

   This is not the ideal solution, but since there's no use case in which a program has multiple top-level prefixes and also at least one NotificationStream (the only program for now that has multiple top-level prefix is NFD-RIB, which has no NotificationStream), I plan to solve this problem in the future.

5. See answer 2.

Updated revision https://gist.github.com/yoursunny/e3c520a590af86a0bb37/7a0b3bfcd94cf636a5b50744814043c2050c13b5

According to note-55, can I deduce that current design is a temporal solution regarding to some specific cases of current NFD managements but not a general and idea solution. The implicit assumptions and rules (that current NFD managements already obeys but there is no clear reason we should make them global or long-term policy) are:

1. top-level prefixes are added after all relPrefix are added (all handlers registered). In another word, the dispatcher can only be "configured" (with handlers) during the "initial" process. After that, no more handlers can be registered.

2. one dispatcher can have two or more top-level prefixes, but they share the same set of relPrefix (so, when adding one top-level prefix, all relPrefix are joined to generate full prefixes).

3. if there are two or more top-level prefixes, chose the first to form the names of all notifications. Even though there is no program who has two or more top-level prefixes as well as at least one notification stream (point-5 in note-55), the dispatcher is still possible required to choose a top-level prefix to form the name of the publishing notification.

If this is the case, I can do the implementation first to help move forward other issues, but I think we should work out a more general design later. Still, I have two more questions about current design:

1) when and who will call addTopPrefix. It should not be called by some manager except the last activated one, because this function must be called after all handlers are registered.

2) should the subscribing of notifications be also handled in the dispatcher?

At last, I'm thinking of another temporal but simpler solution.
I guess one of your original intentions to have top-level prefix and relPrefix separated is to allow registering the top-level prefixes (which are much less than relPrefix) to the RIB/FIB only. But now, due to the issue reported in note-49, we will inserted all full prefixes into the interest-filter table in most cases, except when some prefix can be registered to the RIB/FIB directly without command handler. Why we can not allow the manager to directly register full prefixes for command handler, status dataset handler and notification stream? I think the manager who wants to register some handler or notification stream should know the top-level prefix as well. I'm not clear about the big disadvantages of such a solution. I think the obvious advantages are: we do not require to care about when call addTopPrefix and can ignore the above 3 assumptions and rules.

Reply to note-56:

The first and second assumptions are correct.

The third assumption is wrong. The design allows the dispatcher to choose any top-level prefix for notifications; it doesn't have to be the first that is added.

addTopPrefix is invoked at the end of the initialization procedure, eg Nfd::initializeManagement.

There's no concept of "subscribing of notifications".

PostNotification generates unsolicited Data that is pushed into the ContentStore (will be replaced with an InMemoryStorage after #2182).
NotificationSubscriber expresses Interests to fetch notifications from the ContentStore.

The dispatcher does not reply to those Interests.

It's a design intention for Manager to be independent of the top-level prefix.

Manager does not know the full prefix until an Interest arrived.

When an Interest is passed to Manager, the top-level prefix is supplied as the prefix argument to ControlCommandHandler and StatusDatasetHandler.

The first and second assumptions are correct.

The third assumption is wrong. The design allows the dispatcher to choose any top-level prefix for notifications; it doesn't have to be the first that is added.

any only means any top-level prefix is ok. But a deterministic policy to chose the top-level prefix is required in implementation. Otherwise, (1) if you chose "any" top-level prefix, how the subscriber of this notification know which top-level prefix you chose? (2) if the same notification may be posted two or more times, how to ensure picking the same top-level prefix each time?

addTopPrefix is invoked at the end of the initialization procedure, eg Nfd::initializeManagement.

There's no concept of "subscribing of notifications".

PostNotification generates unsolicited Data that is pushed into the ContentStore (will be replaced with an InMemoryStorage after #2182).
NotificationSubscriber expresses Interests to fetch notifications from the ContentStore.

The dispatcher does not reply to those Interests.

How 'NotificationSubscriber' know the name under which it can express the interest. I think there should be some "contract" between the poster and the subscriber. While dispatcher may be a proper bridge.

It's a design intention for Manager to be independent of the top-level prefix.

Manager does not know the full prefix until an Interest arrived.

If the manager does not know the top-level prefix under which the expected interest should be, how can they identity whether this interest is expected. For example, managerA register a handler with relPrefix /A, and there two top-level prefixes /P and /Q. Due to the current design, any prefixes start with /P/A and /Q/A will be dispatched to managerA. So managerA know which prefix (or both) is expected. I guess currently the top-level prefix /localhop/nfd only works for RibManager. However, after registering this top-level prefix, any relPrefix (even registered by other managers) should join with that prefix to form the full prefix, and thus the registered handler will receive the interest start with /localhop/nfd.

When an Interest is passed to Manager, the top-level prefix is supplied as the prefix argument to ControlCommandHandler and StatusDatasetHandler.

Reply to note-58:

Yes, the implementation needs a deterministic policy to choose the top-level prefix used in generating notifications.
This policy is entirely up to the implementation, and is left unspecified in the spec.

NotificationSubscriber knows the Name because it's supplied by the caller.
The only NotificationStream is "Face Status Change Notification", whose prefix is hardcoded to be ndn:/localhost/nfd/faces/events.

The dispatcher cannot assist in this knowledge because the dispatcher lives on the server side of a management protocol, while the NotificationSubscriber is used on the client side; they are in different processes.

ControlCommandHandler and StatusDatasetHandler are designed to be independent of the top-level prefix.
They must accept and process all Interests passed to them.
Any filtering based on top-level prefix, if necessary, needs to be done in Authorization.

Currently, it's impossible for ndn:/localhop/nfd/faces/create to be passed to FaceManager, because NFD's dispatcher does not register ndn:/localhop/nfd top-level prefix; ndn:/localhop/nfd is registered as top-level prefix only in NFD-RIB's dispatcher, which lives in a separate thread.

Yes, the implementation needs a deterministic policy to choose the top-level prefix used in generating notifications.
This policy is entirely up to the implementation, and is left unspecified in the spec.

Yes, but we should make this rule explicit. I mean we should either specify the rule clearly or make the decision of which top-level prefix is selected retrievable.

NotificationSubscriber knows the Name because it's supplied by the caller.
The only NotificationStream is "Face Status Change Notification", whose prefix is hardcoded to be ndn:/localhost/nfd/faces/events.

so, here comes another "special case" this design for: there is one notification stream and the subscriber knows the whole prefix and make this prefix hardcoded. As the subscriber can know the top-level prefix, why the poster can not?

The dispatcher cannot assist in this knowledge because the dispatcher lives on the server side of a management protocol, while the NotificationSubscriber is used on the client side; they are in different processes.

Where I mentioned "dispatcher be a bridge" I mean the following mechanism:

1. both the poster and subscriber know the relPrefix only (such as face/create)

2. subscriber --> dispatcher: the subscriber sends a request (or calls some function in the dispatcher if it can know the reference of the dispatcher) to subscribe some notification stream. For example, one subscriber requests that it's interested in the notification of face/create and its preferred top-level prefix is /localhost/nfd. Then the dispatcher will assign this top-level prefix (localhost/nfd) to the relPrefix (face/create).

3. poster --> dispatcher: the poster calls postNotification to publish some notification (such as /face/create), then the dispatcher will join the relPrefix to each assigned top-level prefix to form all required full prefixes and send out Data under them. (If there is no subscriber subscribes this notification, no Data will be sent out)

ControlCommandHandler and StatusDatasetHandler are designed to be independent of the top-level prefix.
They must accept and process all Interests passed to them.
Any filtering based on top-level prefix, if necessary, needs to be done in Authorization.

Currently, it's impossible for ndn:/localhop/nfd/faces/create to be passed to FaceManager, because NFD's dispatcher does not register ndn:/localhop/nfd top-level prefix; ndn:/localhop/nfd is registered as top-level prefix only in NFD-RIB's dispatcher, which lives in a separate thread.

You mean there may be two or more dispatchers coexist at the same time? I thought there is only one dispatcher who works as the only entry point of the interest. Anyway, If there may be two or more dispatchers coexist, will two different dispatchers register same top-level prefixes (or should we prevent this)? If two different dispatchers will share some top-level prefixes, should we prevent them from sharing relPrefix as well?

Reply to note-60:

No, the choice of which top-level prefix to use in PostNotification will remain implementation dependent.

Note: This is something that can never happen in today's use case.

The caller of PostNotification, aka the FaceManager, is designed to be independent from the full prefix.

This allows NotificationStream to be consistent with ControlCommand and StatusDataset.

This design is intended to fit all mechanisms of NFD Management protocol without changing any part of the protocol.

The unchanged protocol does not support discovering the notification's prefix.

Also, there's no use case for this.

There's no way to stop multiple dispatchers to co-exist and register conflicting top-level prefixes, because they exist in different applications.

Application designers should avoid this situation by avoiding the conflicts.

In the case of NFD and NFD-RIB, NFD will have top-level prefix ndn:/localhost/nfd, and NFD-RIB will have top-level prefixes ndn:/localhost/nfd/rib and ndn:/localhop/nfd/rib.

No, the choice of which top-level prefix to use in PostNotification will remain implementation dependent.

Note: This is something that can never happen in today's use case.

In my opinion, if the design replies on the assumption that something can never happen, we should highlight this assumption. Otherwise, we should have explicit polices regarding that it may happen.

For this case. I need to select a top-level prefix before can send out Data. If there is only one top-level prefix for the dispatcher registered with some notification stream, I have no other choice but the only top-level prefix. It's a policy, but this should be highlighted in the design. Otherwise, if there may be two or more top-level prefixes for a dispatcher registered with some notification stream, a explicit rule should be specified for choosing top-level prefix. If anyone will work and it's no use to ensure all notification for the same relPrefix be published under the same top-level prefix, it's ok. But this should be specified here. Otherwise, I must know which top-level prefix to chose.

The caller of PostNotification, aka the FaceManager, is designed to be independent from the full prefix.

This allows NotificationStream to be consistent with ControlCommand and StatusDataset.

This design is intended to fit all mechanisms of NFD Management protocol without changing any part of the protocol.

The unchanged protocol does not support discovering the notification's prefix.

Also, there's no use case for this.

I just fell it strange that the subscriber knows more than the poster, namely the consumer knows more about the product than the producer.

There's no way to stop multiple dispatchers to co-exist and register conflicting top-level prefixes, because they exist in different applications.

Application designers should avoid this situation by avoiding the conflicts.

In the case of NFD and NFD-RIB, NFD will have top-level prefix ndn:/localhost/nfd, and NFD-RIB will have top-level prefixes ndn:/localhost/nfd/rib and ndn:/localhop/nfd/rib.

So, it should be specified that what should we do when different dispatchers share at least one top-level prefix as well as at least one relPrefix. Or there is an assumption that although they can share some top-level prefix the relPrefix is kept unique even among different dispatchers.

Reply to note-62:

revision 7fee2dc52a6e62b6a3dcb1f440fdd31aec0250ac specifies that in order to use NotificationStream, there must be exactly one top-level prefix.

There's no use case for other cases today; this will be improved after #2182.

NotificationSubscriber does not know more than the publisher.

The publisher consists a manager (or whoever is calling PostNotification) and the initialization procedure that calls addTopLevelPrefix, and they collectively know the absolute prefix.

It's application developer's responsibility to avoid conflicts among dispatchers, if she chooses to use multiple dispatchers on overlapping prefixes.

revision 7fee2dc52a6e62b6a3dcb1f440fdd31aec0250ac specifies that in order to use NotificationStream, there must be exactly one top-level prefix.

There's no use case for other cases today; this will be improved after #2182.

As this is already specified, it's ok for me. But in response to this rule, I will check whether there is only one top-level prefix when postNotification is called.

NotificationSubscriber does not know more than the publisher.

The publisher consists a manager (or whoever is calling PostNotification) and the initialization procedure that calls addTopLevelPrefix, and they collectively know the absolute prefix.

I'm a litter confused about the role. Take the face/create notification as an example, I think the poster / publisher is the FaceManager who monitors the face table and call postNotification whenever required, while addTopLevelPrefix is called after all managers have been initialized (as by some "brain" on top of any manager).

It's application developer's responsibility to avoid conflicts among dispatchers, if she chooses to use multiple dispatchers on overlapping prefixes.

I do not care who is responsible for this. What I care about is whether there are some problems to the dispatcher itself in such case. If it does harm to the whole dispatcher logic, I will write some codes to resolve these problems or throw an error here.

Reply to note-65:

The question is:

Current temporal design has some assumptions or default polices, such as the dispatcher who wants to register a notification stream can only register one top-level prefix. In addition to specifying them on the header file, shall we 1) treat the cases break those assumptions or polices as "logic impossible" and throw an error when they happens, or 2) just make a note at the header file that breaking them will lead to unexpected situations.

Besides, due to note-63, I'm confusing about who is the publisher of notification stream.

Due to my understanding, one dispatcher will manage or service a group of managers. Managers can register handlers or notification stream to the corresponding dispatcher, but can not register top-level prefix. Because the top-level prefix must be registered after all managers serviced by this dispatcher have been initialized. In another word, managers have no knowledge about the top-level prefix they would "use" later.

In the face/create notification example. I think the manager who registers this notification is the poster / publisher, but it indeed has no knowledge about the top-level prefix used to publish this notification. By contrast, the subscriber (in which the whole prefix is hard-coded) knows. This is what I mean "subscriber knows more than publisher".

While in note-63, you claimed that the publisher consists one manager and the initialization procedure. I'm so confused about this. The initialization procedure initializes all managers, it is independent to any manager and actually runs on top-of those managers in logic. But the subscriber is a manager works on the client side. Don't you fell strange about this? Seems that letting a "low layer" entity "talk to" an entity in other side's corresponding layer plus a high layer procedure.

Reply to note-66:

It's already specified that when PostNotification is invoked, but the number of added top-level prefixes does not equal one:

   *  1. if no top-level prefix has been added, or more than one top-level prefixes have been added,
   *     abort these steps and log an error

It's wrong to throw an exception at this point, because nobody is going to catch this exception.

Instead, log an error at WARNING level.

A publisher is a program that generates Data packets.
For a notification stream, the publisher is the program that contains the dispatcher, one or more managers, and initialization code that creates these instances and calls addTopPrefix.

There is no such thing as "manager works on the client side".
The client side does not have a manager.

oh, I guess I misunderstood the scenario. Think of the following code from RibManager:

  NFD_LOG_INFO("Start monitoring face create/destroy events");
  m_faceMonitor.onNotification.connect(bind(&RibManager::onNotification, this, _1));
  m_faceMonitor.start();

If the RibManager subscribes the notification stream of face create/destroy, then we can achieve the similar effect much easier (subscriber sends out interest and then receives the data under the interested prefix).

Reply to note-68:

RibManager doesn't directly subscribe to notifications. It uses FaceMonitor to get the notifications.

FaceMonitor does get the notifications via the notification stream, but this is there internal business of FaceMonitor, and does not imply RibManager knows the full prefix.

Finally, RibManager is in a different thread than FaceManager and they use two separate dispatchers which has no relationship at all.

RibManager doesn't directly subscribe to notifications. It uses FaceMonitor to get the notifications.
Yes, RibManager, to get face notifications, employs a FaceMonitor that connects a callback RibManager::onNotification to the signal NotificationSubscriber::onNotification. The FaceMonitor expresses interests under the prefix ndn:/localhost/nfd/faces/events (hard-coded) and emit the onNotification signal after receives data. Finally, RibManager::onNotification is called to do something in response to the face events.

I thought this mechanism may be as simple as: the RibManager expresses the Interest (itself or through its dispatcher) and responds to the interested face events when the Data arrives (published by face manager's dispatcher). So I was a little bit confusing as described in note-63.

FaceMonitor does get the notifications via the notification stream, but this is there internal business of FaceMonitor, and does not imply RibManager knows the full prefix.

RibManager can see the header file of FaceMonitor where the full prefix is hard-coded.

Finally, RibManager is in a different thread than FaceManager and they use two separate dispatchers which has no relationship at all.

It does dot matter.

1. If they share the same dispatcher, the subscribe-publish procedures will work without the Interest-Data communication.
2. If they belongs to different dispatcher, the publisher-subscriber procedures can work with the Interest-Data communication. The corresponding dispatchers work as agents to express Interest and to publish Data.
3. Certainly, RibManager can also subscribe the notification directly without the help of its dispatcher.

No matter in which case, I think current subscribe mechanism can be simplified.

Reply to note-70:

The dispatcher is the server side of Management protocol.

The subscribe side of a NotificationStream is the client side of Management protocol, and it is completely unrelated to a Dispatcher.

There is no intention of changing the client side of Management protocol in this issue.

RibManager is the server side of RibMgmt, and the client side of FaceMgmt and FibMgmt.
It uses the dispatcher only for its role of the RibMgmt server side; when it acts as a client, the existing mechanisms (Controller and FaceMonitor) will be used.

In FibManager, for self registration (the command Interest has no FaceId or FaceId = 0), the ControlParameters will be modified. And this happens after validating the command interest but before validating the parameters. So, at least, we should change the definition of Dispatcher::Authorization to take a non-const pointer to ControlParameters.

Besides, for validating parameters, all Command::validate* interfaces and Command::applyDefault* interfaces accept a non-const reference to ControlParameters which is conflict with Dispatcher::validateParameters.

Answer to note-72:

No change is necessary.

Neither Authorization nor ValidateParameters should modify ControlParameters.

ndn::nfd::ControlCommand::validateRequest accepts const ControlParameters&.

The argument to ControlCommandHandler should remain as const ControlParameters&, which is kept as the input parameters.

In case a handler needs to use ndn::nfd::ControlCommand::applyDefaultsToRequest or modify the parameters in any other way, it should make a copy of the input parameters, and modify on that copy.

Neither Authorization nor ValidateParameters should modify ControlParameters.

Please review FibManger.cpp. ControlParameters will be changed after Authorization (through command-validator) but before ValidateParameters. And the modified Parameters will be used in both parameters validation and command handler. I do not want to discuss whether this is proper or improper. I just need to make sure that everything will still work if parameters-validation only use unchanged parameters.

Reply to note-74:

It's wrong to modify ControlParameters or apply defaults to it before validation.
ValidateParameters must operate on unmodified ControlParameters that is parsed from incoming Interest.

ControlCommand::validateRequest methods are designed to accept ControlParameters without defaults applied;
FibAddNextHopCommand::validateRequest and this method on other similar classes are designed to permit FaceId == 0.

I had some problems in refactoring faceManager:

1. In FaceManager::listQueriedFaces, sendNack is required. But there is no interface in dispatcher supports this.
   Shall we make sendData (can be used to send NACK) public or make end() accepts some parameter to indicate whether the status dataset handler succeeds or fails with some code (including that for NACK).

2. When publishing face status, we should avoid the old data be fetched if there is a newer version. Shall we do this by setting NO CACHY policy or setting a freshnessPeriod?

Main issue dor me: reject should have similar form as AuthenticationReject: it should accept type of reject, code, and optional message. The current code uses NACKs, just in case.

What is the use case for changing prefix? Is it intended to be used to append suffic to the name?

I have two questions about note-78:

1. Given a prefix, how to check whether or not it starts with an Interest name.

2. If the setExpiry is not called, should we set a default value such that the response will not reside in the cache for too long time.

201508137 conference call discussed note-79 question 1, and we conclude that it's beneficial to define a format for NACK Content.
The format could follow the current ControlResponse, but this implies that ControlResponse to no longer limited to be part of ControlCommand.

I will update the design.

Answer to note-79 question 2:
Yes, a StatusDatasetHandler may add extra component(s) after Interest Name. #3081 could use it.

Answer to note-80 question 1:
The Interest or its Name can be remembered in a private field of StatusDatasetContext.

Answer to note-80 question 2:
Yes, there is a default value. I'll update design to clarify this.

Answer to note-80 question 1:
The Interest or its Name can be remembered in a private field of StatusDatasetContext.

Yes, there is no problem if the StatusDatasetContext know the Interest. But my problem here is the design in note-78 does not describe the way through which the StatusDatasetContext can get information of the Interest. Shall we let the Constructor of StatusDatasetContext accepts a name parameter?

revision 1870a680381e84881385c2b95f54a0f2178c9899 fulfills note-81.

I decide to keep the term ControlResponse but add a detail that it's reused as Content in producer-generated NACK.

I'm also removing RejectReply::NACK because it has same behavior as RejectReply::RESP403 for StatusDataset.

Answer to note-82:

The Interest, and other fields, can be added as private members of StatusDatasetContext.

"private members can be added as necessary, which is accessible from Dispatcher which is declared as a friend"

The Interest, and other fields, can be added as private members of StatusDatasetContext.

"private members can be added as necessary, which is accessible from Dispatcher which is declared as a friend"

sorry, I do not clear about the intention of exposing private members to a friend class, rather than accept required information through the constructor. As the StatusDatasetContext is a parameter for the handler, we can assign all required arguments to construct the Context that is then passed to the handler.

20150821 conference call approves note-83 design.

revision dfcd197d3410cb4b36dc1e14c1fafafb27a08bf0 adds noncopyable declaration to StatusDatasetContext, and changes some function-by-value to function-by-const-reference.

ndn-cxx-docs:commit:eee136fcfe64d72d251ede7fc55ee589cdb16dda documents the design of Dispatcher.

I have been looking at InMemoryStorage API, and have some initial ideas for incorporating that into the Dispatcher.

InMemoryStorage capability and limitation

InMemoryStorage revision 22 says:

Ideally, the InMemoryStorage should find a data packet that can match the incoming interest, that is, besides longest name prefix matching, the InMemoryStorage should also process selectors. However, some selectors are not meaningful to the data producer (e.g., MustBeFresh). Moreover, the variety of data packets at the producer side is much less than the one in the cache intermediate routers, so that some selectors (e.g., ChilderSelector, Min/MaxSuffixComponents, PublisherPublicKeyLocator) may not be necessary.

Looking at InMemoryStorage::find(const Interest&) code:

• If Interest contains implicit digest, all Selectors are ignored.
• If Interest has no implicit digest,
  • ChildSelector is considered.
  • Interest::matchesData is used to consider MinSuffixComponents, MaxSuffixComponents, Exclude, PublisherPublicKeyLocator.
  • MustBeFresh is ignored.

For the purpose of Management,

• StatusDataset clients use ChildSelector, MustBeFresh, and optionally Exclude for first segment, and no Selectors for other segments.
• Notification subscribers use ChildSelector and Exclude in every Interest.
• Notification subscribers may also use ChildSelector and MustBeFresh in first Interest, and no Selectors in subsequent Interests.

The two limitations of InMemoryStorage, ignoring Selectors when implicit digest is present, and ignoring MustBeFresh, shall not cause problem for Management, under the processing logic below.

Processing Logic

StatusDataset publisher:

• When a version is generated, add all segments into storage, and send the first segment to forwarder.
• Set CachePolicy=NoCache tag to bypass local ContentStore.
• Set FreshnessPeriod to a small value (can be smaller than StatusDatasetContext::getExpiry) to avoid being cached in remote ContentStore for too long.
• Set a timer to erase the segments (by versioned prefix) at expiry time (StatusDatasetContext::getExpiry).
• note: It's wrong to erase old versions whenever a new version is generated, because a client may request a new version with Exclude Selector before the old version expires. Erasing old versions would prevent slower clients from fetching other segments of the old version they are still fetching.
• Use a InMemoryStoragePersistent with unlimited capacity, or a InMemoryStorageFifo with enough capacity for 10 versions if DoS attack (requesting many versions) is a concern. Each dataset can have its own storage instance, so that lookups are faster.

Notification publisher:

• Incoming Interests always lookup InMemoryStorage. Remain silent if no Data is found.
• Set CachePolicy=NoCache tag to bypass local ContentStore.
• Set FreshnessPeriod to a small value (can be smaller than StatusDatasetContext::getExpiry) to avoid being cached in remote ContentStore for too long.
• When a notification is posted, add the Data packet to storage, and also send it to forwarder. Sending to forwarder is necessary to satisfy pending Interests waiting for the next notification; if there's no pending Interest, forwarder would drop the unsolicited Data packet.
• Use a InMemoryStorageFifo with enough capacity for notifications posted within 120 seconds (set capacity empirically, no dynamic adjustment needed). Each notification stream can have its own storage instance, so that lookups are faster.

Generally agree with the processing logic. A few small things:

• Why did you specify "Set FreshnessPeriod to a small value". This should be the same as what we do today with FreshnessPeriod (I think we have 1sec), shouldn't be smaller.

• I'm not convinced that it is necessary to set NoCache flags. No real harm, not real benefit.

What I'm mostly concerned is InMemoryStorage implementation of the rightmost child selector. I checked the current implementation and it seem to be not optimized at all for it :( Given it is a primary use case for both notification stream and status dataset, can we (re-)implement a rightmost selector optimization? Or at least, prioritize this implementation.

I'm not convinced that it is necessary to set NoCache flags. No real harm, not real benefit.

The whole purpose of CachePolicy=NoCache is to prevent duplicating caching in both InMemoryStorage and ContentStore.

Why did you specify "Set FreshnessPeriod to a small value". This should be the same as what we do today with FreshnessPeriod (I think we have 1sec), shouldn't be smaller.

FreshnessPeriod doesn't make a different in local ContentStore due to CachePolicy.

It only makes different when a dataset is accessed remotely (inapplicable to today's NFD Management, but may affect other management protocols).

Setting a shorter FreshnessPeriod allows a remote consumer to get an up-to-date dataset without resorting to Exclude selector.

What I'm mostly concerned is InMemoryStorage implementation of the rightmost child selector. I checked the current implementation and it seem to be not optimized at all for it

Regardless of which algorithm is used for rightmost lookup, it shouldn't be a major bottleneck, because the InMemoryStorage capacity is set to store 10 version of a dataset or 120 seconds of notifications, which won't be too many packets.

Don't optimize until a benchmark proves it's a bottleneck. See also #2626 note-13.

20151013 conference call discussed the design in note-92.

A problem with StatusDataset publisher logic is:

1. at time 0ms, version 1 is generated for client A, which has 100 segments and is set to expire at 1000ms
2. at time 990ms, client B requests the current version, and receives version 1 segment 0
3. client B continues to fetch other segments at a rate of 1 segment/ms
4. at time 1000ms, version 1 is erased
5. now client B cannot fetch any more segments

The proper solution for this problem is:

• InMemoryStorage should allow producer to specify "use this Data to satisfy MustBeFresh Interests in next X ms".
• The parameter above shall be independent from FreshnessPeriod, because it's necessary to set FreshnessPeriod to a small value in order to bypass ContentStore on remote forwarders.
• Multiple versions can be kept in the InMemoryStorage, but only non-expired version would satisfy the request for current version which has MustBeFresh.
• Old versions can be used to satisfy Interests for subsequent segments, which do not have MustBeFresh.

InMemoryStorage extension in note-95 solution is written as #3274.

Some questions about the design in note 92:

1. How many storage instances are there? How about their capacities?
   shall the dispatcher own an instance that shared by all datasets and notification streams?
   OR each dataset / notification stream owns an private instance?
   OR I can chose any one of the above two options as I prefer?

2. For each status dataset, the Interest callback are registered by the dispatcher, so that the dispatcher can process corresponding incoming Interests (such as lookup the storage) before direct them to their handlers. But the case for notification stream is different. No Interest callback registered for a notification stream, how can the dispatcher get the requests from notification subscribers?

3. There is no StatusDatasetContext during the process of posting notification, how to set the FreshnessPeriod (be smaller than StatusDatasetContext::getExpiry would be unreasonable in this case)

My opinions:

1. Let the dispatcher own a storage instance (persistent with unlimited capacity), which is shared by all registered status dataset and notification stream. When it's proved this will lead to issues (performance or safety) we can improve it.

2. When receives an Interest (for some dataset), the dispatcher will lookup the storage first after it's authorized and validated, and then direct it to the registered handler if the lookup fails.

3. in StatusDatasetContext::append and StatusDatasetContext::end, add data segments into the storage and send out the first segment.

4. when a notification stream is registered to the dispatcher, also register an Interest callback so that the dispatcher can receives requests from subscribers of this notification.

5. in Dispatcher::postNotification, add data into the storage and send out the data.

If we follow the above logic, there will be no change in managers.

First of all, note-92 has been rejected in note-95, and is replaced by the design in note-95.

Review on note-98 design:

1. Having a persistent storage with unlimited capacity should help management to serve contents as necessary. However, the design does not specify when inserted Data is cleaned out. As a consequence, this storage will grow indefinitely and ultimate lead to an out-of-memory crash.
   It should be good to go after adding a clean up procedure.
   Note: Although the design could be subject to DoS attack and cause OOM, PIT is also subject to such attacks until #1301 completes, so I won't worry about that.
2. Data is added to the storage as a result of an authorized Interest. What's the purpose of doing authorization before querying the storage for a match?
3. OK.
4. OK. An alternate design is to query every incoming Interest in the storage first, before dispatching to a ControlCommand or StatusDataset handler.
5. OK.

In addition:

• The parameter of "use this Data to satisfy MustBeFresh Interests in next X ms" should be specified by StatusDatasetContext::setExpiry. This is the design intention of setExpiry.
• FreshnessPeriod of all Data packets from the dispatcher should be set to a small value, determined by a compile-time constant in Dispatcher class.
• NDNLPv2 CachePolicy=NoCache should be set on Data packets to prevent duplicate caching on local ContentStore. This is the design intention of CachePolicy field.

What's the purpose of doing authorization before querying the storage for a match?

In logic, only the authorized requester can get response of status datasets. When requester B (who can not be authorized) requests the status of a face, which have previously been requested by requester A (who is authorized) and already exist in the storage. Shall we prevent B from fetching this piece of data?

In practice, currently, all managers employ an authorization method (for status dataset) that accepts all requesters. So, whether we direct requests to the storage after authorization or not, there is no difference in practice.

An alternate design is to query every incoming Interest in the storage first, before dispatching to a ControlCommand or StatusDataset handler.

If the top-level prefix has not been registered (whether or not to register the top-level prefix is optional), the dispatcher will not receive the Interests for notification. So I prefer to register Interest callback for notification stream explicitly.

• The parameter of "use this Data to satisfy MustBeFresh Interests in next X ms" should be specified by StatusDatasetContext::setExpiry.

What's the suggested value of "X"? or what policy should I follow to set its value?

• FreshnessPeriod of all Data packets from the dispatcher should be set to a small value, determined by a compile-time constant in Dispatcher class.

for status dataset, the value can be set as smaller than StatusDatasetContext::getExpiry. But for notification stream, how to set the value.

Answer to note-100:

1. No. Suppose requester B sends its Interest before requester A's Interest was satisfied, when management replies the Data, the Data would be received by both requesters. Therefore, withholding Data from requester B through the authorization procedure does not prevent requester B from obtaining the Data.
   If it's necessary to withhold Data from requester B, encryption should be used instead.
2. Agreed.
3. The default value for expiry should remain the same.
4. FreshnessPeriod can be set to a small value, unrelated to the effective setting of expiry. I suggest 100ms or less. Note that FreshnessPeriod is irrelevant when the management protocol is used on localhost; it only controls how long a remote forwarder considers the Data as non-stale.

I want to schedule an event to erase a packet in storage after a specified expiry time, how to use ndn::Scheduler properly in ndn-cxx?

I tried to create boost::asio::io_service instance as a private member of Dispatcher, which is used to construct another private member (a util::Scheduler). But this can not work.

Yanbiao, you don't need to do this in the dispatcher. For now we will completely ignore problem of old packets and will rely on InMemoryStorage replacement policy to remove excess data. If it will be causing problems, we will figure out some other way, but still within InMemoryStore, not in management.

Reply to note-103:

note-98 design chooses to use persistent InMemoryStorage so dispatcher MUST clean up the storage.

If dispatcher does not want to clean up the storage, choose an InMemoryStorage with a different policy.

In that case, a sufficiently large InMemoryStorage should work fine as long as the node is not under attack.

In my opinion, we should NOT use persistent policy. FIFO would be simple and enough for the job. Capacity of 100 or 200 should be enough as well (though this needs to be configurable).

Hard-coded capacity may negatively impact low-memory platforms. Also, I would like to reduce this value when NFD is embedded in ndnSIM.

Reply to note-107:

This should be discussed on #2182.

Dispatcher design in this issue does not hard code the effective capacity.

I missed that in previous conversations, but why the heck FreshessPeriod is not the same thing as expiry? The whole point of FreshnessPeriod is to play the role of guarding interval for the management not receiving interests.

We agreed to have this value for 1 second, which should be used both in InMemoryStore and remote forwarders (for non-localhost management protocols).