Project

General

Profile

Actions

Repo Protocol Specification » History » Revision 108

« Previous | Revision 108/136 (diff) | Next »
Shuo Chen, 02/26/2014 09:45 AM


Repo Protocol Specification

A Repo supports the network by preserving content and responding to Interests requesting content that it holds. A Repo can exist in any node, and is recommended if applications in that node need to preserve data. The NDN repo protocol is a specification of repo operations including reading, insertion and deletion of data objects in repo.

Repo semantics is based on Command Interests with signed commponents an the end of the name and basic common semantics of NDN.

When some operation of repo including insertion and deletion of data objects is requested, a command interest is sent. The command interest is the interest as command of insertion and deletion and signed as form of command interest for access control. The repo will response the command with the data object.

The repo protocol can be categorized in three sections of reading, insertion and deletion of data objects.

Repo Reading Specification

Repo registers prefixes of data objects it holds into NDN fowarding deamon and the repo will respond the data with such prefixes.

A standard interest is used to fetch content from the repo. The repo will respond when the name of the interest matches the prefix it registered in NFD. If the content in repo matches the interests, it will respond with the data object. When the interest is not matched, it will not respond.

The protocol is listed as bellow

If there is a matching data object:

Requester                     Repo
    |                           |
    |                           |
    |         Interest          |
 t1 |-------------------------->|
    |                           |
    |        Data Object        |
 t2 |<==========================|
    |                           |
    |                           |
    |                           |

If there is no matching data object:

Requester                     Repo
    |                           |
    |                           |
    |         Interest          |
 t1 |-------------------------->|
    |                           |
    |                           |
    |                           |

About Freshness

Solution to handle freshness by repo has not been clearly defined, so producer needs to take care of freshness when it puts things into repo, i.e. explicitly delete obsolete content. The mustbefresh selector will be ignored by repo when fetching content from repo or handling repo command by repo.

Repo Command

For insertion, deletion and other operations of repo, these commands are encoded in the form of sigend command interest. The semantics of repo command interest is as follows:

The name semantics is defined to have following components:

  • <repo prefix> refers to specific prefix repo is listening
  • <command verb> refers to the name of command
  • <RepoCommandParameter> refers to parameters of repo command

The following components are components of singed interest for access control:

  • <timestamp>
  • <random-value>
  • <SignatureInfo>
  • <SignatureValue>

For prefix of repo /ucla/cs/repo/, the command will be defined as this:

/ucla/cs/repo/<command verb>/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

RepoCommandParameter

RepoCommandParameter ::= REPOCOMMANDPARAMETER-TYPE TLV-LENGTH
                           Name?
                           Selectors?
                           StartBlockId?
                           EndBlockId?
                           ProcessId?

Name                  ::= NAME-TYPE TLV-LENGTH NameComponent*
NameComponent         ::= NAME-COMPONENT-TYPE TLV-LENGTH BYTE+

Selectors             ::= SELECTORS-TYPE TLV-LENGTH
                           MinSuffixComponents?
                           MaxSuffixComponents?
                           PublisherPublicKeyLocator?
                           Exclude?
                           ChildSelector?

MinSuffixComponents   ::= MIN-SUFFIX-COMPONENTS-TYPE TLV-LENGTH
                           nonNegativeInteger

MaxSuffixComponents   ::= MAX-SUFFIX-COMPONENTS-TYPE TLV-LENGTH
                           nonNegativeInteger

PublisherPublicKeyLocator ::= KeyLocator

Exclude               ::= EXCLUDE-TYPE TLV-LENGTH Any? (NameComponent (Any)?)+
Any                   ::= ANY-TYPE TLV-LENGTH(=0)

ChildSelector         ::= CHILD-SELECTOR-TYPE TLV-LENGTH
                           nonNegativeInteger

StartBlockId          ::= STARTBLOCKID-TYPE TLV-LENGTH
                           nonNegativeInteger

EndBlockId            ::= ENDBLOCKID-TYPE TLV-LENGTH
                           nonNegativeInteger

ProcessId            ::= PROCESSID-TYPE TLV-LENGTH
                           nonNegativeInteger

Name

Name in command parameter is TLV-encoded block and represents the name or prefix of the data repo will process. The definition and format of name section are the same as the name section of NDN interest.

Repo Command Selectors

Repo command supports parts of interest selectors of interest to indicate which contents to process. The definition of standard NDN selectors is described in NDN Selectors Doc. The concrete definitions of both standard NDN selectors and repo command selectors are the same.

The difference between standard NDN interest and Repo Deletion Command interest that, the standard NDN selectors just matches one data packet that conforms to the selector conditions, but repo command selectors would matches any data packets. For example, if Interest is expressed for /ndn/edu and Exclude specifies one name component ucla, in standard NDN interest, the data producers will first exclude all the data packets with prefix /ndn/edu but not /ndn/edu/ucla, and then just selects one data packet. In repo deletion command interest, it will select all the data packets with prefix /ndn/edu but not /ndn/edu/ucla. However, in repo insert command, the repo will fetch one data just like standard interest selectors.

Repo command supports parts of standard NDN interests including MinSuffixComponents, MaxSuffixComponents, PublisherPublicKeyLocator, Exclude, ChildSelector. If command contains other selectors, repo will ignore these not supported selectors. In addition, selectors are just supported in delete command. If other commands contains selectors, repo will ignore selectors when processing these commands.

The form of selectors is as follows:

Selectors             ::= SELECTORS-TYPE TLV-LENGTH
                           MinSuffixComponents?
                           MaxSuffixComponents?
                           PublisherPublicKeyLocator?
                           Exclude?
                           ChildSelector?

MinSuffixComponents   ::= MIN-SUFFIX-COMPONENTS-TYPE TLV-LENGTH
                           nonNegativeInteger

MaxSuffixComponents   ::= MAX-SUFFIX-COMPONENTS-TYPE TLV-LENGTH
                           nonNegativeInteger

PublisherPublicKeyLocator ::= KeyLocator

Exclude               ::= EXCLUDE-TYPE TLV-LENGTH Any? (NameComponent (Any)?)+
Any                   ::= ANY-TYPE TLV-LENGTH(=0)

ChildSelector         ::= CHILD-SELECTOR-TYPE TLV-LENGTH
                           nonNegativeInteger

StartBlockId, EndBlockId

StartBlockId and EndBlockId are used to process segmented data. StartBlockId indicate the first segment number and EndBlockId indicate the last segment number. Repo will process segment data whose segment id between StartBlockId and EndBlockId. If StartBlockId is missing, the first segment id the repo process is 0; If EndBlockId is missing, this scenario is described in specific process in Repo Insertion Command section and Repo Deletion Command section.

Conflict of Selectors and StartBlockId, EndBlockId

Repo cannot process command with both selectors and StartBlockId, EndBlockId in RepoCommandParameter. If the RepoCommandParameter carries both, repo will ignore this command interest and return with error code of 405.

ProcessId

ProcessId is used by insertion and deletion check command to indicate specific insertion and deletion process. The ProcessId is fetched by repo command response of insertion and deletion command.

Repo Command Response

Repo command response is the response data packet of repo command interest. The response contains statuscode to indicate the status of command process and other information. A TLV-encoded block called RepoCommandResponse is encoded in content of the data packet.

RepoCommandResponse   ::= INSERTSTATUS-TYPE TLV-LENGTH
                           ProcessId?
                           StatusCode?
                           StartBlockId?
                           EndBlockId?
                           InsertNum?
                           DeleteNum?

ProcessId            ::= PROCESSID-TYPE TLV-LENGTH
                            nonNegativeInteger 

StatusCode            ::= STATUSCODE-TYPE TLV-LENGTH
                            nonNegativeInteger    

StartBlockId          ::= STARTBLOCKID-TYPE TLV-LENGTH
                            nonNegativeInteger

EndBlockId            ::= ENDBLOCKID-TYPE TLV-LENGTH
                            nonNegativeInteger

InsertNum             ::= INSERTNUM-TYPE TLV-LENGTH
                            nonNegativeInteger

DeleteNum             ::= DELETENUM-TYPE TLV-LENGTH
                            nonNegativeInteger

ProcessId

ProcessId is a random number generated by repo to indicate the number of the command process. Client could use this ProcessId to check the status of specific command.

StatusCode

StatusCode indicates the status of repo command process. The statuscodes of insert and deletion command are described in following insertion and deletion specification.

StartBlockId, EndBlockId

StartBlockId and EndBlockId are the same as those of RepoCommandParameter. If either of those in RepoCommandParameter is missing, repo will set them as the Id known for now. For example, if StartBlockId is missing in RepoCommandParameter, StartBlockId in response will be set 0. If EndBlockId is missing in RepoCommandParameter, EndBlockId will be set null untill Repo get FinalBlockId in data packet. If FinalBlockId in returned data packet is less than EndBlockId, the EndBlockId will be set FinalBlockId.

InsertNum, DeleteNum

InsertNum is used in reponse of insertion status check to indicate how many data packets have been successfully inserted into the repo. DeleteNum is used in response of deletion command and deletion check command. DeleteNum indicates how many data packets have been successfully inserted from repo.

Repo Trust Model

The trust model of repo depends on people who deploy the repo service, such as PKI. Repo can specify their own verification policies, and data consumers can specify their own trust anchors. The NDN FAQ shows how NDN trust managment works.

Repo Insertion Specification

Repo insertion command requests that the repo retrieve and store content. This command interest is a signed interest and will be validated with access control policy defined by the repo. When the interested is validated and name of the data is not existed in the repo. The repository will reponse with a data object containing OK status and start to send the interest to fetch the data to insert.

Segmented data insertion is also supported in the insertion protocol. Segmantation info is defined as a TLV-encoded SegmantationInfo block. If the content is segmented, the final segment id will be encoded in this block.

Insertion command interest semantics

The name semantics is follows the format of the repo command. The is defined as insert.
For example, for <repo prefix> as /ucla/cs/repo, the following is an example:

/ucla/cs/repo/insert/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

Insertion status check

During the insertion progress, the requester could send insertion status check command to check the status of the insertion progress. This status check command is also signed interest. The semantics of insertion status check command is shown as follows:

equals to insert check. For example:

/ucla/cs/repo/insert check/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

RepoCommandParameter

RepoCommandParameter of insertion and insertion check command follows that in Repo Command section. Name, Selectors, StartBlockId, EndBlockId are used in insertion. Name and ProcessId are used in insertion check command.

In insertion command, Name represents the name or prefix of data for repo to fetch. If selectors are set, repo will use these selectors to fetch the data. If StartBlockId or EndBlockId is set, the repo will retrieve segmented data with segment number between StartBlockId and EndBlockId. If Repo Insert Command carries both Selectors and StartBlockId, EndBlockId, repo would ignore it and response with statuscode of 405.

In insertion check command, Name represents name or prefix of data for repo to fetch. ProcessId is set by the RepoCommandResponse to indicate specified process.

Insertatation status response

This insert status data object could be the response data object of both insert command and insert check command. It will follow the format of repo command response.

StatusCode indicates the status of insertion. InsertNum represents how many data has been intersted into the repo. StartBlockId and EndBlockId is the start and end segment id of data inserted. InsertNum is the how many data segments have been inserted. ProcessId indicates an ID of process, a random number gernerated by repo.

For insert cammand, status code will be set according to definition below, StartBlockId and EndBlockId will be set according to the insert command. If StartBlockId in RepoCommandParameter is missing, it will be set 0 in response. If EndBlockId is missing, it will not be set.

For insert check command, status code will be set according to definition below, StartBlockId and EndBlockId will be set according to the StartBlockId and EndBlockId that repo is using, insertnum will be set according to insertion progress. ProcessId is set according to the ID of process checked. If EndBlockId has not been decided, this EndBlockId will not be set in response.

StatusCode Definition:

StatusCode Description
100 The command is OK. can start to fetch the data
200 All the data has been inserted
300 This insertion is in progress
401 This insertion command or insertion check command is invalidated
402 Selectors and BlockId both present
403 Malformed Command
404 No such this insertion is in progress
405 EndBlockId Missing Timeout

EndBlockId Missing Timeout

If StartBlockId presents but EndBlockId is missing, and returned data packets do not contain FinalBlockId, the repo will continuosly fetch the data. An EndBlockId missing timeout is set to prevent this ocassion. The repo will start a timer when StartBlockId presents but EndBlockId is missing. When timeout happens, repo will stop fetching data to store and end insert process. If an insert check command arrives during this insert process, the time of timer is set to 0. If data packet containing FinalBlockId arrives, this timeout timer will be dismissed.

Protocol Process

  1. start to authorize the command; if authorization does not fail immediately, go to step 3

  2. send a negative response indicating authorization failure, and abort these steps, insert process ends (StatusCode: 401)

  3. if both StartBlockId and EndBlockId are missing, go to step 7

  4. if either StartBlockId or EndBlockId is present, and one of supported selectors is present. send negative reponse back and abort steps, insert process ends

  5. if both StartBlockId and EndBlockId are present, and StartBlockId is less than or equal to EndBlockId, go to step 7

  6. send a negative response indicating malformed command, and abort these steps, insert process ends (StatusCode: 403)

  7. wait for authorization completion

  8. if authorization fails, go to step 2 (StatusCode: 401)

  9. send a positive response indicating insert is in progress (StatusCode: 200)

  10. if either StartBlockId or EndBlockId is present, go to step 16

  11. start to retrieve Name with selectors in insert command

  12. wait for retrieval completion

  13. if retrieval fails, go to step 27

  14. store retrieved Data packet

  15. abort these steps, insert process ends

  16. if StartBlockId is missing, set StartBlockId 0. If EndBlockId is missing, EndBlockId will be missing unless get FinalBlockId in comming data packets, start EndBlockId Misiing Timeout timer.

  17. append StartBlockId to Name

  18. start to retrieve Name

  19. wait for retrieval completion

  20. if retrieval fails, go to step 26

  21. store retrieved Data packet

  22. if retrieved Data packet contains FinalBlockId, and FinalBlockId is less than EndBlockId or EndBlockId is missing, let EndBlockId be FinalBlockId

  23. if the last component of Name is greater than or equal to EndBlockId, abort these steps, insert process ends

  24. increment the last component of Name

  25. go to step 17

  26. retrieve data with this data another 2 times. If these 2 retrieval both failes, abort these steps. if success, go to step 20

  27. retrieve data with this data another 2 times. If these 2 retrieval both failes, abort these steps. if success, go to step 13

If EndBlockId Missing Timeout timer starts, repo will monitor this timer during step 17~26. If this timeout occurs, abort insert command process immediately.

Implementation MAY pipeline the Interests.

Repo insert check command progress

Implementation MAY publish a notification of status regarding insert progress. The process of status check is as follows:

  1. start to authorize the insert status command

  2. send a negative response indicating authorization failure, and abort these steps

  3. start to check the progress of the insert with the data name in the command. If no such progress is found, go to 4. or go to 5.

  4. reponse status with status code, abort check process.

  5. ckeck the status of insertion. return the status of insertion progress. If a EndBlockId Missing Timeour timer is running, set this timer to 0. About check process.

Protocol diagram:

Requester                     Repo                          Data producer
    |                           |                                 |
    |                           |                                 |
  +---+  Insert command       +---+                               |
  |   | --------------------> |   |                               |
  +---+                       |   |                               |
    |                         |   |                               |
  +---+   Confirm start       |   |                               |
  |   | <==================== |   |                               |
  +---+   Reject command      +---+                               |
    |     (with status code)    |                                 |
    |                         +---+     Interest for Data       +---+
    |                         |   | --------------------------> |   |
    |                         +---+                             |   |
    |                           |                               |   |
    |                         +---+       Data segment          |   |
    |                         |   | <========================== |   |
    |                         +---+                             +---+
    |                           |                                 |
    |                           ~                                 ~
    |                           ~                                 ~
    |                           |                                 |
    |                         +---+     Interest for Data       +---+
    |                         |   | --------------------------> |   |
    |                         +---+                             |   |
    |                           |                               |   |
    |                         +---+       Data segment          |   |
    |                         |   | <========================== |   |
    |                         +---+                             +---+
    |                           |                                 |
    |                           |                                 |
    |                           ~                                 ~
    |                           ~                                 ~
    |                           |                                 |
    |                           |                                 |
    |                           |                                 |
  +---+   Status interest     +---+                               |
  |   | --------------------> |   |                               |
  +---+                       |   |                               |
    |                         |   |                               |
  +---+    Status response    |   |                               |
  |   | <==================== |   |                               |
  +---+                       +---+                               |
    |                           |                                 |
    |                           |                                 |

Repo Deletion Specification

Deletion of one content object or content objects under certain prefix are both supported in repo specification. Selectors are used to select multiple content objects. These selectors are different from conventional selectors of interest packet. The conventional selecors of interest would select one data packet of repo. Selecors of delete command will return any data packets which applies for these selectors. Deletion of segmented data is also supported.

Deletion command interest semantics

Deletion command interest follows the format of the repo command. For example:

/ucla/cs/repo/delete/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue

Deletion Command RepoCommandParameter

Format of deletion command RepoCommandParameter follows that of Repo Command. Name, Selector, StartBlockId, EndBlockId, ProcessId are used to construct RepoCommandParameter.

Name is the name or the prefix of data repo will delete.

Selector is used to select data.

StartBlockId and EndBlockId are used to delete segment data. Data of segment id between StartBlockId and EndBlockId will be deleted by repo.

ProcessId is a random number generated by client to indicate the deletion process. Repo will match this ProcessId with this deletion process.

Deletion Command Selectors

Deletion command selectors follows the repo command selectors.

Deletion status check

During the deletion progress, the requester could send deletion status check command to check the status of the deletion progress. This status check command is also signed interest. The semantics of deletion status check command follows the format of repo command. <command verb> equals to delete check. For example:

/ucla/cs/repo/delete/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

Deletion status check RepoCommandParameter

Name and ProcessId are used to indicate certain deletion process. Repo uses Name and ProcessId to match certain deletion process. If it matches, repo will respond progress of deletion. If just one of Name or ProcessId matches, it fails.

Deletion Check Command Selectors

Selectors are not supported. Repo will ignore selectors when processing deletion status check command.

Deletion status response

Deletion status response is used as response of repo deletion command and repo deletion check command.

Name, StatusCode, Selector, StartBlockId, EndBlockId, ProcessId, DeletenNum are used in deletion status response. Name, ProcessId, Selector is the same as that of delete command. StatusCode indicates the status of deletion. DeleteNum indicates how many data packets have been deleted.

In Repo Deletion Command, if just StartBlockId presents, EndBlockId may be the largest segment ID repo holds. If just EndBlockId presents, StartBlockId will be set 0

StatusCode Definition:

StatusCode Description
200 All the data has been deleted
300 This deletion is in progress
401 This deletion or deletion check is invalidated
402 Selectors and BlockId both present
403 Malformed Command
404 No such this deletion is in progress

Protocol process

  1. start to authorize the command; if authorization does not fail, go to step 3

  2. send a negative response indicating authorization failure, and abort these steps, end deletion process. (StatusCode: 401)

  3. check whether a deletion process of same RepoCommandParameter exists, waiting for deletion process ends.

  4. If selectors and one of StartBlockId and EndBlockId presents, send a negative response and abort these steps, end deletion process. (StatusCode: 402)

  5. If selectors present, go to step 8

  6. check whether StartBlockId or EndBlockId presents. If both presents but StartBlockId is larger than EndBlockId, return negative response and end deletion process. (StatusCode: 403) Or go to step 9

  7. If StartBlockId, EndBlockId and selectors are all missing, go to step 10

  8. delete all the data that conforms to the name and selectors, go to step 11

  9. delete all the data packets of segment id between StartBlockId and EndBlockId. If StartBlockId is missing, StartBlockId is set to be 0. If EndBlockId is missing, EndBlockId is set to be the largest segment id that repo holds. go to step 11

  10. delete data exact matches the name. got to step 11

  11. If lifetime of interest does not expire, return status response of positive statuscode. If lifetime of interest has expired, wait for interest the same RepoCommandParameter and return this status response. End Deletion process. (StatusCode: 200)

Client will set deletion command with big lifetime. If life time expires, client will re-express the command.

Implementation MAY publish a notification of status regarding delete progress. The process of status check is as follows:

  1. start to authorize the delete status command

  2. send a negative response indicating authorization failure, and abort these steps (StatusCode: 401)

  3. start to check the progress of the delete with the data name in the command. If no such progress is found, go to 4. or go to 5.

  4. reponse status with status code of 404 (StatusCode: 404)

  5. ckeck te status of delete. return the status data content (StatusCode: 300)

Protocol diagram:

Requester                     Repo 
    |                           |                                 
    |                           |                                 
  +---+  Delete command       +---+                               
  |   | --------------------> |   |                               
  +---+                       +---+                               
    |                           |                                 
    |                           |                                 
    |                           |                                 
  +---+   Status interest     +---+                               
  |   | --------------------> |   |                               
  +---+                       |   |                               
    |                         |   |                               
  +---+    Status response    |   |                               
  |   | <==================== |   |                               
  +---+                       +---+                               
    |                           |                                 
    |                           |                                 
    |                           |                                 
  +---+   Confirm Deletion    +---+                               
  |   | <==================== |   |                               
  +---+   Reject command      +---+                               
    |     (with status code)    |    
    |                           |

Updated by Shuo Chen about 10 years ago · 108 revisions