repo-ng

Repo Protocol Specification

===========================

A Repo supports the network by preserving content and responding to Interests requesting content that it holds. A Repo can exist in any node, and is recommended if applications in that node need to preserve data. The NDN repo protocol is a specification of repo operations including reading, insertion and deletion of data objects in repo.

Repo semantics is based on [[ndn-cxx:SignedInterest|Signed Interests]] with signed components an the end of the name and basic common semantics of NDN. 

When some operation of repo including insertion and deletion of data objects is requested, a command interest is sent. The command interest is the interest as command of insertion and deletion and signed as form of command interest for access control. The repo will response the command with the data object.

The repo protocol can be categorized in three sections of reading, insertion and deletion of data objects.

## Repo Reading Specification

Repo registers prefixes of data objects it holds into NDN forwarding daemon and the repo will respond the data with such prefixes.

A standard interest is used to fetch content from the repo. The repo will respond when the name of the interest matches the prefix it registered in NFD. If the content in repo matches the interests, it will respond with the data object. When the interest is not matched, it will not respond.

The protocol is listed as bellow

If there is a matching data object:

    Requester                     Repo

        |                           |

        |                           |

        |         Interest          |

     t1 |-------------------------->|

        |                           |

        |        Data Object        |

     t2 |<==========================|

        |                           |

        |                           |

        |                           |

If there is no matching data object:

    Requester                     Repo

        |                           |

        |                           |

        |         Interest          |

     t1 |-------------------------->|

        |                           |

        |                           |

        |                           |

### About Freshness

Solution to handle freshness by repo has not been clearly defined, so producer needs to take care of freshness when it puts things into repo, i.e. explicitly delete obsolete content. The `MustBeFresh` selector will be ignored by repo when fetching content from repo or handling repo command by repo.

## Repo Command

For insertion, deletion and other operations of repo, these commands are encoded in the form of [[ndn-cxx:SignedInterest|Signed Interests]]:

    /<repo-prefix>/<command-verb>/................./.........................................

                                  \______  _______/ \__________________  ___________________/

                                         \/                            \/

                                 RepoCommandParameter   Signed Interest additional components

The semantics of repo command interest is as follows:

The name semantics is defined to have following components:

* ``<repo prefix>`` refers to specific prefix repo is listening

* ``<command verb>`` refers to the name of command

* ``<RepoCommandParameter>`` refers to parameters of repo command

The following components are components of singed interest for access control:

* ``<timestamp>``

* ``<random-value>``

* ``<SignatureInfo>``

* ``<SignatureValue>``

For prefix of repo /ucla/cs/repo/, the command will be defined as this:

    /ucla/cs/repo/<command verb>/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

### RepoCommandParameter

    RepoCommandParameter ::= REPOCOMMANDPARAMETER-TYPE TLV-LENGTH

                               Name?

                               Selectors?

                               StartBlockId?

                               EndBlockId?

                               ProcessId?

    Name                  ::= NAME-TYPE TLV-LENGTH NameComponent*

    NameComponent         ::= NAME-COMPONENT-TYPE TLV-LENGTH BYTE+

    Selectors             ::= SELECTORS-TYPE TLV-LENGTH

                               MinSuffixComponents?

                               MaxSuffixComponents?

                               PublisherPublicKeyLocator?

                               Exclude?

                               ChildSelector?

    MinSuffixComponents   ::= MIN-SUFFIX-COMPONENTS-TYPE TLV-LENGTH

                               nonNegativeInteger

    MaxSuffixComponents   ::= MAX-SUFFIX-COMPONENTS-TYPE TLV-LENGTH

                               nonNegativeInteger

    PublisherPublicKeyLocator ::= KeyLocator

    Exclude               ::= EXCLUDE-TYPE TLV-LENGTH Any? (NameComponent (Any)?)+

    Any                   ::= ANY-TYPE TLV-LENGTH(=0)

    ChildSelector         ::= CHILD-SELECTOR-TYPE TLV-LENGTH

                               nonNegativeInteger

    StartBlockId          ::= STARTBLOCKID-TYPE TLV-LENGTH

                               nonNegativeInteger

    EndBlockId            ::= ENDBLOCKID-TYPE TLV-LENGTH

                               nonNegativeInteger

    ProcessId            ::= PROCESSID-TYPE TLV-LENGTH

                               nonNegativeInteger

#### Repo Command Selectors

Repo command supports parts of interest selectors of interest to indicate which contents to process. The definition of standard NDN selectors is described in [NDN Selectors Doc](http://named-data.net/doc/ndn-tlv/interest.html#selectors). The concrete definitions of both standard NDN selectors and repo command selectors are the same.

The difference between standard NDN interest and Repo Deletion Command interest that, the standard NDN selectors just matches **one** data packet that conforms to the selector conditions, but repo command selectors would matches **any** data packets. For example, if Interest is expressed for /ndn/edu and Exclude specifies one name component ucla, in standard NDN interest, the data producers will first exclude all the data packets with prefix /ndn/edu but not /ndn/edu/ucla, and then just selects one data packet. In repo deletion command interest, it will select all the data packets with prefix /ndn/edu but not /ndn/edu/ucla. However, in repo insert command, the repo will fetch one data just like standard interest selectors.

Repo command supports parts of standard NDN interests including MinSuffixComponents, MaxSuffixComponents, PublisherPublicKeyLocator, and Exclude. ChildSelector is supported in insertion command but not deletion command. If command contains other selectors, repo will ignore these not supported selectors. In addition, selectors are just supported in delete command. If other commands contains selectors, repo will ignore selectors when processing these commands.

The form of selectors is as follows:

    Selectors             ::= SELECTORS-TYPE TLV-LENGTH

                               MinSuffixComponents?

                               MaxSuffixComponents?

                               PublisherPublicKeyLocator?

                               Exclude?

                               ChildSelector?

    MinSuffixComponents   ::= MIN-SUFFIX-COMPONENTS-TYPE TLV-LENGTH

                               nonNegativeInteger

    MaxSuffixComponents   ::= MAX-SUFFIX-COMPONENTS-TYPE TLV-LENGTH

                               nonNegativeInteger

    PublisherPublicKeyLocator ::= KeyLocator

    Exclude               ::= EXCLUDE-TYPE TLV-LENGTH Any? (NameComponent (Any)?)+

    Any                   ::= ANY-TYPE TLV-LENGTH(=0)

    ChildSelector         ::= CHILD-SELECTOR-TYPE TLV-LENGTH

                               nonNegativeInteger

#### StartBlockId, EndBlockId

StartBlockId and EndBlockId are used to process segmented data. StartBlockId indicate the first segment number and EndBlockId indicate the last segment number. Repo will process segment data whose segment id between StartBlockId and EndBlockId. If StartBlockId is missing, the first segment id the repo process is 0; If EndBlockId is missing, this scenario is described in specific process in Repo Insertion Command section and Repo Deletion Command section.

#### Conflict of Selectors and StartBlockId, EndBlockId

Repo cannot process command with both selectors and StartBlockId, EndBlockId in RepoCommandParameter. If the RepoCommandParameter carries both, repo will ignore this command interest and return with error code of 405.

#### ProcessId

ProcessId is used by insertion and deletion check command to indicate specific insertion and deletion process. The ProcessId is fetched by repo command response of insertion and deletion command.

## Repo Command Response

Repo command response is the response data packet of repo command interest. The response contains statuscode to indicate the status of command process and other information. A TLV-encoded block called ``RepoCommandResponse`` is encoded in content of the data packet.

    RepoCommandResponse   ::= INSERTSTATUS-TYPE TLV-LENGTH

                               ProcessId?

                               StatusCode

                               StartBlockId?

                               EndBlockId?

                               InsertNum?

                               DeleteNum?

    ProcessId            ::= PROCESSID-TYPE TLV-LENGTH

                                nonNegativeInteger 

    StatusCode            ::= STATUSCODE-TYPE TLV-LENGTH

                                nonNegativeInteger    

    StartBlockId          ::= STARTBLOCKID-TYPE TLV-LENGTH

                                nonNegativeInteger

    EndBlockId            ::= ENDBLOCKID-TYPE TLV-LENGTH

                                nonNegativeInteger

    InsertNum             ::= INSERTNUM-TYPE TLV-LENGTH

                                nonNegativeInteger

    DeleteNum             ::= DELETENUM-TYPE TLV-LENGTH

                                nonNegativeInteger

### Name

Name indicates the Name in repocommandparameter of repo command

### ProcessId

ProcessId is a random number generated by repo to indicate the number of the command process. Client could use this ProcessId to check the status of specific command.

### StatusCode

StatusCode indicates the status of repo command process. The statuscodes of insert and deletion command are described in following insertion and deletion specification.

### StartBlockId, EndBlockId

StartBlockId and EndBlockId are the same as those of RepoCommandParameter. If either of those in RepoCommandParameter is missing, repo will set them as the Id known for now. For example, if StartBlockId is missing in RepoCommandParameter, StartBlockId in response will be set 0. If EndBlockId is missing in RepoCommandParameter, EndBlockId will be set null untill Repo get FinalBlockId in data packet. If FinalBlockId in returned data packet is less than EndBlockId, the EndBlockId will be set FinalBlockId.

### InsertNum, DeleteNum

InsertNum is used in reponse of insertion status check to indicate how many data packets have been successfully inserted into the repo. DeleteNum is used in response of deletion command and deletion check command. DeleteNum indicates how many data packets have been successfully inserted from repo.

## Repo TLV Type Encoding Number

Type                  | Number

----------------------| -------

RepoCommandParameter  | 201

StartBlockId          | 204

EndBlockId            | 205

ProcessId             | 206

RepoCommandResponse   | 207

StatusCode            | 208

InsertNum             | 209

DeleteNum             | 210

## Repo Trust Model

The trust model of repo depends on people who deploy the repo service, such as PKI. Repo can specify their own verification policies, and data consumers can specify their own trust anchors. The NDN [FAQ](http://named-data.net/project/faq/#How_does_NDN8217s_8220trust_management8221_work) shows how NDN trust managment works.

## Repo Insertion Specification

Repo insertion command requests that the repo retrieve and store content. This command interest is a signed interest and will be validated with access control policy defined by the repo. When the interested is validated and name of the data is not existed in the repo. The repository will reponse with a data object containing OK status and start to send the interest to fetch the data to insert.

Segmented data insertion is also supported in the insertion protocol. Segmantation info is defined in RepoCommandParameter. If the content is segmented, the final segment id will be encoded in this block.

### Insertion command interest semantics

The name semantics follows the format of the repo command. The <command verb> is defined as ``insert``.

For example, for ``<repo prefix>`` as ``/ucla/cs/repo``, the following is an example:

    /ucla/cs/repo/insert/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

### Insertion status check

During the insertion progress, the requester could send insertion status check command to check the status of the insertion progress. This status check command is also signed interest. The semantics of insertion status check command is shown as follows:

<command verb> equals to ``insert check``. For example:

    /ucla/cs/repo/insert check/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

### RepoCommandParameter

RepoCommandParameter of insertion and insertion check command follows that in Repo Command section. Name, Selectors, StartBlockId, EndBlockId are used in insertion. Name and ProcessId are used in insertion check command.

In insertion command, Name represents the name or prefix of data for repo to fetch. If selectors are set, repo will use these selectors to fetch the data. If StartBlockId or EndBlockId is set, the repo will retrieve segmented data with segment number between StartBlockId and EndBlockId. If Repo Insert Command carries both Selectors and StartBlockId, EndBlockId, repo would ignore it and response with statuscode of 405.

In insertion check command, Name represents name or prefix of data for repo to fetch. ProcessId is set by the RepoCommandResponse to indicate specified process.

### Insertatation status response

This insert status data object could be the response data object of both insert command and insert check command. It will follow the format of repo command response.

StatusCode indicates the status of insertion. InsertNum represents how many data has been intersted into the repo. StartBlockId and EndBlockId is the start and end segment id of data inserted. InsertNum is the how many data segments have been inserted. ProcessId indicates an ID of process, a random number gernerated by repo.

For insert cammand, status code will be set according to definition below, StartBlockId and EndBlockId will be set according to the insert command. If StartBlockId in RepoCommandParameter is missing, it will be set 0 in response. If EndBlockId is missing, it will not be set.

For insert check command, status code will be set according to definition below, StartBlockId and EndBlockId will be set according to the StartBlockId and EndBlockId that repo is using, insertnum will be set according to insertion progress. ProcessId is set according to the ID of process checked. If EndBlockId has not been decided, this EndBlockId will not be set in response.

StatusCode Definition:

``StatusCode`` | Description

-------------- | ----------------------------------------------

``100``        | The command is OK. can start to fetch the data

``200``        | All the data has been inserted

``300``        | This insertion is in progress

``401``        | This insertion command or insertion check command is invalidated

``402``        | Selectors and BlockId both present

``403``        | Malformed Command

``404``        | No such this insertion is in progress

``405``        | EndBlockId Missing Timeout

### EndBlockId Missing Timeout

If StartBlockId presents but EndBlockId is missing, and returned data packets do not contain FinalBlockId, the repo will continuosly fetch the data. An EndBlockId missing timeout is set to prevent this ocassion. The repo will start a timer when StartBlockId presents but EndBlockId is missing. When timeout happens, repo will stop fetching data to store and end insert process. If an insert check command arrives during this insert process, the time of timer is set to 0. If data packet containing FinalBlockId arrives, this timeout timer will be dismissed.

### Protocol Process

1. start to authorize the command; if authorization does not fail immediately, go to step 3

2. send a negative response indicating authorization failure, and abort these steps, insert process ends (StatusCode: 401)

3. if  both StartBlockId and EndBlockId are missing, go to step 7

4. if either StartBlockId or EndBlockId is present, and one of supported selectors is present. send negative reponse back and abort steps, insert process ends (StatusCode: 402)

5. if both StartBlockId and EndBlockId are present, and StartBlockId is less than or equal to EndBlockId, go to step 7

6. send a negative response indicating malformed command, and abort these steps, insert process ends (StatusCode: 403)

7. wait for authorization completion

8. if authorization fails, go to step 2 (StatusCode: 401)

9. send a positive response indicating insert is in progress (StatusCode: 200)

10. if either StartBlockId or EndBlockId is present, go to step 16

11. start to retrieve Name with selectors in insert command

12. wait for retrieval completion

13. if retrieval fails, go to step 27

14. store retrieved Data packet

15. abort these steps, insert process ends

16. if StartBlockId is missing, set StartBlockId 0. If EndBlockId is missing, EndBlockId will be missing unless get FinalBlockId in comming data packets, start EndBlockId Misiing Timeout timer.

17. append StartBlockId to Name

18. start to retrieve Name

19. wait for retrieval completion

20. if retrieval fails, go to step 26

21. store retrieved Data packet

22. if retrieved Data packet contains FinalBlockId, and FinalBlockId is less than EndBlockId or EndBlockId is missing, let EndBlockId be FinalBlockId

23. if the last component of Name is greater than or equal to EndBlockId, abort these steps, insert process ends

24. increment the last component of Name

25. go to step 17

26. retrieve data with this data another 2 times. If these 2 retrieval both failes, abort these steps. if success, go to step 21

27. retrieve data with this data another 2 times. If these 2 retrieval both failes, abort these steps. if success, go to step 14

If EndBlockId Missing Timeout timer starts, repo will monitor this timer during step 17~26. If this timeout occurs, abort insert command process immediately.

Implementation MAY pipeline the Interests.

### Repo insert check command progress

Implementation MAY publish a notification of status regarding insert progress. The process of status check is as follows:

1. start to authorize the insert status command, if fails go to 2, if success, go to 3

2. send a negative response indicating authorization failure, and abort these steps (StatusCode: 401)

3. start to check the progress of the insert with the data name in the command. If no such progress is found, go to 4. or go to 5.

4. reponse status with status code, abort check process. (StatusCode: 404)

5. ckeck the status of insertion. return the status of insertion progress. If a EndBlockId Missing Timeour timer is running, set this timer to 0. About check process. (StatusCode: 300)

### Protocol diagram:

    Requester                     Repo                          Data producer

        |                           |                                 |

        |                           |                                 |

      +---+  Insert command       +---+                               |

      |   | --------------------> |   |                               |

      +---+                       |   |                               |

        |                         |   |                               |

      +---+   Confirm start       |   |                               |

      |   | <==================== |   |                               |

      +---+   Reject command      +---+                               |

        |     (with status code)    |                                 |

        |                         +---+     Interest for Data       +---+

        |                         |   | --------------------------> |   |

        |                         +---+                             |   |

        |                           |                               |   |

        |                         +---+       Data segment          |   |

        |                         |   | <========================== |   |

        |                         +---+                             +---+

        |                           |                                 |

        |                           ~                                 ~

        |                           ~                                 ~

        |                           |                                 |

        |                         +---+     Interest for Data       +---+

        |                         |   | --------------------------> |   |

        |                         +---+                             |   |

        |                           |                               |   |

        |                         +---+       Data segment          |   |

        |                         |   | <========================== |   |

        |                         +---+                             +---+

        |                           |                                 |

        |                           |                                 |

        |                           ~                                 ~

        |                           ~                                 ~

        |                           |                                 |

        |                           |                                 |

        |                           |                                 |

      +---+   Status interest     +---+                               |

      |   | --------------------> |   |                               |

      +---+                       |   |                               |

        |                         |   |                               |

      +---+    Status response    |   |                               |

      |   | <==================== |   |                               |

      +---+                       +---+                               |

        |                           |                                 |

        |                           |                                 |

## Repo Deletion Specification

Deletion of one content object or content objects under certain prefix are both supported in repo specification. Selectors are used to select multiple content objects. These selectors are different from conventional [selectors](http://named-data.net/doc/ndn-tlv/interest.html#selectors) of interest packet. The conventional selecors of interest would select one data packet of repo. Selecors of delete command will return any data packets which applies for these selectors. Deletion of segmented data is also supported.

### Deletion command interest semantics

Deletion command interest follows the format of the repo command. For example:

    /ucla/cs/repo/delete/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue

### Deletion Command RepoCommandParameter

Format of deletion command RepoCommandParameter follows that of Repo Command. Name, Selector, StartBlockId, EndBlockId, ProcessId are used to construct RepoCommandParameter.

Name is the name or the prefix of data repo will delete.

Selector is used to select data. ChildSelector is **not** supported in deletion command.

StartBlockId and EndBlockId are used to delete segment data. Data of segment id between StartBlockId and EndBlockId will be deleted by repo.

ProcessId is a random number generated by client to indicate the deletion process. Repo will match this ProcessId with this deletion process.

### Deletion Command Selectors

Deletion command selectors follows the [repo command selectors](http://redmine.named-data.net/projects/repo-ng/wiki/Repo_Protocol_Specification#Repo-Command-Selectors).

### Deletion status check

During the deletion progress, the requester could send deletion status check command to check the status of the deletion progress. This status check command is also signed interest. The semantics of deletion status check command follows the format of repo command. ``<command verb>`` equals to ``delete check``. For example:

    /ucla/cs/repo/delete/<RepoCommandParameter>/<timestamp>/<random-value>/<SignatureInfo>/<SignatureValue>

### Deletion status check RepoCommandParameter

Name and ProcessId are used to indicate certain deletion process. Repo uses Name and ProcessId to match certain deletion process. If it matches, repo will respond progress of deletion. If just one of Name or ProcessId matches, it fails.

### Deletion Check Command Selectors

Selectors are not supported. Repo will ignore selectors when processing deletion status check command.

### Deletion status response

Deletion status response is used as response of repo deletion command and repo deletion check command.

Name, StatusCode, Selector, StartBlockId, EndBlockId, ProcessId, DeletenNum are used in deletion status response. Name, ProcessId, Selector is the same as that of delete command. StatusCode indicates the status of deletion. DeleteNum indicates how many data packets have been deleted.

In Repo Deletion Command, if just StartBlockId presents, EndBlockId may be the largest segment ID repo holds. If just EndBlockId presents, StartBlockId will be set 0

StatusCode Definition:

``StatusCode`` | Description

-------------- | --------------------------------------

``200``        | All the data has been deleted

``300``        | This deletion is in progress

``401``        | This deletion or deletion check is invalidated

``402``        | Selectors and BlockId both present

``403``        | Malformed Command

``404``        | No such this deletion is in progress

### Protocol process

1. start to authorize the command; if authorization does not fail, go to step 3

2. send a negative response indicating authorization failure, and abort these steps, end deletion process. (StatusCode: 401)

3. check whether a deletion process of same RepoCommandParameter exists, waiting for deletion process ends.

4. If selectors and one of StartBlockId and EndBlockId presents, send a negative response and abort these steps, end deletion process. (StatusCode: 402)

5. If selectors present, go to step 8

6. check whether StartBlockId or EndBlockId presents. If both presents but StartBlockId is larger than EndBlockId, return negative response and end deletion process. (StatusCode: 403) Or go to step 9

7. If StartBlockId, EndBlockId and selectors are all missing, go to step 10

8. delete all the data that conforms to the name and selectors, go to step 11

9. delete all the data packets of segment id between StartBlockId and EndBlockId. If StartBlockId is missing, StartBlockId is set to be 0. If EndBlockId is missing, EndBlockId is set to be the largest segment id that repo holds. go to step 11

10. delete all the data with prefix same as name. got to step 11

11. If lifetime of interest does not expire, return status response of positive statuscode. If lifetime of interest has expired, wait for interest the same RepoCommandParameter and return this status response. End Deletion process. (StatusCode: 200)

Client will set deletion command with big lifetime. If life time expires, client will re-express the command.

Implementation MAY publish a notification of status regarding delete progress. The process of status check is as follows:

1. start to authorize the delete status command

2. send a negative response indicating authorization failure, and abort these steps (StatusCode: 401)

3. start to check the progress of the delete with the data name in the command. If no such progress is found, go to 4. or go to 5.

4. reponse status with status code of 404 (StatusCode: 404)

5. ckeck te status of delete. return the status data content (StatusCode: 300)

### Protocol diagram:

    Requester                     Repo 

        |                           |                                 

        |                           |                                 

      +---+  Delete command       +---+                               

      |   | --------------------> |   |                               

      +---+                       +---+                               

        |                           |                                 

        |                           |                                 

        |                           |                                 

      +---+   Status interest     +---+                               

      |   | --------------------> |   |                               

      +---+                       |   |                               

        |                         |   |                               

      +---+    Status response    |   |                               

      |   | <==================== |   |                               

      +---+                       +---+                               

        |                           |                                 

        |                           |                                 

        |                           |                                 

      +---+   Confirm Deletion    +---+                               

      |   | <==================== |   |                               

      +---+   Reject command      +---+                               

        |     (with status code)    |    

        |                           |