Project

General

Profile

Actions

Task #1438

closed

nfd-status: Define format and implement --xml option

Added by Alex Afanasyev over 10 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Tools
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:

Description

With --xml option, nfd-status should produce XML output, which can be XSLT'ed to something else.

Part of this task is to propose a specific XML schema, which should be approved before nfd-status modifications.


Files

nfd-status.xsd (4.73 KB) nfd-status.xsd Chengyu Fan, 04/08/2014 03:44 PM
nfd-status-v1.0.xsd (3.31 KB) nfd-status-v1.0.xsd Chengyu Fan, 04/09/2014 01:51 PM
nfd-status-v1.1.xsd (3.21 KB) nfd-status-v1.1.xsd Chengyu Fan, 04/11/2014 11:31 AM
nfd-status-v1.2.xsd (2.91 KB) nfd-status-v1.2.xsd Chengyu Fan, 04/11/2014 03:43 PM
nfd-status-v1.2.xml (1.83 KB) nfd-status-v1.2.xml Junxiao Shi, 04/13/2014 11:22 AM
nfd-status-v1.3.xsd (2.93 KB) nfd-status-v1.3.xsd Chengyu Fan, 04/14/2014 11:55 AM
nfd-status-v1.4.xsd (2.93 KB) nfd-status-v1.4.xsd Chengyu Fan, 04/14/2014 10:16 PM
nfd-status-v1.4.xml (2.24 KB) nfd-status-v1.4.xml Junxiao Shi, 04/16/2014 04:43 PM
Actions #1

Updated by Chengyu Fan over 10 years ago

  • Assignee set to Chengyu Fan
Actions #2

Updated by Chengyu Fan over 10 years ago

For startTime and currentTime in nfd status info, the format is as below:

startTime=20140408T163048.785000

I'm curious why there is a "T" in the middle? A separator?

Actions #3

Updated by Alex Afanasyev over 10 years ago

Yes, it is a separator (between date and time). It has to do something with ISO datetime format, but it is actually not exactly the ISO format.

Actions #4

Updated by Chengyu Fan over 10 years ago

The attached is the naive design for XML schema of nfd-status.
I'm not very familiar with XMl, please check the attached schema and give me some clues on how to improve it.

Actions #5

Updated by Junxiao Shi over 10 years ago

Comments on nfd-status.xsd (SHA256 hash 8e1ee2c02f2af0d54694b9f8deeb3dc10e268ad8932ae3154b235b1b49e9d663):

  • startTime and currentTime elements should use dateTime primitive type, not string with pattern
  • typo: nInInerests and nOutInerests
  • define a packetCounters type which has the same structure of faces/face/counters, and use this type for both forwarder-level packet counters and face packet counters
  • faceid should be renamed to faceId, and use nonNegativeInteger type
  • remote and local should be renamed to remoteUri and localUri
Actions #6

Updated by Chengyu Fan over 10 years ago

The dateTime format uses "YYYY-MM-DDThh:mm:ss", but the nfd-status produces startTime like "YYMMDDTsecs.usecs".
Do we need to keep exactly the same format as nfd-status?

Actions #7

Updated by Alex Afanasyev over 10 years ago

Whatever right now is in nfd-status was just a quick hack with whatever was available at the moment. We probably want to modify nfd-status at some point to show normal representation (even without T in the middle, may be in locale-specific).

For xml output we should use the standard datetime, since it is what XML parsers (XSLT converters) understand and can make use of. For that, I actually suggest you to extend ndn-cpp-dev library (in particular src/util/time.hpp) by adding a new method(s) to format strings. You can take current toIsoString as an example, which uses boost::posix_time to do actual formatting (you may need to check in the docs how to do formatting).

Actions #8

Updated by Chengyu Fan over 10 years ago

Junxiao Shi wrote:

Comments on nfd-status.xsd (SHA256 hash 8e1ee2c02f2af0d54694b9f8deeb3dc10e268ad8932ae3154b235b1b49e9d663):

  • startTime and currentTime elements should use dateTime primitive type, not string with pattern
  • typo: nInInerests and nOutInerests
  • define a packetCounters type which has the same structure of faces/face/counters, and use this type for both forwarder-level packet counters and face packet counters
  • faceid should be renamed to faceId, and use nonNegativeInteger type
  • remote and local should be renamed to remoteUri and localUri

Do you mean the the packetCounters type for both face and general NFD status?

Actions #9

Updated by Chengyu Fan over 10 years ago

Alex Afanasyev wrote:

Whatever right now is in nfd-status was just a quick hack with whatever was available at the moment. We probably want to modify nfd-status at some point to show normal representation (even without T in the middle, may be in locale-specific).

For xml output we should use the standard datetime, since it is what XML parsers (XSLT converters) understand and can make use of. For that, I actually suggest you to extend ndn-cpp-dev library (in particular src/util/time.hpp) by adding a new method(s) to format strings. You can take current toIsoString as an example, which uses boost::posix_time to do actual formatting (you may need to check in the docs how to do formatting).

So the newly added function should return the dateTime format exactly as "YYYY-MM-DDThh:mm:ss.SSSSSS"? And how about the ID to submit this change?

Actions #10

Updated by Alex Afanasyev over 10 years ago

Are you asking about ChangeID? It suppose to be automatically generated. If it wasn't, during push gerrit will complain and will give a hint of the command to run to enable the automatic ChangeId generation. After you run the script, just amend the commit and ChangeId will appear.

Actions #11

Updated by Chengyu Fan over 10 years ago

No, what I'm asking is the task #.
Since I need to change both the ndn-cpp-dev and nfd, I'm curious that do I use the same task # 1438 for both ndn-cpp-dev and nfd?

Actions #12

Updated by Alex Afanasyev over 10 years ago

Oh that. You can always create a task yourself, but I will create right now for you. (And it is not strictly necessary to have task in the first place, but it is good to have.)

Actions #13

Updated by Chengyu Fan over 10 years ago

the xml schema for nfd-status v1.0. correct the naming and typo as Junxiao suggested, create a packetCounters type.

Actions #14

Updated by Alex Afanasyev over 10 years ago

Looks good to me. The only thing that may or may not be changed is

<xs:element type="xs:string" name="prefix"/>

We can actually use xs:anyURI type, since prefix also kind of in URI form, though not exactly.

Btw. Don't forget to include this schema somewhere in documentation (docs/ folder). I think it should go to docs/_static and then included in one or several documentation files. Here is an example how include works in sphinx:

.. literalinclude:: ../../examples/ndn-simple.cc
    :language: c++
    :linenos:
    :lines: 20-27,48-
    :emphasize-lines: 30-33,37-49

(Last three lines are optional; xsd language should exist, if not, you can try xml)

Actions #15

Updated by Junxiao Shi over 10 years ago

  • xs:element declarations except nfdStatus should be changed to xs:complexType or xs:simpleType declarations, because only nfdStatus is allowed to appear as the root element
  • nInInterests and nOutInterests should be merged into nInterests, because the parent element (incomingPackets or outgoingPackets) already states the direction; same applies to nInDatas and nOutDatas. alternately, n(In|Out)(Interests|Datas) can appear directly under packetCounters
  • uptime is redundant; if you insist to include it, it should be xs:duration primitive type
  • rename hop to nextHop
  • rename fib to fibEntry
  • rename fibs to fib
Actions #16

Updated by Chengyu Fan over 10 years ago

Sorry I don't get the message you updated:

xs:element declarations except nfdStatus should be changed to xs:complexType or xs:simpleType declarations, because only nfdStatus is allowed to appear as the root element

Do you mean I'd better nest the elements declaration in the context, like the previous submission?
Or do you mean that I need to just use xs:complexType and xs:simpleType to declare our types without using xsd:element?

Actions #17

Updated by Junxiao Shi over 10 years ago

See http://www.w3.org/TR/2001/REC-xmlschema-0-20010502/#po.xsd on how to define and reference a named complexType.

Actions #18

Updated by Chengyu Fan over 10 years ago

modified as Junxiao suggested

Actions #19

Updated by Junxiao Shi over 10 years ago

Comments on nfd-status-v1.1.xsd SHA256=7d5acb891a4e92902ee4d2857a70fc6cec19a61487e74d59bbe75b4b5829a2df:

  • incomingPacketsType and outgoingPacketsType are the same. They should be merged as unidirectionalPacketCountersType.
  • packetCountersType can be renamed as bidirectionalPacketCountersType, to be analog to unidirectionalPacketCountersType.
  • nInterests and nDatas can be nested under unidirectionalPacketCountersType, instead of defining separate simpleTypes, because these types are same as nonNegativeInteger and have no additional restriction.
  • Be consistent on whether name or type attribute appears first in <xs:element> element.
  • Subelements in face element should be defined as faceType, and face element can reference this type.
Actions #20

Updated by Chengyu Fan over 10 years ago

  • incomingPacketsType and outgoingPacketsType are the same. They should be merged as unidirectionalPacketCountersType.

how about add a new element called "<direction>" in this unidirectionalPacketCountersType, restrict it to "incoming/outgoing"? Because we need a field to represent the packet direction.
The latter bidirectionalPacketCountersType could fix the minOccurs and maxOccurs as 2, because this field must contain incoming packets and outgoing packets.

Actions #21

Updated by Junxiao Shi over 10 years ago

<direction> element is redundant. unidirectionalPacketCountersType has one element per packet type. The element name represents direction.

XML example:

<packetCounters>
  <incoming>
    <interest>60</interest>
    <data>30</data>
  </incoming>
  <outgoing>
    <interest>35</interest>
    <data>50</data/>
  </outgoing>
</packetCounters>

incoming and outgoing elements have same type but different names.

Actions #22

Updated by Chengyu Fan over 10 years ago

I see. you are right. I will correct it.

Actions #24

Updated by Junxiao Shi over 10 years ago

Comment for nfd-status-v1.2.xsd SHA256=5430ff1bc62f88cd9a39c933d161df6505b0cc3c0f0521c2ac8d297857256ce5:

  • targetNamespace="http://www.named-data.net" is too broad. What about ndn:/localhost/nfd/status/1?
  • rename hopCost to cost
  • set minOccurs to zero for face, fibEntry, and nextHop elements: we might need it someday

I'm attaching an example document that validates on current schema.

xmllint --schema nfd-status-v1.2.xsd nfd-status-v1.2.xml

Actions #25

Updated by Chengyu Fan over 10 years ago

  • targetNamespace="http://www.named-data.net" is too broad. What about ndn:/localhost/nfd/status/1?

I have no problem with the prefix, and I guess that the number "1" in the targetNamespace represents the childSelector=1. But I think it is confusing for someone does not know this.
How about change it as ndn:/localhost/nfd/status/childSelector/1?

BTW, in nfd-status, the faces uses prefix "/localhost/nfd/faces/list" and "/localhost/nfd/fib/list". I'm curious that don't they belong to status category?

  • set minOccurs to zero for face, fibEntry, and nextHop elements: we might need it someday

According to the nfd-status command, it seems like the faces and fib may not show if we don't use options like "-f", and "-b". Not sure if this is what you are concerning...
If it is the case, since another option "-v" is also used, do we care the case that version information is empty?

And I don't get why the nextHop can be empty: I think if there is an fibEntry, there should be at least one nextHop, isn't it? (ex: /localhost/nfd nexthops={faceid=1 (cost=0)})

Actions #26

Updated by Junxiao Shi over 10 years ago

The number "1" in suggested targetNamespace is completely unrelated to ChildSelector. It's a revision number.

When a backwards-incompatible change is made to the XML Schema, this number is incremented.

When nfd-status is set to XML output, it should ignore -v -f -b options, and always return a whole document.

The protocol doesn't forbid a StatusDataset to contain zero block, so face and fibEntry should have minOccurs="0", although this won't happen in reality.

I have checked that FibMgmt protocol requires at least one nextHop, thus nextHop can have minOccurs="1".

Actions #28

Updated by Junxiao Shi over 10 years ago

targetNamespace is still incorrect. The scheme name, "ndn:", cannot be omitted.

targetNamespace="ndn:/localhost/nfd/status/1"
Actions #29

Updated by Chengyu Fan over 10 years ago

Sorry I made such mistake.

Just to make sure: for the next step, the nfd-status needs to produce the xml output when user specifies.
Then the caller can use this xml output to generate result in other format. (ex: httpserver can then convert the xml output into xhtml)
Correct?

Actions #30

Updated by Junxiao Shi over 10 years ago

This XML schema looks good.

Actions #31

Updated by Chengyu Fan over 10 years ago

Do you have any suggestions that which library to use to generate the xml stream?
Is the Boost.PropertyTree a good choice? Or do you have any better choice?

Actions #32

Updated by Alex Afanasyev over 10 years ago

Personally, I wouldn't use any library. Would just generate raw xml output using std::ostream. Any library would force to have unnecessary overhead (the whole document will be first prepared in memory, and then converted to text, and then you will be able to send it out).

Actions #33

Updated by Junxiao Shi over 10 years ago

It's tricky to generate XML by hand. Problem will arise when someone decides to have a '&' in FaceUri: you'll need to use entity encoding.

There must be a library that can write XML to a stream without preparing whole document in memory. In .NET we have System.Xml.XmlWriter; C++ should have something similar.

Actions #34

Updated by Chengyu Fan over 10 years ago

It seems like tinyxml2 can generate the xml stream without creating a document first. However, it is an additional library.
If it is not suggested to use additional library, I will start this task using std::ostream.

Actions #35

Updated by Alex Afanasyev over 10 years ago

I don't want to have extra dependency, especially when it is for "generation" of xml to be used in a small support tool. If boost cannot help with generating escaped data for XML, then I would prefer dealing with it manually and use std::ostream.

Actions #36

Updated by Chengyu Fan over 10 years ago

Okay, I will start using std::ostream.

Actions #37

Updated by Chengyu Fan over 10 years ago

The remote URI "ether://01:00:5e:00:17:aa" is not a valid value of xs:anyURI. And I did not find any specification to describe it.
One similar one I found is IPv6: "http://[1080:0:0:0:8:800:200C:4171]". Can I use the same way to represent the ether URI in xml? I may need to modify ndn-cpp-dev/management/nfd-fib-entry.hpp to add such URI format.

The duration time generated "x seconds" format, which is not a valid value for Duration Data Type in xsd either. I'm planning to modify src/util/time.hpp as well.

Do you guy have better way to do it?

Actions #38

Updated by Alex Afanasyev over 10 years ago

What should be the duration's format?

Actions #39

Updated by Chengyu Fan over 10 years ago

The time interval is specified in the form "PnYnMnDTnHnMnS".
For example: P5Y2M10DT15H

Actions #40

Updated by Alex Afanasyev over 10 years ago

Yes. This would require new implementation.

Actions #41

Updated by Junxiao Shi over 10 years ago

The FaceUri ether://01:00:5e:00:17:aa is not valid.
I'm fine with changing it to ether://[01:00:5e:00:17:aa].

The change should be in core/face-uri.cpp: set m_isV6 = true;.
Regular expressions used for parsing need to be changed as well.

XML-Schema standard has no restriction on the range of a unit in duration.
PT193360.145S is a valid duration (confirmed with xmllint).
It's unnecessary to calculate year/month/day (this avoids the problems of daylight saving time adjustment, 28/29/30/31 days in a month, leap second, etc).

Actions #42

Updated by Chengyu Fan over 10 years ago

If PT193360.145S is allowed, I'll just leave it there.

In core/face-uri.cpp, there is no m_v6, I think you are talking about m_isV6, which is the one for ether FaceUri.

Actions #43

Updated by Chengyu Fan over 10 years ago

Questions:

To include the schema in documentation, do I need to include the code below in the index.rst file?

.. literalinclude:: _static/nfd-status.xsd
:language: c++

Then do I still use ./waf to compile it? How can I check if it works?

To generate raw xml output, Alex suggested std::ostream. Does that mean the tool can accept the file name to store the xml data, and if no file name provided, it just print out to the std::cout?

Actions #44

Updated by Junxiao Shi over 10 years ago

The tool should always print the XML to stdout.

NFD status is volatile information and doesn't worth writing to a file. If someone wants it into a file, she can use redirection which is offered by major shells.

Actions #45

Updated by Junxiao Shi over 10 years ago

Remember to set correct Task Status and % Done, if you are working on the code.

Actions #46

Updated by Alex Afanasyev over 10 years ago

Create a separate .rst file that would have the literalinclude line you mentioned. After that you will just need to include this new file into toc section of index.rst.

To build documentation ./waf docs (or ./waf sphinx) can be used. However, the docs commit (http://gerrit.named-data.net/#/c/658/) is not yet merged, so I would suggest waiting a little bit (I will try to resolve the issues and merge it today).

Actions #47

Updated by Chengyu Fan over 10 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 80
Actions #48

Updated by Chengyu Fan over 10 years ago

  • Status changed from In Progress to Code review
  • % Done changed from 80 to 100
Actions #49

Updated by Chengyu Fan over 10 years ago

  • Status changed from Code review to Closed
Actions

Also available in: Atom PDF