Task #1438
closednfd-status: Define format and implement --xml option
100%
Description
With --xml
option, nfd-status should produce XML output, which can be XSLT'ed to something else.
Part of this task is to propose a specific XML schema, which should be approved before nfd-status modifications.
Files
Updated by Chengyu Fan over 10 years ago
For startTime and currentTime in nfd status info, the format is as below:
startTime=20140408T163048.785000
I'm curious why there is a "T" in the middle? A separator?
Updated by Alex Afanasyev over 10 years ago
Yes, it is a separator (between date and time). It has to do something with ISO datetime format, but it is actually not exactly the ISO format.
Updated by Chengyu Fan over 10 years ago
- File nfd-status.xsd nfd-status.xsd added
The attached is the naive design for XML schema of nfd-status.
I'm not very familiar with XMl, please check the attached schema and give me some clues on how to improve it.
Updated by Junxiao Shi over 10 years ago
Comments on nfd-status.xsd (SHA256 hash 8e1ee2c02f2af0d54694b9f8deeb3dc10e268ad8932ae3154b235b1b49e9d663):
- startTime and currentTime elements should use dateTime primitive type, not string with pattern
- typo: nInInerests and nOutInerests
- define a packetCounters type which has the same structure of faces/face/counters, and use this type for both forwarder-level packet counters and face packet counters
- faceid should be renamed to faceId, and use nonNegativeInteger type
- remote and local should be renamed to remoteUri and localUri
Updated by Chengyu Fan over 10 years ago
The dateTime format uses "YYYY-MM-DDThh:mm:ss", but the nfd-status produces startTime like "YYMMDDTsecs.usecs".
Do we need to keep exactly the same format as nfd-status?
Updated by Alex Afanasyev over 10 years ago
Whatever right now is in nfd-status was just a quick hack with whatever was available at the moment. We probably want to modify nfd-status at some point to show normal representation (even without T in the middle, may be in locale-specific).
For xml output we should use the standard datetime, since it is what XML parsers (XSLT converters) understand and can make use of. For that, I actually suggest you to extend ndn-cpp-dev library (in particular src/util/time.hpp) by adding a new method(s) to format strings. You can take current toIsoString as an example, which uses boost::posix_time to do actual formatting (you may need to check in the docs how to do formatting).
Updated by Chengyu Fan over 10 years ago
Junxiao Shi wrote:
Comments on nfd-status.xsd (SHA256 hash 8e1ee2c02f2af0d54694b9f8deeb3dc10e268ad8932ae3154b235b1b49e9d663):
- startTime and currentTime elements should use dateTime primitive type, not string with pattern
- typo: nInInerests and nOutInerests
- define a packetCounters type which has the same structure of faces/face/counters, and use this type for both forwarder-level packet counters and face packet counters
- faceid should be renamed to faceId, and use nonNegativeInteger type
- remote and local should be renamed to remoteUri and localUri
Do you mean the the packetCounters type for both face and general NFD status?
Updated by Chengyu Fan over 10 years ago
Alex Afanasyev wrote:
Whatever right now is in nfd-status was just a quick hack with whatever was available at the moment. We probably want to modify nfd-status at some point to show normal representation (even without T in the middle, may be in locale-specific).
For xml output we should use the standard datetime, since it is what XML parsers (XSLT converters) understand and can make use of. For that, I actually suggest you to extend ndn-cpp-dev library (in particular src/util/time.hpp) by adding a new method(s) to format strings. You can take current toIsoString as an example, which uses boost::posix_time to do actual formatting (you may need to check in the docs how to do formatting).
So the newly added function should return the dateTime format exactly as "YYYY-MM-DDThh:mm:ss.SSSSSS"? And how about the ID to submit this change?
Updated by Alex Afanasyev over 10 years ago
Are you asking about ChangeID? It suppose to be automatically generated. If it wasn't, during push gerrit will complain and will give a hint of the command to run to enable the automatic ChangeId generation. After you run the script, just amend the commit and ChangeId will appear.
Updated by Chengyu Fan over 10 years ago
No, what I'm asking is the task #.
Since I need to change both the ndn-cpp-dev and nfd, I'm curious that do I use the same task # 1438 for both ndn-cpp-dev and nfd?
Updated by Alex Afanasyev over 10 years ago
Oh that. You can always create a task yourself, but I will create right now for you. (And it is not strictly necessary to have task in the first place, but it is good to have.)
Updated by Chengyu Fan over 10 years ago
- File nfd-status-v1.0.xsd nfd-status-v1.0.xsd added
the xml schema for nfd-status v1.0. correct the naming and typo as Junxiao suggested, create a packetCounters type.
Updated by Alex Afanasyev over 10 years ago
Looks good to me. The only thing that may or may not be changed is
<xs:element type="xs:string" name="prefix"/>
We can actually use xs:anyURI
type, since prefix also kind of in URI form, though not exactly.
Btw. Don't forget to include this schema somewhere in documentation (docs/
folder). I think it should go to docs/_static
and then included in one or several documentation files. Here is an example how include works in sphinx:
.. literalinclude:: ../../examples/ndn-simple.cc
:language: c++
:linenos:
:lines: 20-27,48-
:emphasize-lines: 30-33,37-49
(Last three lines are optional; xsd language should exist, if not, you can try xml)
Updated by Junxiao Shi over 10 years ago
- xs:element declarations except nfdStatus should be changed to xs:complexType or xs:simpleType declarations, because only nfdStatus is allowed to appear as the root element
- nInInterests and nOutInterests should be merged into nInterests, because the parent element (incomingPackets or outgoingPackets) already states the direction; same applies to nInDatas and nOutDatas. alternately, n(In|Out)(Interests|Datas) can appear directly under packetCounters
- uptime is redundant; if you insist to include it, it should be xs:duration primitive type
- rename hop to nextHop
- rename fib to fibEntry
- rename fibs to fib
Updated by Chengyu Fan over 10 years ago
Sorry I don't get the message you updated:
xs:element declarations except nfdStatus should be changed to xs:complexType or xs:simpleType declarations, because only nfdStatus is allowed to appear as the root element
Do you mean I'd better nest the elements declaration in the context, like the previous submission?
Or do you mean that I need to just use xs:complexType and xs:simpleType to declare our types without using xsd:element?
Updated by Junxiao Shi over 10 years ago
See http://www.w3.org/TR/2001/REC-xmlschema-0-20010502/#po.xsd on how to define and reference a named complexType.
Updated by Chengyu Fan over 10 years ago
- File nfd-status-v1.1.xsd nfd-status-v1.1.xsd added
modified as Junxiao suggested
Updated by Junxiao Shi over 10 years ago
Comments on nfd-status-v1.1.xsd SHA256=7d5acb891a4e92902ee4d2857a70fc6cec19a61487e74d59bbe75b4b5829a2df:
- incomingPacketsType and outgoingPacketsType are the same. They should be merged as unidirectionalPacketCountersType.
- packetCountersType can be renamed as bidirectionalPacketCountersType, to be analog to unidirectionalPacketCountersType.
- nInterests and nDatas can be nested under unidirectionalPacketCountersType, instead of defining separate simpleTypes, because these types are same as nonNegativeInteger and have no additional restriction.
- Be consistent on whether name or type attribute appears first in <xs:element> element.
- Subelements in face element should be defined as faceType, and face element can reference this type.
Updated by Chengyu Fan over 10 years ago
- incomingPacketsType and outgoingPacketsType are the same. They should be merged as unidirectionalPacketCountersType.
how about add a new element called "<direction>" in this unidirectionalPacketCountersType, restrict it to "incoming/outgoing"? Because we need a field to represent the packet direction.
The latter bidirectionalPacketCountersType could fix the minOccurs and maxOccurs as 2, because this field must contain incoming packets and outgoing packets.
Updated by Junxiao Shi over 10 years ago
<direction> element is redundant. unidirectionalPacketCountersType has one element per packet type. The element name represents direction.
XML example:
<packetCounters>
<incoming>
<interest>60</interest>
<data>30</data>
</incoming>
<outgoing>
<interest>35</interest>
<data>50</data/>
</outgoing>
</packetCounters>
incoming and outgoing elements have same type but different names.
Updated by Chengyu Fan over 10 years ago
I see. you are right. I will correct it.
Updated by Chengyu Fan over 10 years ago
- File nfd-status-v1.2.xsd nfd-status-v1.2.xsd added
Updated by Junxiao Shi over 10 years ago
- File nfd-status-v1.2.xml nfd-status-v1.2.xml added
Comment for nfd-status-v1.2.xsd SHA256=5430ff1bc62f88cd9a39c933d161df6505b0cc3c0f0521c2ac8d297857256ce5:
targetNamespace="http://www.named-data.net"
is too broad. What aboutndn:/localhost/nfd/status/1
?- rename hopCost to cost
- set minOccurs to zero for face, fibEntry, and nextHop elements: we might need it someday
I'm attaching an example document that validates on current schema.
xmllint --schema nfd-status-v1.2.xsd nfd-status-v1.2.xml
Updated by Chengyu Fan over 10 years ago
targetNamespace="http://www.named-data.net"
is too broad. What aboutndn:/localhost/nfd/status/1
?
I have no problem with the prefix, and I guess that the number "1" in the targetNamespace represents the childSelector=1. But I think it is confusing for someone does not know this.
How about change it as ndn:/localhost/nfd/status/childSelector/1
?
BTW, in nfd-status, the faces uses prefix "/localhost/nfd/faces/list" and "/localhost/nfd/fib/list". I'm curious that don't they belong to status category?
- set minOccurs to zero for face, fibEntry, and nextHop elements: we might need it someday
According to the nfd-status command, it seems like the faces and fib may not show if we don't use options like "-f", and "-b". Not sure if this is what you are concerning...
If it is the case, since another option "-v" is also used, do we care the case that version information is empty?
And I don't get why the nextHop can be empty: I think if there is an fibEntry, there should be at least one nextHop, isn't it? (ex: /localhost/nfd nexthops={faceid=1 (cost=0)})
Updated by Junxiao Shi over 10 years ago
The number "1" in suggested targetNamespace is completely unrelated to ChildSelector. It's a revision number.
When a backwards-incompatible change is made to the XML Schema, this number is incremented.
When nfd-status is set to XML output, it should ignore -v -f -b options, and always return a whole document.
The protocol doesn't forbid a StatusDataset to contain zero block, so face and fibEntry should have minOccurs="0", although this won't happen in reality.
I have checked that FibMgmt protocol requires at least one nextHop, thus nextHop can have minOccurs="1".
Updated by Chengyu Fan over 10 years ago
- File nfd-status-v1.3.xsd nfd-status-v1.3.xsd added
Updated by Junxiao Shi over 10 years ago
targetNamespace is still incorrect. The scheme name, "ndn:", cannot be omitted.
targetNamespace="ndn:/localhost/nfd/status/1"
Updated by Chengyu Fan over 10 years ago
- File nfd-status-v1.4.xsd nfd-status-v1.4.xsd added
Sorry I made such mistake.
Just to make sure: for the next step, the nfd-status needs to produce the xml output when user specifies.
Then the caller can use this xml output to generate result in other format. (ex: httpserver can then convert the xml output into xhtml)
Correct?
Updated by Chengyu Fan over 10 years ago
Do you have any suggestions that which library to use to generate the xml stream?
Is the Boost.PropertyTree a good choice? Or do you have any better choice?
Updated by Alex Afanasyev over 10 years ago
Personally, I wouldn't use any library. Would just generate raw xml output using std::ostream. Any library would force to have unnecessary overhead (the whole document will be first prepared in memory, and then converted to text, and then you will be able to send it out).
Updated by Junxiao Shi over 10 years ago
It's tricky to generate XML by hand. Problem will arise when someone decides to have a '&' in FaceUri: you'll need to use entity encoding.
There must be a library that can write XML to a stream without preparing whole document in memory. In .NET we have System.Xml.XmlWriter; C++ should have something similar.
Updated by Chengyu Fan over 10 years ago
It seems like tinyxml2 can generate the xml stream without creating a document first. However, it is an additional library.
If it is not suggested to use additional library, I will start this task using std::ostream.
Updated by Alex Afanasyev over 10 years ago
I don't want to have extra dependency, especially when it is for "generation" of xml to be used in a small support tool. If boost cannot help with generating escaped data for XML, then I would prefer dealing with it manually and use std::ostream.
Updated by Chengyu Fan over 10 years ago
Okay, I will start using std::ostream.
Updated by Chengyu Fan over 10 years ago
The remote URI "ether://01:00:5e:00:17:aa" is not a valid value of xs:anyURI. And I did not find any specification to describe it.
One similar one I found is IPv6: "http://[1080:0:0:0:8:800:200C:4171]". Can I use the same way to represent the ether URI in xml? I may need to modify ndn-cpp-dev/management/nfd-fib-entry.hpp to add such URI format.
The duration time generated "x seconds" format, which is not a valid value for Duration Data Type in xsd either. I'm planning to modify src/util/time.hpp as well.
Do you guy have better way to do it?
Updated by Alex Afanasyev over 10 years ago
What should be the duration's format?
Updated by Chengyu Fan over 10 years ago
The time interval is specified in the form "PnYnMnDTnHnMnS".
For example: P5Y2M10DT15H
Updated by Alex Afanasyev over 10 years ago
Yes. This would require new implementation.
Updated by Junxiao Shi over 10 years ago
- File nfd-status-v1.4.xml nfd-status-v1.4.xml added
The FaceUri ether://01:00:5e:00:17:aa
is not valid.
I'm fine with changing it to ether://[01:00:5e:00:17:aa]
.
The change should be in core/face-uri.cpp
: set m_isV6 = true;
.
Regular expressions used for parsing need to be changed as well.
XML-Schema standard has no restriction on the range of a unit in duration.
PT193360.145S
is a valid duration (confirmed with xmllint
).
It's unnecessary to calculate year/month/day (this avoids the problems of daylight saving time adjustment, 28/29/30/31 days in a month, leap second, etc).
Updated by Chengyu Fan over 10 years ago
If PT193360.145S is allowed, I'll just leave it there.
In core/face-uri.cpp, there is no m_v6, I think you are talking about m_isV6, which is the one for ether FaceUri.
Updated by Chengyu Fan over 10 years ago
Questions:
To include the schema in documentation, do I need to include the code below in the index.rst file?
.. literalinclude:: _static/nfd-status.xsd
:language: c++
Then do I still use ./waf to compile it? How can I check if it works?
To generate raw xml output, Alex suggested std::ostream. Does that mean the tool can accept the file name to store the xml data, and if no file name provided, it just print out to the std::cout?
Updated by Junxiao Shi over 10 years ago
The tool should always print the XML to stdout.
NFD status is volatile information and doesn't worth writing to a file. If someone wants it into a file, she can use redirection which is offered by major shells.
Updated by Junxiao Shi over 10 years ago
Remember to set correct Task Status and % Done, if you are working on the code.
Updated by Alex Afanasyev over 10 years ago
Create a separate .rst
file that would have the literalinclude line you mentioned. After that you will just need to include this new file into toc
section of index.rst
.
To build documentation ./waf docs
(or ./waf sphinx
) can be used. However, the docs commit (http://gerrit.named-data.net/#/c/658/) is not yet merged, so I would suggest waiting a little bit (I will try to resolve the issues and merge it today).
Updated by Chengyu Fan over 10 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 80
Updated by Chengyu Fan over 10 years ago
- Status changed from In Progress to Code review
- % Done changed from 80 to 100
Updated by Chengyu Fan over 10 years ago
- Status changed from Code review to Closed