NFD

Python traceback from the failure (http://jenkins.named-data.net/job/Docs-Checkers/job/NFD-Doxygen-Checker/2514/console):

Traceback (most recent call last):
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Scripting.py", line 118, in waf_entry_point
    run_commands()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Scripting.py", line 178, in run_commands
    ctx=run_command(cmd_name)
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Scripting.py", line 169, in run_command
    ctx.execute()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Scripting.py", line 366, in execute
    return execute_method(self)
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Build.py", line 93, in execute
    self.execute_build()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Build.py", line 100, in execute_build
    self.compile()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Build.py", line 167, in compile
    self.producer.start()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Runner.py", line 214, in start
    self.refill_task_list()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Runner.py", line 135, in refill_task_list
    tasks=self.biter.next()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Build.py", line 402, in get_build_iterator
    self.post_group()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Build.py", line 385, in post_group
    tgpost(tg)
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/Build.py", line 359, in tgpost
    f()
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/.waf-2.0.6-593191f496fe8c66231dfd5df26167ae/waflib/TaskGen.py", line 123, in post
    v()
  File ".waf-tools/sphinx_build.py", line 60, in apply_sphinx
    confData = imp.load_source('sphinx_conf', conf.abspath())
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/docs/conf.py", line 49, in <module>
    addExtensionIfExists('sphinxcontrib.doxylink')
  File "/home/jenkins/workspace/Docs-Checkers/NFD-Doxygen-Checker/docs/conf.py", line 41, in addExtensionIfExists
    __import__(extension)
  File "/usr/local/lib/python2.7/dist-packages/sphinxcontrib/doxylink/__init__.py", line 1, in <module>
    from .doxylink import setup
  File "/usr/local/lib/python2.7/dist-packages/sphinxcontrib/doxylink/doxylink.py", line 170
    def find_url2(mapping, symbol: str) -> dict:
                                 ^
SyntaxError: invalid syntax

Well, this is annoying...

File "/usr/local/lib/python2.7/dist-packages/sphinxcontrib/doxylink/doxylink.py", line 170
def find_url2(mapping, symbol: str) -> dict:

The ": str" is a type annotation. That syntax was introduced in python 3.5 or thereabouts, which means all earlier versions of python are going to fail pretty hard when trying to parse it. I don't know if breaking py2.7 compat was intentional, but given there hasn't been a commit to doxylink repo in 3+ months, I suppose it was.

How about we switch the docs builders to using "python3 waf" instead of "./waf"?

According to the GitHub releases page, doxylink 1.4 broke compatibility with Python 3. Therefore, I believe we'll either need to change our requirements to Python 3 or require both Python 2 and Python 3.

*broke compatibility with Python 2

Are we currently telling people to install doxylink from pip?

Therefore, I believe we'll either need to change our requirements to Python 3 [...]

I'm all for moving to python3 for the build system, but I don't think Ubuntu 14.04 has a new enough python3 for doxylink.

[...] or require both Python 2 and Python 3.

I would be against this.

Davide Pesavento wrote:

Are we currently telling people to install doxylink from pip?

Yes. In the install instructions in each repository.

Therefore, I believe we'll either need to change our requirements to Python 3 [...]

I'm all for moving to python3 for the build system, but I don't think Ubuntu 14.04 has a new enough python3 for doxylink.

The version installed on the the Ubuntu 14.04 agents at Arizona is 3.4.3. Per https://github.com/sphinx-contrib/doxylink/blob/master/doc/index.rst, doxylink requires 3.4, so we should be good. Also, we don't build documentation on 14.04 and will drop support for it soon (when 18.04 is released).

[...] or require both Python 2 and Python 3.

I would be against this.

Agreed.

As a short-term fix, just disable doxylink module for documentation checking. It doesn't produce any warnings that we should be concerned about.

I am ok to require Python3 for documentation builds. We don't need to impose this requirement on non-documentation builds. For that, we just need to ensure that external sphinx modules are not enabled, which is a bit tricky and I'm not sure how exactly to do.

Eric Newberry wrote:

Davide Pesavento wrote:

Are we currently telling people to install doxylink from pip?

Yes. In the install instructions in each repository.

And if I run python2's pip to install doxylink, is it stupid enough to install a version that's not compatible with python2?

I am ok to require Python3 for documentation builds. We don't need to impose this requirement on non-documentation builds

Yes, but we need a simple way to document this for users. And to me it sounds much simpler if we just tell them to always use e.g. "python3 waf <foo>", regardless of what they're building. The downside is that on mac python3 is not installed by default.

we just need to ensure that external sphinx modules are not enabled, which is a bit tricky and I'm not sure how exactly to do.

Or we just don't load sphinxcontrib.doxylink from docs/conf.py if running on python2.

we just need to ensure that external sphinx modules are not enabled, which is a bit tricky and I'm not sure how exactly to do.

Or we just don't load sphinxcontrib.doxylink from docs/conf.py if running on python2.

I was thinking something like this when building manpages (the only thing that we build for ./waf that may get affected). However, just disabling all extensions with python2 seem to be simpler and should address the issues.

Sure, I'm fine either way, if you know how to "disable all extensions"... (I don't)

I'm working on commits to disable all extensions when building with Python 2. These should be up for review soon.

Eric Newberry wrote:

Commits to disable all extensions with Python 3 have been pushed for ChronoSync, ndn-cxx, ndns, NFD, and NLSR.

We should also update the build/installation instructions to tell users to use python3 for docs, I guess?

Davide Pesavento wrote:

Eric Newberry wrote:

Commits to disable all extensions with Python 3 have been pushed for ChronoSync, ndn-cxx, ndns, NFD, and NLSR.

We should also update the build/installation instructions to tell users to use python3 for docs, I guess?

Yes. But should we switch over to requiring Python 3 entirely?

Yes. But should we switch over to requiring Python 3 entirely?

I agree with requiring Python 3. ‘waf’ binary needs to have python3 shebang line.

Junxiao Shi wrote:

Yes. But should we switch over to requiring Python 3 entirely?

I agree with requiring Python 3. ‘waf’ binary needs to have python3 shebang line.

I think we should do this. I'll go ahead and test out an entirely Python 3 build process to see if it works.

Eric Newberry wrote:

I think we should do this. I'll go ahead and test out an entirely Python 3 build process to see if it works.

Don't. In today's NFD call the consensus was to only apply a minimal fix to disable doxylink when running under python2 for now. Moving the build process entirely over to python3 will be done at a later time.

Eric Newberry wrote:

Since this breaks all docs builds on Jenkins, we should try to get this merged asap.

Slight correction: NLSR doesn't appear to use doxylink (only googleanalytics), so it should still be working.