Task #3220
closed
Added by Junxiao Shi about 9 years ago.
Updated about 9 years ago.
Description
Set a hard-limit on the duration of each build, so that if a build enters a deadlock or endless loop, it can be aborted automatically when hitting the timeout, not requiring manual intervention of an admin.
This issue includes:
- determine the appropriate timeout of each project
- install Build-timeout Plugin
- configure the timeouts
In the past week, there has been 3 incidents of a build not terminating after more than 8 hours.
Every time I have to email Jenkins admins in order to abort the build.
I suggest the timeout to be at least 1.5x normal build time.
Specifically,
- ndn-cxx: 180 minutes
- NFD: 180 minutes
- ndn-tools: 90 minutes
One complication is: when a build starts but slaves aren't available, the master's clock is already running.
We should set the timeout at each slave rather than at the master, so that a build is aborted only if the timeout is exceeded since all slaves become available for the build.
I have configured the timeout to fail when there is no activity. I have no idea how it related to waiting for slaves.
We have only tools available for us. If plugin doesn't have support for what you described, we out of luck.
- Status changed from New to Feedback
- Assignee set to Alex Afanasyev
There isn't a problem in waiting for slaves to become available.
However, in http://jenkins.named-data.net/job/NFD/3182/OS=Ubuntu-14.04-32bit/console the log says:
Build timed out (after 3 minutes). Marking the build as failed.
This timeout is too short, as compared to the suggested values in note-1.
Even if it's for inactivity, it's still too short, as compared to Travis's 10-minute inactivity timout.
Junxiao Shi wrote:
This timeout is too short, as compared to the suggested values in note-1.
Even if it's for inactivity, it's still too short, as compared to Travis's 10-minute inactivity timout.
Agreed, 3 minutes is definitely too short. For example when git triggers a gc, it can take several minutes. I'd set the inactivity timeout to at least 10-15 minutes.
I think I mentioned in on gerrit, but will repeat here. I didn't intend to set 3 minute timeout, I wanted to set up 3 hour timeout and was mistaken by the configuration option units (I though it is 180 minutes, which in reality was 180 seconds).
- Status changed from Feedback to Closed
- Target version set to v0.4
- % Done changed from 0 to 100
Also available in: Atom
PDF