Task #3220
closedJenkins: build timeout
100%
Description
Set a hard-limit on the duration of each build, so that if a build enters a deadlock or endless loop, it can be aborted automatically when hitting the timeout, not requiring manual intervention of an admin.
This issue includes:
- determine the appropriate timeout of each project
- install Build-timeout Plugin
- configure the timeouts
Updated by Junxiao Shi over 9 years ago
In the past week, there has been 3 incidents of a build not terminating after more than 8 hours.
Every time I have to email Jenkins admins in order to abort the build.
I suggest the timeout to be at least 1.5x normal build time.
Specifically,
- ndn-cxx: 180 minutes
- NFD: 180 minutes
- ndn-tools: 90 minutes
One complication is: when a build starts but slaves aren't available, the master's clock is already running.
We should set the timeout at each slave rather than at the master, so that a build is aborted only if the timeout is exceeded since all slaves become available for the build.
Updated by Alex Afanasyev over 9 years ago
I have configured the timeout to fail when there is no activity. I have no idea how it related to waiting for slaves.
We have only tools available for us. If plugin doesn't have support for what you described, we out of luck.
Updated by Junxiao Shi about 9 years ago
- Status changed from New to Feedback
- Assignee set to Alex Afanasyev
There isn't a problem in waiting for slaves to become available.
However, in http://jenkins.named-data.net/job/NFD/3182/OS=Ubuntu-14.04-32bit/console the log says:
Build timed out (after 3 minutes). Marking the build as failed.
This timeout is too short, as compared to the suggested values in note-1.
Even if it's for inactivity, it's still too short, as compared to Travis's 10-minute inactivity timout.
Updated by Davide Pesavento about 9 years ago
Junxiao Shi wrote:
This timeout is too short, as compared to the suggested values in note-1.
Even if it's for inactivity, it's still too short, as compared to Travis's 10-minute inactivity timout.
Agreed, 3 minutes is definitely too short. For example when git triggers a gc, it can take several minutes. I'd set the inactivity timeout to at least 10-15 minutes.
Updated by Alex Afanasyev about 9 years ago
I think I mentioned in on gerrit, but will repeat here. I didn't intend to set 3 minute timeout, I wanted to set up 3 hour timeout and was mistaken by the configuration option units (I though it is 180 minutes, which in reality was 180 seconds).
Updated by Junxiao Shi about 9 years ago
- Status changed from Feedback to Closed
- Target version set to v0.4
- % Done changed from 0 to 100