Project

General

Profile

Actions

Task #3220

closed

Jenkins: build timeout

Added by Junxiao Shi over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Category:
Build
Target version:
Start date:
Due date:
% Done:

100%

Estimated time:
1.50 h

Description

Set a hard-limit on the duration of each build, so that if a build enters a deadlock or endless loop, it can be aborted automatically when hitting the timeout, not requiring manual intervention of an admin.

This issue includes:

  • determine the appropriate timeout of each project
  • install Build-timeout Plugin
  • configure the timeouts
Actions #1

Updated by Junxiao Shi over 8 years ago

In the past week, there has been 3 incidents of a build not terminating after more than 8 hours.
Every time I have to email Jenkins admins in order to abort the build.

I suggest the timeout to be at least 1.5x normal build time.

Specifically,

  • ndn-cxx: 180 minutes
  • NFD: 180 minutes
  • ndn-tools: 90 minutes

One complication is: when a build starts but slaves aren't available, the master's clock is already running.

We should set the timeout at each slave rather than at the master, so that a build is aborted only if the timeout is exceeded since all slaves become available for the build.

Actions #2

Updated by Alex Afanasyev over 8 years ago

I have configured the timeout to fail when there is no activity. I have no idea how it related to waiting for slaves.

We have only tools available for us. If plugin doesn't have support for what you described, we out of luck.

Actions #3

Updated by Junxiao Shi over 8 years ago

  • Status changed from New to Feedback
  • Assignee set to Alex Afanasyev

There isn't a problem in waiting for slaves to become available.

However, in http://jenkins.named-data.net/job/NFD/3182/OS=Ubuntu-14.04-32bit/console the log says:

Build timed out (after 3 minutes). Marking the build as failed.

This timeout is too short, as compared to the suggested values in note-1.

Even if it's for inactivity, it's still too short, as compared to Travis's 10-minute inactivity timout.

Actions #4

Updated by Davide Pesavento over 8 years ago

Junxiao Shi wrote:

This timeout is too short, as compared to the suggested values in note-1.

Even if it's for inactivity, it's still too short, as compared to Travis's 10-minute inactivity timout.

Agreed, 3 minutes is definitely too short. For example when git triggers a gc, it can take several minutes. I'd set the inactivity timeout to at least 10-15 minutes.

Actions #5

Updated by Alex Afanasyev over 8 years ago

I think I mentioned in on gerrit, but will repeat here. I didn't intend to set 3 minute timeout, I wanted to set up 3 hour timeout and was mistaken by the configuration option units (I though it is 180 minutes, which in reality was 180 seconds).

Actions #6

Updated by Junxiao Shi over 8 years ago

  • Status changed from Feedback to Closed
  • Target version set to v0.4
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF