Continuous Integration

Production

Our production Jenkins instance is available at https://fx-test-jenkins.stage.mozaws.net and access is restricted according to this documentation.

Sandbox, aka “Dev Jenkins”

Our sandbox Jenkins instance is available at https://fx-test-jenkins-dev.stage.mozaws.net:8443 and requires a connection to the Mozilla VPN. See the documentation for further information regarding this instance.

Plugin Updates

  1. Whomever is able to respond and take action first, files a bug in Cloud Services | FXTest-Infra, cc:ing the rest of the core Jenkins/infra team, assigning the bug to themselves, and checking the “Security” checkbox at the bottom of the bug. Include the Jenkins advisory text, with a link (like https://jenkins.io/security/advisory/2017-04-26/), the name of and link to the affected plugin(s), as well as the version to which you’ve upgraded Jenkins dev. Please use this Bugzilla template, to file.
  2. After filing, it’s time to upgrade the plugin(s):
  3. Update Jenkins dev:
  • Log in to the Jenkins dev instance
  • Click on “Manage Jenkins” on the left
  • Click on “Prepare for Shutdown”
  • Click on “Manage Plugins”
  • Click the “Check Now” button
  • Click the checkbox(es) next to the affected plugin(s), and click the “Download now and install after restart” button
  • Also select the checkbox to “Restart Jenkins when installation is complete and no jobs are running”
  • Under “Build Queue”, click the “cancel” link to allow Jenkins to safely restart
  • Run the sanity.pipeline job, vet the results, looking for new, related failures
  • Once the upgrades have completed on dev, resolve the Bugzilla bug as fixed
  1. Kick off the “run all builds” test job
  2. Carefully vet the results
  3. If all goes well, follow the instructions for updating plugins on production Jenkins

Plugin Addition

  1. Coordinate with and give peers a heads-up that you’re installing a new plugin on dev (and why)
  2. Install the plugin
  3. Restart Jenkins
  4. Run the sanity.pipeline job, and try to ensure there are no new, related failures
  5. Once you’re comfortable with the results, do the following:
  6. File a bug using this Bugzilla template, requesting the plugin(s) installation. Include the following info:
  • the plugin name(s), version(s), link(s) on https://plugins.jenkins.io/
  • mention that it’s been successfully tested on the dev instance.
  1. Once Ops installs the plugin on Prod, make sure to:
  • test affected job(s), and
  • ping back in the Production-update bug with the appropriate resolution/verification data

Ops-QA Pipeline

  1. Get the project and its test repositories into Service Book
  2. Your test repo should have the following:
  1. Create a Jenkins job with the following syntax: project.test_type_test_env (e.g. kinto.integration.stage), using the Pipeline from SCM option, and pointing the Jenkinsfile
  2. Once your project is set up properly (runs, and hopefully passes):
  3. File a bug (example: bug 1384404, in the Cloud Services product, FXTest-infra component, requesting Ops enable your jobs in their pipeline