Continuous Integration


Our production Jenkins instance is available at and access is restricted according to this documentation.

Sandbox, aka “Dev Jenkins”

Our sandbox Jenkins instance is available at and requires a connection to the Mozilla VPN. See the documentation for further information regarding this instance.

Plugin Updates

  1. Whomever is able to respond and take action first, files a bug in Cloud Services | FXTest-Infra, cc:ing the rest of the core Jenkins/infra team, assigning the bug to themselves, and checking the “Security” checkbox at the bottom of the bug. Include the Jenkins advisory text, with a link (like, the name of and link to the affected plugin(s), as well as the version to which you’ve upgraded Jenkins dev. Please use this Bugzilla template, to file.
  2. After filing, it’s time to upgrade the plugin(s):
  3. Update Jenkins dev:
  • Log in to the Jenkins dev instance
  • Click on “Manage Jenkins” on the left
  • Click on “Prepare for Shutdown”
  • Click on “Manage Plugins”
  • Click the “Check Now” button
  • Click the checkbox(es) next to the affected plugin(s), and click the “Download now and install after restart” button
  • Also select the checkbox to “Restart Jenkins when installation is complete and no jobs are running”
  • Under “Build Queue”, click the “cancel” link to allow Jenkins to safely restart
  • Run the sanity.pipeline job, vet the results, looking for new, related failures
  • Once the upgrades have completed on dev, resolve the Bugzilla bug as fixed
  1. Kick off the “run all builds” test job
  2. Carefully vet the results
  3. If all goes well, follow the instructions for updating plugins on production Jenkins

Plugin Addition

  1. Coordinate with and give peers a heads-up that you’re installing a new plugin on dev (and why)
  2. Install the plugin
  3. Restart Jenkins
  4. Run the sanity.pipeline job, and try to ensure there are no new, related failures
  5. Once you’re comfortable with the results, do the following:
  6. File a bug using this Bugzilla template, requesting the plugin(s) installation. Include the following info:
  • the plugin name(s), version(s), link(s) on
  • mention that it’s been successfully tested on the dev instance.
  1. Once Ops installs the plugin on Prod, make sure to:
  • test affected job(s), and
  • ping back in the Production-update bug with the appropriate resolution/verification data

Ops-QA Pipeline

The current flow for a project integrated into the Cloud Ops deploy pipeline is as follows:

  1. A tagged or pushed build from dev deploys to staging
  2. Cloud Ops’ deploy-pipeline script calls qaTest("kinto", "stage"), which remotely runs the project’s corresponding staging (“stage”) test job, e.g. kinto.stage, in our Jenkins instance
  3. If our tests pass (returning exit code/return status of “0”), and after manual confirmation from Ops, the build gets promoted and pushed to production

Getting a project’s tests into the deploy pipeline:

  1. A suggestion is to have your project build and run tests in Jenkins, from a Docker image
  2. Create a Jenkins job with the following syntax: project.test_env (e.g. kinto.stage), using the Pipeline from SCM option, and pointing to the Jenkinsfile
  3. Once your project runs and passes in Jenkins:
  4. File a bug (example: bug 1384404), in the most-appropriate component for your project, under the Cloud Services product, requesting Ops enable your jobs in their pipeline
  5. Next, from Ops’ side, there is a qaTest.groovy file which calls run_jenkins_job, which, in turn, authenticates with QA (prod) Jenkins, and will run /job/${project}.${envName}

Build notifications

Notifications can differ between projects, however typically whenever a build fails a notification is sent to the fte-ci group. When the result of a build changes, a notification is sent to the #fx-test-alerts IRC channel on