Continuous Integration¶
Production¶
Our production Jenkins instance is available at https://qa-master.fxtest.jenkins.stage.mozaws.net/ and access is restricted according to this documentation.
Sandbox, aka “Dev Jenkins”¶
Our sandbox Jenkins instance is available at https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/ and requires a connection to the Mozilla VPN. See the Mozilla VPN documentation for further information regarding this instance.
Plugin Updates¶
- Whomever is able to respond and take action first, files a bug in Cloud Services | FXTest-Infra, cc:ing the rest of the core Jenkins/infra team, assigning the bug to themselves, and checking the “Security” checkbox at the bottom of the bug. Include the Jenkins advisory text, with a link (like https://jenkins.io/security/advisory/2017-04-26/), the name of and link to the affected plugin(s), as well as the version to which you’ve upgraded Jenkins dev. Please use this Bugzilla template, to file.
- After filing, it’s time to upgrade the plugin(s):
- Update Jenkins dev: * Log in to the Jenkins dev instance * Click on “Manage Jenkins” on the left * Click on “Prepare for Shutdown” * Click on “Manage Plugins” * Click the “Check Now” button * Click the checkbox(es) next to the affected plugin(s), and click the “Download now and install after restart” button * Also select the checkbox to “Restart Jenkins when installation is complete and no jobs are running” * Under “Build Queue”, click the “cancel” link to allow Jenkins to safely restart * Run the sanity.pipeline job, vet the results, looking for new, related failures * Once the upgrades have completed on dev, resolve the Bugzilla bug as fixed
- Kick off the “run all builds” test job
- Carefully vet the results
- If all goes well, follow the instructions for updating plugins on production Jenkins
Plugin Addition¶
- Coordinate with and give peers a heads-up that you’re installing a new plugin on dev (and why)
- Install the plugin
- Restart Jenkins
- Run the
sanity.pipelinejob, and try to ensure there are no new, related failures - Once you’re comfortable with the results, do the following:
- File a bug using this Bugzilla template, requesting the plugin(s) installation. Include the following info: * the plugin name(s), version(s), link(s) on https://plugins.jenkins.io/ * mention that it’s been successfully tested on the dev instance.
- Once Ops installs the plugin on Prod, make sure to: * test affected job(s), and * ping back in the Production-update bug with the appropriate resolution/verification data
Ops-QA Pipeline¶
The current flow for a project integrated into the Cloud Ops deploy pipeline is as follows:
- A tagged or pushed build from dev deploys to staging
- Cloud Ops’ deploy-pipeline script calls
qaTest("kinto", "stage"), which remotely runs the project’s corresponding staging (“stage”) test job, e.g.kinto.stage, in our Jenkins instance - If our tests pass (returning exit code/return status of “0”), and after manual confirmation from Ops, the build gets promoted and pushed to production
Getting a project’s tests into the deploy pipeline:
- A suggestion is to have your project build and run tests in Jenkins, from a Docker image
- Create a Jenkins job with the following syntax: project.test_env (e.g. kinto.stage), using the
Pipeline from SCMoption, and pointing to the Jenkinsfile - Once your project runs and passes in Jenkins:
- File a bug (example: bug 1384404), in the most-appropriate component for your project, under the Cloud Services product, requesting Ops enable your jobs in their pipeline
- Next, from Ops’ side, there is a qaTest.groovy file which calls run_jenkins_job, which, in turn, authenticates with QA (prod) Jenkins, and will run /job/${project}.${envName}