What would the process of updating the code and continuous integration environment look like in an ideal world?
A developer [1] pushes code changes to a repository branch and they are immediately tested on a build server. When (s)he is satisfied with the results, he [2] creates a pull request. Before a peer developer looks at the changes, they are already reviewed by an automated code analyser. And finally the reviewer [3] approves the pull request to be merged to the main branch.
So there is good news! There are already services available online that enable this process without you having to run complex CI environment. If you continue reading this article you’ll learn about the systems we used to implement it for the minimesos project at Container Solutions and all the integration details.
In our project we use GitHub as source code repository, Travis CI for running the builds, Pull Approve for structuring code reviews and SonarQube for monitoring code quality.
When code changes are pushed to the source repository, GitHub uses webhooks to notify Pull Approve and Travis CI. Unless this push belongs to already existing pull request, Pull Approve ignores this event. However Travis CI pulls the code, runs a build and notifies the author of the changes about the results by email.
When a pull request is created in GitHub (PR), Pull Approve extends the PR with a required check: one or more authorised reviewers have to approve the changes by commenting the PR with certain keywords. Travis CI pulls the code again and starts building the PR. It also adds both the PR and latest related push builds to the PR as additional checks. Our PR build on Travis completes with a SonarQube preview scan. This scan analyses updated code and compares results with the data, which is pulled from the SonarQube server. It places new findings as inline comments in the GitHub PR and adds the SonarQube check.
When a reviewer is satisfied with the code changes he/she comments on the PR. The notified about the new comment Pull Approve checks if its content contains pre-defined keywords and, if yes, enables Merge button by marking its check as passed. Merging of the PR to the main branch creates another commit. Travis CI detects the build on the main branch and completes it by updating the SonarQube server with new findings, so they can be used for reporting on future pull requests.
This is it. Clear and easy. And now we’ll dive into implementation details.
Let’s start with SonarQube - the only component which we host ourselves. SonarQube is a web application which stores data in a database. For simplicity and reproducibility we created Docker Compose docker-compose.yml file.
sonarqube:
image: "sonarqube:5.3"
command: -Dsonar.web.context=/sonar
ports:
- "9000:9000"
- "5432:5432"
environment:
- SONARQUBE_JDBC_URL=jdbc:postgresql://localhost:5432/sonar
- SONARQUBE_JDBC_PASSWORD=[SONAR_DB_USER_PASSWORD]
restart: always
volumes:
- sonarqube-extensions-plugins:/opt/sonarqube/extensions/plugins
- /opt/sonarqube
db:
image: postgres
net: container:sonarqube
environment:
- POSTGRES_USER=sonar
- POSTGRES_PASSWORD=[SONAR_DB_USER_PASSWORD]
restart: always
volumes:
- postgresql-data:/var/lib/postgresql/data
- postgresql-run:/run/postgresql
Update the file with a difficult to guess SONAR_DB_USER_PASSWORD and start SonarQube by sudo docker-compose up -d. When SonarQube starts, you should access it on http://localhost:9000 and change default admin password from admin to something else. Database data is stored in mapped host directories and therefore it’s persisted when containers are restarted. SonarQube plugins can be installed by downloading JAR files to sonarqube-extensions-plugins directory and restarting the server. Install GitHub Plugin to enable integration with GitHub.
Important thing to add: for future integration with Travis CI both application and database should be accessible from Travis CI build boxes.
In order to integrate your project with Pull Approve, login with your GitHub account on https://pullapprove.com/ and you’ll be offered to enable integration for repositories you are authorised to control. Then you need to add .pullapprove.yml file with Pull Approve settings to the root of your repository. Here is the content of this file in our repository.
All available configuration options are described on Pull Approve site.
And now let’s review the integration with Travis CI. It starts the same way as integration with Pull Approve - login to the site, enable the integration and add .travis.yml file. The contents of the file heavily depends on the build tool, languages and technologies which are used in your project. Our version of the file became quite complicated because we need to use certain version of docker and making it available on build boxes deserves a separate blog post. Travis CI site contains a detailed description of available options and samples for nearly all commonly used technologies. Based on the images above you have already concluded that our build runs somewhat differently depending on what is being built. This logic is implemented in travis.sh, which is being called from .travis.yml.
For every build run Travis CI provides a number of default environment variables, which enable determining what is being built. When using TRAVIS_PULL_REQUEST we detect PR build we extend the build with SonarQube preview scan and add system properties to configure connections with SonarQube server and GitHub. Installed GitHub Plugin for SonarQube enables commenting on pull request in GitHub.
Values of the system properties are taken from our repository specific environment variables
The GitHub authentication token and SonarQube database user password should not be visible in Travis CI logs. Therefore they are marked as secure environment variables. Actually, because environment variables in Travis CI are not editable, only the person, who is adding these variables will know the values.
Since pull requests in GitHub can be initiated from forks of your repository, Travis CI does not expose the secure variables to untrusted code. We use TRAVIS_SECURE_ENV_VARS default variable to detect builds on externally modified code and to avoid use of sensitive data. The environment variable TRAVIS_BRANCH is used to detect build on default branch and to refresh data in SonarQube database.
Although both Pull Approve and Travis CI are very new services and it took a while to understand the integration details, the created setup seems to be very stable and flexible. Most importantly, it enables us to keep an eye on quality of our code. We use Gradle as the build tool for our project. However integration of none of the components depends on it and you should be able to use Maven, Ant or shell scripts as long as you have a way to invoke SonarQube runner.
Do you think we can improve or simplify our setup even further? Do you want to share your experience of setting up CI environment using online services? Please, do not hesitate to let us know by commenting below or reaching out to @sadovnikov or @containersoluti on Twitter.