Python test coverage
SonarCloud supports the reporting of test coverage information as part of the analysis of your Python project.
However, SonarCloud does not produce the coverage report itself. Instead, you must set up a third-party tool to produce the report as part of your build process. You then need to configure your analysis to tell the SonarScanner where the report is located so that it can pick it up and send it to SonarCloud, where it will be displayed on your project dashboard along with the other analysis metrics.
Usually, when you import a new Python project, automatic analysis starts immediately. But, since coverage is not yet supported under automatic analysis, you will need to use CI-based analysis instead. This requires disabling automatic analysis. Here are the steps you need to follow:
If you have not yet imported your python project, just add an empty file called
sonar-project.properties to the root of your repository, and then perform the import. SonarCloud will assume that you want to set up a CI-based analysis and display the onboarding tutorial.
If you have already imported your project, then SonarCloud has already run at least once using automatic analysis. Don’t worry, you can still convert your project to use a CI-based approach. Simply go to Administration > Analysis Method and switch SonarCloud Automatic Analysis to OFF. Then, on the same screen, under Supported analysis methods find your preferred CI and select Follow the tutorial.
At this point, you should be in the onboarding tutorial specific to your CI. Follow the tutorial and when it asks, What option best describes your build?, choose Other (for JS, TS, Go, Python, PHP, ...). When you are done with the tutorial, you should have a functioning CI-based analysis setup for your Python project. The next step is to adjust it to get coverage working.
To enable coverage you need to:
- Adjust your build process so that the coverage tool runs before the scanner report generation step runs.
- Make sure that the coverage tool writes its report file to a defined path in the build environment.
- Configure the scanning step of your build so that the scanner picks up the report file from that defined path.
The details of setting up coverage within your build process depend on which tools you are using. In our example we use:
- Tox, to configure the tests
- Pytest, to execute the tests
- Coverage, (the Coverage.py tool,) to measure code coverage, and
- GitHub Actions, to perform the build.
In this example, we invoke
pytest and use the
pytest-cov plugin which, in turn, uses Coverage.py. Simply add the text below to the
tox.ini file at the root of your project:
Tox.ini (Coverage.py and Pytest)
Alternatively, in this example, we start the test by invoking the Coverage.py tool (the command
coverage) with the
pytest invocation as an argument.
Alternative tox.ini (Coverage.py only)
Note that we specify
relative_files = True in the
tox.ini file to ensure that GitHub Actions will correctly parse your coverage results.
The following shows how to configure the GitHub Actions build file for your Python project so that it works in conjunction with the
tox.ini configuration file described above to generate code coverage. Your
build.yml file should look something like this:
First of all, install all of your project dependencies and then invoke
tox to run your tests and generate a coverage report file.
If, as here, you do not specify an output file, the scanner will look for report paths located under the default
If you are using a different package manager and/or a different testing tool these details will be different.
The essential requirements are that the tool produces its report in the Cobertura XML format and writes it to a place from which the scanner can then pick it up.
The next step is to add
sonar.python.coverage.reportPaths to your analysis parameters. This parameter must be set to the path of the report file produced by your coverage tool. In this example, that path is set to the default produced by Coverage.py. It is set in the
sonar-project.properties file, located in the project root:
Wildcards and a comma-delimited list of paths are supported. See Test Coverage Parameters for details.
This property is usually set in the
sonar-project.properties file, located in the project root. Alternatively, you can also set it in the command line of the scanner invocation or in the SonarCloud interface under
Your Organization > Your Project > Administration > General Settings > Languages > Python > Tests and Coverage > Path to coverage report(s)