previous episode

UC Berkeley - Chem 280 - Foundations of Programming and Software Engineering

next episode

Python Testing using pytest

Overview

Teaching: 60 min
Exercises: 10 min
Questions

  • How is a Python module tested?

Objectives

  • Explain the overall structure of testing.

  • Explain the reasons why testing is important.

  • Understand how to write tests using the pytest framework.

Lesson Contents

Until now, we have been writing functions and checking their behavior using assert statements. While this seems to work, it can be tedious and prone to error. In this lesson, we’ll discuss how to write tests and run them using the pytest testing framework.

Using a testing framework will allow us to easily define tests for all of our functions and modules, and to test these each time we make a change. This will ensure that our code is behaving the way we expect, and that we do not break any features existing in the code by making new changes.

This episode explains the importance of code testing and demonstrates the possible capabilities.

Why testing

Software should be tested regularly throughout the development cycle to ensure correct operation. Thorough testing is typically an afterthought, but for larger projects, it is essential for ensuring changes in some parts of the code do not negatively affect other parts.

Software testing is checking the behavior of part of the code (such as a method, class, or a module) by comparing its expected output or behavior with the observed one. We will explain this in more detail shortly.

Unit vs Regression vs Integration testing

There are three main levels of testing:

In this lesson, we are focusing on unit testing. The same concepts here can be applied to perform Integration tests across modules.

The pytest testing framework

We recommend using the pytest testing framework. Other testing frameworks are available (such as unittest and nose tests); however, the combination of easy implementation, parametrization of tests, fixtures, and test marking make pytest an ideal testing framework.

If you don’t have pytest installed or it’s not updated to version 3, install it using:

$ pip install -U pytest-cov

Running our first test

When we run pytest, it will look for directories and files which start with test or test_. It then looks inside of those files and executes any functions that begin with the word test_. This syntax lets pytest know that these functions are tests. If these functions do not result in an error, pytest counts the function as passing. If an error occurs, the test fails.

Create a directory called tests in your mcsim package. In your tests directory, create an empty file called test_coord.py. It is a good idea to separate your tests into different modules based on which python file you will be testing.

Let’s write a test for our calculate_distance function.

Add the following contents to the test_coord.py file.

"""
Tests for the coord module
"""
# import your package.
# we can do this because we used setup.py and pip install.
import mcsim

from mcsim.monte_carlo import calculate_distance

In this first part of the file, we are getting ready to write our test. We first import the libraries and modules we need. We will only need the mcsim package we have installed in the previous lesson.

Next, we write our test. When you write a test to be run with pytest, it must be inside a function that begins with the word test. We set up our calculation, then we use the function we are testing and compare the output with our expected output. This is very similar to what we did before with our assert statements, except that it is now inside of a function:

def test_calculate_distance():
    point_1 = [0, 0, 0]
    point_2 = [1, 0, 0]

    expected_distance = 1
    dist1 = calculate_distance(point_1, point_2)
    
    assert dist1 == expected_distance

We can run this test using pytest in our terminal. In the folder containing both mcsim and tests, type

$ pytest

You should see an output similar to the following.

==================== test session starts ====================
platform darwin -- Python 3.7.6, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/jessica/lesson-follow/monte-carlo-lj
plugins: cov-2.10.0
collected 1 item                                            

tests/test_coord.py .

Here, pytest has looked through our directory and its subdirectories for anything matching test*. It found the tests folder, and within that folder, it found the file test_coord.py. It then executed the function test_calculate_distance within that module. Since our assertion was True, our test did not result in an error and the test passed.

We can see the names of the tests pytest ran by adding a -v tag to the pytest command.

$ pytest -v

Using the command argument -v will result in pytest listing which tests are executed and whether they pass or not. There are a number of additional command line arguments to explore.

==================== test session starts ====================
platform darwin -- Python 3.7.6, pytest-5.4.2, py-1.8.1, pluggy-0.13.1 -- /Users/jessica/miniconda3/envs/molssi_best_practices/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/jessica/lesson-follow/monte-carlo-lj
plugins: cov-2.10.0
collected 1 item                                            

tests/test_coord.py::test_calculate_distance PASSED   [100%]

Now we see that pytest displays the test name for us, as well as PASSED next to the test name.

Failing tests

Let’s see what happens when a test fails.

In case of test failure, Pytest will show detailed output from doing its own analysis to discover the error by inspecting your objects at runtime. Change the value of the expected variable in your test function to 2 and rerun the test.

$ pytest -v

==================== test session starts ====================
platform darwin -- Python 3.7.6, pytest-5.4.2, py-1.8.1, pluggy-0.13.1 -- /Users/jessica/miniconda3/envs/molssi_best_practices/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/jessica/lesson-follow/monte-carlo-lj
plugins: cov-2.10.0
collected 1 item                                            

tests/test_coord.py::test_calculate_distance FAILED   [100%]

========================= FAILURES ==========================
__________________ test_calculate_distance __________________

    def test_calculate_distance():
        point_1 = [0, 0, 0]
        point_2 = [1, 0, 0]
    
        expected_distance = 2
        dist1 = calculate_distance(point_1, point_2)
    
>       assert dist1 == expected_distance
E       assert 1.0 == 2
E         +1.0
E         -2

tests/test_coord.py:23: AssertionError

Pytest shows a detailed failure report, including the source code around the failing line. The line that failed is marked with >. Next, it shows the values used in the assert comparison at runtime, that is ` 1.0 == 2`. This runtime analysis is one of the advantages of pytest that help you debug your code.

Check Your Understanding

What happens if you leave your expected_value equal to 2, but remove the assertion? Change your assertion line to the following

expected_distance == calculated_distance

-

Answer

If you remove the word assert, you should notice that your test still passes. This is because the expression evaluated to False, but since there was no Assertion, there was no error. Since there was no error, pytest counted it as a passing test. The assert statement causes an error when it evaluates to False.

It’s very important to remember that pytest counts a test as failing when some type of exception occurs.

Change the expected value back to 1 so that your tests pass and make sure you have the assert statement.

Exercise

Create a second test for the calculate_distance function. Use the following points and box length:

r1 = [0, 0, 0]
r2 = [0, 0, 8]
box_length = 10

With periodic boundaries, these points correspond to a distance of 2.

Verify that your test is working by running pytest. You should now see two passing tests.

Solution

def test_calculate_distance2():
    point_1 = [0, 0, 0]
    point_2 = [0, 0, 8]
    box_length = 10

    expected_distance = 2
    dist1 = calculate_distance(point_1, point_2, box_length=box_length)
    assert dist1 == expected_distance

Advanced features of pytest

Python Decorators

Some of pytest’s advanced features make use of decorators. Decorators are a very powerful tool in programming, which we will not explore in depth here. You can think of them as functions that act on other functions. To decorate a particular function, you write the name of the decorator, preceded by @, in the line above the def statement:

@decorator
def foo():
    pass

Pytest Marks

Pytest marks allow you to mark your functions. There are built in marks for pytest and you can also define your own marks. Marks are implemented using decorators. One of the built-in marks in pytest is @pytest.mark.skip. Modify your test_calculate_distance function to use this mark.

@pytest.mark.skip
def test_calculate_distance():
    point_1 = [0, 0, 0]
    point_2 = [1, 0, 0]

    expected_distance = 1
    dist1 = calculate_distance(point_1, point_2)
    
    assert dist1 == expected_distance

When you run your tests, you will see that this test is now skipped:

tests/test_coord.py::test_calculate_distance SKIPPED  [ 50%]
tests/test_coord.py::test_calculate_distance2 PASSED  [100%]

You might also use the pytest.mark.xfail if you expect a test to fail.

Edge and Corner Cases

Edge cases

The situation where the test examines either the beginning or the end of a range, but not the middle, is called an edge case. In a simple, one-dimensional problem, the two edge cases should always be tested along with at least one internal point. This ensures that you have good coverage over the range of values.

Anecdotally, it is important to test edges cases because this is where errors tend to arise. Qualitatively different behavior happens at boundaries. As such, they tend to have special code dedicated to them in the implementation.

Corner cases

When two or more edge cases are combined, it is called a corner case. If a function is parametrized by two linear and independent variables, a test that is at the extreme of both variables is in a corner.

Code Coverage

Now that we have a set of modules and associated tests, we want to see how much of our package is “covered” by our tests. We’ll measure this by counting the lines of our packages that are touched, i.e. used, during our tests.

We already have everything we need for this since we installed pytest-cov earlier which includes the coverage tools on top of the pytest package.

We can assess our code coverage as follows:

pytest --cov=mcsim

==================== test session starts ====================
platform darwin -- Python 3.7.6, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /Users/jessica/lesson-follow/monte-carlo-lj
plugins: cov-2.10.0
collected 6 items                                           

tests/test_coord.py ......                            [100%]

---------- coverage: platform darwin, python 3.6.8-final-0 -----------
Name                   Stmts   Miss  Cover
------------------------------------------
mcsim/coord.py            33      8    76%
mcsim/energy.py           34     34     0%
mcsim/monte_carlo.py      51     51     0%
------------------------------------------
TOTAL                    118     93    21%

===================== 6 passed in 0.07s =====================

The output shows how many statements (i.e. not comments) are in a file, how many weren’t executed during testing, and the percentage of statements that were.

To improve our coverage, we also want to see exactly which lines we missed and we can determine this using the .coverage file produced by pytest. You can see an html version by running the command

pytest --cov=mcsim --cov-report=html

You will now have an additional folder called htmlcov in the directory. You can open the index.html file in this folder in your browser to view your code coverage report.

Do we need to get 100% coverage?

Short answer: no. Code coverage is a useful tool to assess how comprehensive our set of tests are and in general the higher our code coverage the better. However, trying to achieve 100% coverage on packages any larger than this sample package is a bit unrealistic and would require more time than that last bit of coverage is worth.

Key Points

  • A good set of tests covers individual functions/features and behavior of the software as a whole.

  • It’s better to write tests during development so you can check your progress along the way. The longer you wait to start the harder it is.

  • Try to test as much of your package as you can, but don’t go overboard, most packages don’t have 100% test coverage.

previous episode

next episode