Python testing primer

Ismael Mendonça 41 min read time Jan 6, 2021

This guide is intended as a very quick introduction to testing in pytest using unittest's mock library. After going through this guide you should be able to have a general understanding of how pytest works, how to mock and patch and the overall concepts for testing in python. There are useful links at the end of the guide for more in-depth information about pytest and testing in python. This guide is organized as follows:

Why pytest?

In python there are three main libraries for testing: pytest, unittest and nose (which is itself an extension of unittest). The unittest library is part of the Python Standard Library and provides a full framework for testing in Python, it's heavily inspired by JUnit from Java, which is why it has constructs like: self.assertTrue(foo) that are not very pythonic. On the other hand, nose was built as an extension of unittest, it's unittest with plugins. It's currently under maintenance and not in active development.

pytest is a more idiomatic alternative to unittest, it was created to address unittest's issues: lots of boilerplate code, non-pythonic approach, heavyweight setup/teardown and lack of flexibility.

The bigger advantage of pytest over unittest is its powerful fixture system, it follows the DRY(Don't repeat yourself) principle and contributes to the reduction of boilerplate code and more readable and isolated tests. They are a good alternative to the traditional setUp/tearDown, of unittest classes. The pytest fixture system also has an internal caching mechanism that it uses to speed up the testing process and avoid re-instantiating fixtures.

Given the following unittest example:

1# Example of Unittest
2import unittest
3
4class TestCompanyNames(unittest.TestCase):
5
6    def setUp(self):
7        self.companies = ["Facebook", "Apple", "Google", "Netflix"]
8
9    def test_facebook(self):
10        self.assertEqual("Facebook", self.companies[0])
11
12    def test_google(self):
13        self.assertEqual("Google", self.companies[1])
14
15    def test_join(self):
16        self.assertEqual(
17         ",".join(self.companies), 
18             "Facebook,Apple,Google,Netflix"
19    )
20

And its pytest equivalent:

1# Example of Unittest
2import pytest
3
4@pytest.fixture
5def companies():
6    return ["Facebook", "Apple", "Google", "Netflix"]
7
8def test_facebook(companies):
9    assert companies[0] == "Facebook"
10
11def test_google(companies):
12    assert companies[1] == "Google"
13
14def test_join(companies):
15    assert ",".join(companies) == "Facebook,Apple,Google,Netflix"
16

We can derive the following:

  • There's a lot less set up and boilerplate from the fact that pytest doesn't need to structure the tests in classes. However, you can still structure your tests in classes if you need to group your tests for semantic purposes, for state sharing or parallelization with pytest-xdist.
  • The companies data is not bounded only to the local scope of the TestCompanyNames class. It could be reused through your tests and it's instantiated just once in the scope it's defined.
  • The code is more pythonic and easier to understand.
  • The tests don't need to be scoped inside a class, which make them less prone to errors of sharing information between them since they don't need to share state.

In pytest it's also possible to parametrize tests and fixtures. The same test code can run again with different parameters in order to reuse the code and avoid logic repetition. The alternative in unittest is using subtests. However, subtests can increase the complexity of the test code making it hard to maintain, it also breaks the principle of tests being responsible for testing just one thing.

pytest comes with a plugin system that makes it very extensible*, there's a large amount of pytest plugins and a big community. The pytest* ecosystem is quite big and is actively maintained, for this reason it works well for different projects using different python libraries or frameworks.

Test discovery

Test discovery in pytest is done recursively starting at the paths defined testpaths configuration or any other directory specified in the call of the pytest command. The rules that pytest uses to identify tests are the following:

  • For a test file to be picked up by pytest it has to be prefixed with test_*.py or *_test.py. Example: test_companies.py or companies_test.py.

  • For a function/method to be picked up by pytest it has to start with test. Example:

1def test_companies():
2    ...
3
4def testothercompany():
5    ...
6
  • For a class to be picked up by pytest it has to start with Test. Example:
1class TestCompanies:
2    ...
3

See the docs for more details.

Running pytest

Running pytest is very straightforward:

1pytest <path to tests>
2

There are, however, some useful flags:

  • -x: stop the tests as soon as one fails.
  • -s: short cut to disable capturing (stout, sterr). This is useful to allow interactive debugging since pytest by default ignores stdin.
  • -v <verbose level> : increase the verbosity.
  • -m: run tests specified by certain mark.
  • -r f: extra test summary for failed tests.
  • --pdb: will invoke the pdb debugger on every fail.
  • --maxfail=<X>: fail at most X amount of tests.
  • --show-capture=no: disable reporting of captured content (stdout, stderr and logs) on failed tests completely.
  • --disable-warnings: suppress warning summary.
  • --ignore=<PATH>: ignore all the tests in <PATH>.
  • --sw: fail and continue from there next time.
  • --durations=<n>: Show the n slowest tests.

Running tests by node (test) ids:

1pytest test_companies.py::TestCompanies::test_company
2

This will run the method test_company of the class TestCompanies defined in the test_companies.py file.

Fixtures

Background

In pytest, fixtures are a way to define objects that will be reused throughout your tests. They are initialized only once in the scope in which they are defined. Fixtures are introduced to each test by means of dependency injection by passing them as parameters to the tests. Example:

1import datetime
2from unittest.mock import MagicMock
3
4# The fixture is used as a parameter in this test
5def test_dates(monkeypatch):
6    mock_date = MagicMock()
7    expected_date = datetime.datetime(2020,11,4)
8    mock_date.now.return_value = expected_date
9
10    monkeypatch.setattr(datetime, "datetime", mock_date)
11    assert datetime.datetime.now() == expected_date
12

pytest has a very powerful fixture system, it comes with a set of builtin fixtures that can be used directly in your tests. The fixture system is extendable, there are different libraries that add new fixtures to the pytest environment, for example: pytest-tornado include fixtures for app, ioloop, http_server and http_port, and pytest-django include fixtures for admin_user, database and settings.

Custom fixtures can also be defined using the @pytest.fixture decorator. By default, pytest loads a conftest.py file where local plugins and fixtures can be defined. The fixtures that are defined in the conftest.py file will be available for all the tests in the test suite:

1# -- conftest.py
2import pytest
3
4@pytest.fixture
5def fang_companies():
6    return ["Facebook", "Apple", "Google", "Netflix"]
7
8# -- test_companies.py
9
10# The fixture is used as a parameter in this test
11def test_companies(fang_companies):
12    assert "Facebook" in fang_companies
13

The scope of a fixture defined in a file different than conftest.py will be available only for such file, it won't be available in other test files.

pytest fixtures can have different scopes: function, class, module, package and session. The fixture objects are created when a test request them and are destroyed based on their defined scope, from the docs (and adding comprehensible test scope level definitions):

  • Function: the default scope, the fixture is destroyed at the end of the test. — The scope will be at function level.
  • Class: the fixture is destroyed during teardown of the last test in the class. — The scope will be at class level.
  • Module: the fixture is destroyed during teardown of the last test in the module. — The scope will be at file level.
  • Package: the fixture is destroyed during teardown of the last test in the package. — The scope will be at directory level.
  • Session: the fixture is destroyed at the end of the test session. The scope will be at the — test suite level.

The scope is simply defined as a keyword argument to the fixture:

1# -- conftest.py
2import pytest
3
4@pytest.fixture(scope="session") # This test will have a session scope
5def fang_companies():
6    return ["Facebook", "Apple", "Google", "Netflix"]
7

If your tests require some data, it's a good practice to create fixtures for them, pytest has an automatic caching mechanism for fixtures which will serve as an advantage to avoid loading data for each test.

pytest only caches one instance of a fixture at a time, which means that when using a parametrized fixture, pytest may invoke a fixture more than once in the given scope.

Fixture loading order

Fixtures are instantiated starting from the higher-scoped ones and ending in the more specific or narrow-scoped ones. Fixtures are loaded in the following order:

  1. Session fixtures.
  2. Package Fixtures.
  3. Module Fixtures.
  4. Class Fixtures.
  5. Function/method Fixtures.

The relative order of fixtures of same scope follows the declared order in the test function and honours dependencies between fixtures. Fixtures with autouse will be instantiated before explicitly used fixtures.

Autouse fixtures

It's handy to have fixtures to be instantiated automatically, for instance, tests that require a clean-up of the data store before or after being executed. These fixtures will be instantiated automatically according to the scope in which they are defined. For a fixture to be used automatically, just add the parameter autouse to the fixture definition:

1# -- conftest.py
2import pytest
3
4@pytest.fixture(scope="function", autouse=True)
5def clear_storage():
6    """ Clears the storage before each test """
7    store = Storage()
8    store.clear_all()
9

If the scope is not explicitly defined, the fixture will follow these rules:

  • If it is defined in a test module, all its test functions automatically uses it.
  • If it is defined in a conftest.py file then all tests in all test modules below its directory will invoke the fixture.
  • If it is defined in a plugin, it will be invoked for all tests in all projects where the plugin is installed. Be careful with this case, it might cause some undesired side-effects if you don't know what the plugin might be doing.

Parametrizing fixtures

Fixtures can be parametrized to use different values for each test where they are used. A parametrized fixture makes the test where it's used to run as many times as parameters the fixture has. Each test will run with a different parameter specified by the fixture:

1@pytest.fixture(params=[1, 2], ids=["1 passenger", "2 passengers"])
2def named_passengers(request):
3    passengers = [
4        {
5            "first_name": "User",
6            "last_name": "One",
7        },
8        {
9            "first_name": "User",
10            "last_name": "Two",
11        },
12    ]
13    return passengers[0: request.param]
14

In this case the test will run with the fixture values 1 and 2, which are available using the request context object. The ids parameter will indicate the id of the test that will run for each parameter.

Fixture setUp/tearDown

Fixtures ship with a mechanism for setting up and tearing down, very similar to the setUp and tearDown approach of xunit tests. This is accomplished by using the yield keyword:

1import pytest
2
3@pytest.fixture
4def my_fixture():
5    # setUp
6    
7    yield
8    
9    # tearDown
10

This is very useful for database fixtures or fixtures that made use of I/O operations or data store initialization/termination.

1import pytest
2
3@pytest.fixture(scope="class", autouse=True)
4def setup_db(io_loop):
5    """ Fixture to setup the db by performing some cleanups 
6    and closing the engine connection after each test class """
7    setup_db()
8    yield
9    db = get_db()
10    db.engine.close()
11

Note: this is an illustrative example, it won't run if you copy-paste it, it'll require the database to be configured and setup. ****

Mocking and patching

Background

Mocks are objects used in testing to fake other objects in order to assert information from them. In theory, there are different types of mocks:

  • Test stub: provide a canned response to method calls.
  • Test spy: real objects that behave like normal except when a specific condition is met.
  • Mocks: verify behaviour (calls) to a method.

In python, these three concepts are used interchangeably when we are referring to mocks. Mocks are useful for the following reasons:

  • Mocks help to eliminate dependencies: for unit tests, mocking the test dependencies will help you focus on the test at hand.
  • Mocks allow you to test that a method has been called, specially for those without a return value.
  • They are useful to test error handling.
  • Mocks eliminate dependencies with data stores and I/O operations.

Patching (or monkey patching) refers to the operation of replacing one real object for a fake one, usually a mock. It's specially useful for testing API calls, where you don't want to call the real API during tests or in case of mocking the data storage to avoid doing real I/O operations during test.

Mock and patch in pytest

Patching is the main mocking mechanism that python offers, it temporarily replaces your target (the object to be patched) with a different object, it's useful when you don't want to actually call an object. By default, patch will return a MagicMock object.

There are several alternatives to patch and mock in python, here's some of the most popular ones:

  • pytest's monkeypatch: pytest includes a fixture called monkeypatch this fixture serves as the API for patching in pytest:
1import time
2
3def testmonkey(monkeypatch):
4    """ Fakes the sleep call. This test will run fast"""
5    mock = lambda s: None
6    monkeypatch.setattr(time, "sleep", mock)
7    time.sleep(10)
8    assert True
9
  • The patch method of the mocker fixture: the mocker fixture from the pytest-mock library acts as a thin wrapper over the mock library API. It adds some functionality like spy or stub to the standard mock API:
1import time
2
3def testmocker(mocker):
4    """ Fakes the sleep call. This test will run fast""" 
5    mocker.patch.object(time, "sleep")
6    time.sleep(10)
7    assert True
8
  • Patch from unittest.mock python library: the python standard library offers a mock library which is highly maintained and functional. It's under the unittest module and offers as well as the ones above, mocking and patching options:
1import time
2from unittest.mock import patch
3
4def test_patch():
5    """ Fakes the sleep call. This test will run fast. Note the need
6    for the context manager when using unittest mock. """
7    with patch.object(time, "sleep"):
8        time.sleep(10)
9        assert True
10

For this guide we will stick to unittest.mock, which is the standard library for writing mocks in python. As mocker is a wrapper over the unittest library, all the concepts discussed here will apply to this library as well. If you want a more detailed information about what mocker brings, head over to their documentation.

Patching

There are mainly three ways to patch in pytest:

  • Using the patch decorator:
1import pytest
2import time
3from unittest import mock
4
5def sleepy_function():
6    time.sleep(10)
7
8@mock.patch("time.sleep")
9def test_mock_decorator(m_time):
10    """ Test patching the sleep function, this test will run fast"""
11    sleepy_function()
12    assert True
13
14@mock.patch("time.sleep")
15class TestSleep:
16    def test_mock_method(self, m_time):
17        sleepy_function()
18        assert True
19
  • Using a context manager:
1import pytest
2import time
3from unittest import mock
4
5def sleepy_function():
6    time.sleep(10)
7
8def test_mock_context():
9    """ Test patching the sleep function, this test will run fast"""
10    with mock.patch("time.sleep") as m_time:
11        sleepy_function()
12        assert True
13
  • Manually patching:
1import pytest
2import time
3from unittest import mock
4
5def sleepy_function():
6    time.sleep(10)
7
8def test_mock_manual():
9    """ Test patching the sleep function, this test will run fast"""
10    m_time_patcher = mock.patch("time.sleep")
11    m_time_patcher.start()
12    sleepy_function()
13    assert True
14    m_time_patcher.stop()
15

Depending on the case at hand, each patching strategy has its use. It's important to notice that patching manually can be dangerous, you'll have to explicitly call start() and stop() on the patcher so that the patch won't leak to other tests. This won't happen with the decorator or the context manager since both of them handle the clean up by themselves.

Avoid patching manually unless you are very sure of what you are doing. Patching manually can cause unexpected behaviours if the patcher is not stopped correctly. It's recommended to stick to the decorator or the context manager.

These strategies also define different scopes for the patching to be applied:

  • The manual patch (start/stop) will work as long as the stop() method for the patch is not called.
  • The class decorator will apply the patch for each method of a class.
  • The function decorator will apply the patch for the whole test function.
  • The context manager will apply the patch only inside of the context manager scope.

The patch method not only accepts a string of the dotted path to the target object. There are other interesting methods in the patch API:

  • mock.patch.object : for patching methods or attributes of an object.
  • mock.patch.dict : for patching dictionaries and mapping-like objects.
  • mock.patch.multiple : to perform multiple patches over the same target.

If you want to go more in depth on patching, the details about each one of these are very well explained in the unittest mock official documentation.

Identifying the target for patch

Identifying the target to patch can be tricky, it has the following rules:

  • The target must be importable from your test file.
  • You should patch where the object is used and not where the object is defined. Example:
1#-- my_module.py
2
3import logging
4from mockredis import redis
5
6def get_key():
7    redis.get("my_key")
8
9#-- test_redis.py
10
11# patch my_module.redis instead of mockredis.redis
12@patch("my_module.redis.get", return_value="hello")    
13def test_redis_get(m_redis_get):
14    assert m_redis_get("my_key") == "hello"
15

If you run dir(my_module) in the python shell you'll see the redis object that is imported in the module. The focus of patching is on the objects defined in the target module's scope.

The result of patching is the replacement of the target module or object by a fake object that we call a mock. A mock is a fake object that we can use to assert different things about the target object that we are mocking. In pytest we deal with two different mock objects:

  • Mock: is the core mock class from unittest it allows you to create stub objects for your tests.
  • MagicMock: is a wrapper over Mock which will provide you with a mock that has the python object's "magic methods" (dunder methods) already mocked.
1# With magic mock
2mock = MagicMock()
3mock.__str__.return_value = 'mock test'
4
5# With mock
6mock = Mock()
7mock.__str__ = Mock(return_value='mock test')
8str(mock)
9

Patching builtin functions

There are times where you need to mock some python builtin functions, a good example is when you want to open a file for reading but you don't really want to do any I/O operations. To patch builtin functions you just have to add builtins to the target object path:

1#-- my_module.py
2def read_file():
3    f = open("somefile.txt", "r")
4    return f.read()
5
6#-- test_builtin.py
7from unittest import mock
8from my_module import read_file
9
10def test_mock_open():
11    file_mock = mock.MagicMock()
12    file_mock.read.return_value = "Hello world"
13
14    # Patching the open function
15    with mock.patch("builtins.open", return_value=file_mock): 
16        assert read_file() == "Hello world"
17

Patch parameters

The mocking behaviour and results from the patch call can be specified with the parameters passed to the patch call:

  • return_value: specifies the value to be returned when the mocked object of the patch is called.
1#-- my_module.py
2from datetime import datetime
3
4def time_now():
5    return datetime.now()
6    
7
8# -- test_distro.py
9from datetime import datetime
10from unittest import mock
11
12def test_distros():
13    with mock.patch(
14        "my_module.time_now", 
15        return_value=datetime(2020,12,11)
16    ) as m_time_now:
17        assert m_time_now == datetime(2020,12,11)
18

  • side_effect: the name of this parameter can be confusing, what it really does is that it defines the behaviour or what will happen when calling the mocked object. A side effect can be:

    • A callable: **the object to be called when the mock is called.

    • An exception class: will raise an exception when the mock is called.

    • A list of callable objects: each of these objects will be returned consecutively as the mock is being called in the application. The first call to the mock will return the first element of the list, the second call will return the second element and so on.

      1#-- req.py
2import requests
3
4def fetch():
5    result = requests.get("google.com")
6    return result
7
8# -- test_api.py
9import pytest
10import requests
11from unittest import mock
12
13def test_callable():
14    def mock_fetch():
15        return "train1"
16
17    with mock.patch("req.fetch", side_effect=mock_fetch) as m_fetch:
18        assert m_fetch() == "train1" # This will call the mock_fetch instead
19
20def test_fetch_exception():
21    with mock.patch("req.fetch", side_effect=requests.Timeout) as m_fetch:
22        with pytest.raises(requests.Timeout):
23            m_fetch()  # This test will pass
24
25def test_fetch_multiple_returns():
26    with mock.patch("req.fetch", side_effect=["train1", "train2"]) as m_fetch:
27        # Each call will return a different result from the side_effect list
28        result_train1 = m_fetch()
29        result_train2 = m_fetch()
30        assert result_train1 == "train1"
31        assert result_train2 == "train2"
32

  • new_callable: specifies the callable object or class that will be used to create the mock object:

1#-- req.py
2import requests
3
4def fetch():
5    result = requests.get("google.com")
6    return result
7
8# -- test_api.py
9import pytest
10import requests
11from unittest import mock
12
13def test_new_callable():
14    with mock.patch("req.fetch", new_callable=mock.Mock) as m_fetch:
15        m_fetch.return_value = "train1"
16        assert m_fetch() == "train1"  # m_fetch is an instance of Mock instead of MagicMock
17        assert isinstance(m_fetch, mock.Mock)
18
  • new: will replace the target object with a new object
1def test_new():
2    class NewFetcher:
3        ...
4
5    with mock.patch("req.fetch", new=NewFetcher) as m_fetch:
6        assert issubclass(m_fetch, NewFetcher)
7

For a full description of the patch method parameters refer to the unittest mock patch documentation.

Mocking

Mock objects are used to replace target objects by a fake objects. You can call them, dynamically create attributes that will behave as new mocks and access these attributes. They keep track of how you use them so that you can make assertions about their behaviour. The main two mock classes we will work with are MagicMock and Mock, see the difference between Mock and MagicMock.

Mocking gotchas

Because of the way that MagicMock works —by simply mocking a target object— any method used in the mocked object will be dynamically created as if it existed in the target object. Mocking has the following problems:

  • Mocks don't know about misspelled asserts. The following test will pass:
1from unittest import mock
2
3def hello_world():
4    print("Hello world")
5
6def test_mock_foo():
7    with mock.patch("builtins.print") as m_print:
8        hello_world()
9        m_print.assrt_called_with("Hello world")
10
  • Mocked objects can be called incorrectly:
1# -- train_station.py
2class TrainStation:
3    stations = {
4        "barcelona": "sants"
5    }
6    def get_train_station(self, station_id):
7        return stations[station_id]
8
9# -- test_stations.py
10def test_my_mock():
11
12    with mock.patch.object(TrainStation, "get_train_station") as m_get:
13        m_get("barcelona", "madrid", "valencia") # Called get with the wrong parameters
14        assert m_get.called_once()
15
  • MagicMock doesn't know about the attributes of a mocked object:
1# -- train_station.py
2class TrainStation:
3    stations = {
4        "barcelona": "sants"
5    }
6    def get_train_station(self, station_id):
7        return stations[station_id]
8
9# -- test_stations.py
10def test_my_mock():
11
12    with mock.patch("train_station.TrainStation") as m_train_station:
13        m_train_station.this_method_does_not_exist() # This will test won't complain
14        assert True
15

Mock spec, autospec, spec_set

To avoid such issues, the spec parameter can be used to create a mock that resembles more the target object being mocked.

Setting spec=True makes your mock know about the methods of the object being mocked. In general terms: it makes the mock look like the object you are patching. Calling dir() over the MagicMock instance will also return the object's method signatures:

1# -- train.py
2
3class Train:
4    def origin(self, name):
5        ...
6
7    def destination(self, name):
8        ...
9
10# -- test_spec.py
11from unittest.mock import patch
12
13@patch("train.Train", spec=True)
14def test_spec(m_train):
15    print(dir(m_train))
16    assert False
17
18# Result of dir(m_train):
19$> 
20['__class__',
21 '__delattr__',
22 ...
23 'assert_any_call',
24 'assert_called',
25 ...
26 'call_args',
27 'call_args_list',
28 'dest', # Train class method
29 'origin', # Train class method
30 'reset_mock',
31 'return_value',
32 'side_effect']
33

Trying to access a method that's not part of the Mock API or the Train object will result in an AttributeError. The spec parameter can also be used directly in a mock to wrap a target object. This is useful for keeping the object's API along with having the mocking capabilities:

1from unittest import mock
2from train import Train
3
4mock_obj = Mock(spec=Train)
5dir(mock_obj) # The output will include the Train's class methods
6mock_obj.random_method() # Will raise an AttributeError
7

However, by using spec, the mock won't know about the attributes of attributes of your objects, it doesn't introspect them, that's where the autospec parameter is useful. autospec doesn't know about dynamically created attributes, so it'll raise an exception if you add an attribute in the __init__ method of a class. Based on the example above:

1from unittest.mock import patch
2
3@patch("train.Train", spec=True)
4def test_spec(m_train):
5    m_train.origin()  # This call WILL NOT fail 
6    m_train.origin.assert_called_once()
7
8@patch("train.Train", autospec=True)
9def test_spec2(m_train):
10
11    m_train.origin()  # This call WILL fail because it's missing the name parameter
12    m_train.origin.assert_called_once()
13

The spec and autospec parameters prevent from getting attributes that don't exist, but you can still set them in the mock. If you want to avoid adding attributes that don't exist to your mock, you could use the spec_set parameter, which prevents you from setting such attributes:

1from unittest.mock import patch
2
3@patch("train.Train", autospec=True, spec_set=True)
4def test_spec2(m_train):
5
6    m_train.new_origin = "Barcelona" # This call will raise an AttributeError
7    m_train.origin.assert_called_once()
8

Note: spec or autospec can be used along with spec_set, but *if you try to pass spec and autospec* together, the test will fail with TypeError: Can't specify spec and autospec.

You can also pass an object to the autospec call, this will set the spec of the mock to be the same as the object you are passing.

autospec could be slightly dangerous, any code that will run by instrospection (accessing attributes) will be triggered by autospec and it'll run. This refers to code that makes use of descriptors or properties. Make sure that your code is safe for introspection.

Asserting on mocks

Mocks are very useful for testing code behaviour: test that a method or function is called, what parameters where passed to the call, how many times it's called, etc. Mock objects save this information in their instance objects. The mock API provides several methods to test behaviour, to name a few:

  • assert_called: tests that the mock has been called
  • assert_called_with(*args, kwargs):** **tests that the mock has been called with certain parameters.
  • assert_called_once: **tests a mock has been called once.
  • call_count: returns the number of times a mock is called.

They are quite self-explanatory, however, you can find the complete explanation and examples in the unittest documentation for mock.

Parametrizing tests

Similar to fixtures, tests can be parametrized. You should use the @pytest.mark.parametrize decorator to parametrize the tests. The syntax is slightly different compared to fixture parametrization. The first parameter to the decorator is a string specifying the variables that contain the parametrized values, and the second parameter is a list of tuples representing the different test run values. Each tuple contain the values of the parametrized variables for that test run. Example: 

1import pytest
2
3def passenger_ids(passenger_name: str) -> str:
4    if passenger_name == "Bob":
5        return "1" 
6    elif passenger_name == "Alice":
7        return "2"
8   
9    return "3"
10
11@pytest.mark.parametrize(
12    "passenger_name,passenger_id", [("Bob", "1"), ("Alice", "2")])
13def test_passenger_ids(passenger_name: str, passenger_id: str):
14    assert passenger_ids(passenger_name) == passenger_id
15

Class parametrization and dynamic generation of test methods is not possible by default in pytest, you can check the parameterized pytest plugin which extends the default pytest parametrization system.

Asynchronous testing

If you want to test asynchronous functions, your test should also be asynchronous. pytest provides a library called pytest-asyncio that allows you to define tests and fixtures to be called asynchronously. If you are using the Tornado framework, you should then use the pytest-tornado library for asynchronous testing

Async testing with asyncio

Tests can be marked as asynchronous with pytest-asyncio. Asynchronous tests will let you call asynchronous functions that you want to test in your code. To mark a test as asynchronous you should use the @pytest.mark.asyncio mark decorator and write your test as a coroutine using the async/await syntax. Example:

1@pytest.mark.asyncio
2async def test_get_trainstation(redis):
3    train_station = await redis.get("station_key")
4    assert train_station == "Barcelona Sants"
5

Note: this is an illustrative example, it won't run if you copy-paste it, it'll require redis to be set up and the key to exists in the redis store.

Warning: mixing pytest-asyncio and Tornado can lead to undesired effects in your tests. Tornado abstracts the use of the asyncio event loop in the IOLoop, mixing these two concepts can produce errors when dealing with loop logic and executing asynchronous coroutines. Make sure you use pytest-tornado whenever you are testing code from the Tornado framework.

Async testing with Tornado

Even though Tornado is built on asyncio it's recommended to write tests using the pytest-tornado library. In Tornado, if you want to test asynchronous code you should mark your test with the @pytest.mark.gen_test decorator. Besides the async/await syntax, gen_test also support writing your test as a generator using the yield keyword.

Avoid mixing asyncio testing with tornado, *it could lead to undesired effects and errors because of the way Tornado works with the IOLoop, keep your code consistent and write your tests in the framework you are using.*

Async fixtures

Asynchronous fixtures are defined similarly to regular fixtures, except that they have to be coroutines or generators. An async *fixture is defined using the async/await syntax and decorated as a pytest fixture. Example using pytest-tornado*:

1import pytest
2
3@pytest.fixture(scope="function", autouse=True)
4async def setup_db(io_loop):    
5    # setup
6    await utils_setup_db()   
7    
8    yield
9
10    # teardown
11    db = await get_db()
12    db.engine.close()
13
14

Note: this is an illustrative example, it won't run if you copy-paste it since it requires a lot of database set up.

Both libraries pytest-tornado and pytest-asyncio come with fixtures out of the box to make it easier to work with the event loop, and in the case of tornado: with the application, the http server, the client, and the urls.

Async mocks

When working with asynchronous tests it's possible that you will need to mock asynchronous functions, and your tests should be able to call these mocks asynchronously. The unittest.mock library provides an AsyncMock class to handle asynchronous mocks. This class is available starting from python 3.8. Example:

1import pytest
2from unittest import mock
3
4class FakeRedis:
5    mock_store = mock.AsyncMock()
6    mock_store.return_value = {"station_key": "Barcelona Sants"}
7
8    @property
9    async def store(self):
10        return await self.mock_store()
11
12    async def get(self, key):
13        store = await self.store
14        return store.get(key)
15
16# Instantiate a fake redis store for the sake of the example
17redis = FakeRedis()
18
19@pytest.mark.asyncio
20async def test_get_trainstation():
21    train_station = await redis.get("station_key")
22    assert train_station == "Barcelona Sants"
23

If you are working with a python version below 3.8, you could use the asynctest library. This library helps with testing for asyncio libraries and includes mock helpers for asynchronous code.

As of python 3.9, the unittest mock library is capable of providing you with a MagicMock or an AsyncMock depending on the target object to be mocked:

  • If the target object is an async function or method the call to patch will return an AsyncMock.
  • If the target object is an ordinary object, it'll return a MagicMock.

If you are working with a version below python 3.9, you should build your asynchronous mocks yourself or use the the asynctest library. You can make use of the patch parameters like new_callable and new to use your own async mocks when testing asynchronous code.

Useful links

Libraries and documentation

Sources

More info