Programming is not just about writing code. It is important to verify that the code does what it should (and about fixing it if needed). The process of verification that the program works as expected is called testing.
You have probably tested your programs by trying to execute them, entered some input data and looked if the results are correct. It is harder to do it for bigger programs. Bigger programs have more possibilities what to do and it is harder to verify that all possibilities do what they should.
That is why developers write code that verifies their program instead of testing their programs manually.
Automated tests are functions that check that the program works correctly. You can execute the tests anytime and verify that the code works. The main benefit is that you can change the code in the future and let the tests verify that the change does not break existing functionalitye.
Up to now, we have used only the modules that come installed with Python,
for example, modules such as math or turtle.
There are many more libraries that are not included in Python
but you can install them to your Python environment and use them.
The library for testing in Python is called unittest.
It is quite difficult to use this library so we will use a better one.
We will install the library pytest which is easy to use and very popular.
You install libraries into your active virtual environment. We have learned how to create and activate a virtual environment in the lesson about Python installation. Make sure that you have activated a virtual environment.
Submit the following command. (It is a command-line command,
just as cd or mkdir; do not enter it into the Python console.)
(venv)$ python -m pip install pytest
What does Pip do?
python -m pip calls Python and tells it to execute the
pip module. This module can install and uninstall libraries.
(Similarly, when you created a virtual environment, you used the
command python -m venv – the venv module can create virtual environments.)
And the arguments install pytest tell Pip to install pytest.
You can display the help for the Pip module using the command
python -m pip --help.
For Windows users
If you use Windows, it is important to run Python programs using
python program.py, and not just program.py.
Although we always show python in our lessons,
it could work without it so far.
If you do not use the command python in the beginning, the program
could start in a different Python and different virtual environment,
where the pytest module might not have been installed.
We will show testing thtough a very simple example.
There is a function add that can add two numbers.
There is another function that tests if the
add function returns correct results for specific numbers.
Make a copy of the code into a file named test_addition.py
in a new empty directory.
The naming of files and test functions is important for pytest (with default settings).
It is important for names of files containing tests and test functions
to start with test_.
def add(a, b):
return a + b
def test_add():
assert add(1, 2) == 3
What does the test function do?
The assert statement evaluates the expression that follows it.
If the result is not true then it raises an exception
and it makes the test fail.
You can imagine that assert a == b does following:
if a != b:
raise AssertionError('Test failed!')
Do not use assert outside of test functions for now.
For "regular" code, the assert has functionality that
we will not explain now.
You execute tests with the command python -m pytest -v,
followed by the name of the file containing the tests.
By using this command you are telling: Python:
Execute the module named pytest,
in verbose mode, for the given file.
$ python -m pytest -v test_addition.py
============= test session starts =============
platform darwin -- Python 3.6.5, pytest-3.9.1, py-1.7.0, pluggy-0.8.0 --
rootdir: learn, inifile:
collecting ... collected 1 items
test_addition.py::test_add PASSED
============= 1 passed in 0.01 seconds =============This command scans the given file and calls all functions that start
with test_. It checks that they do not raise any exceptions,
for example, an exception raised by assert.
If an exception occurs, pytest shows a red message with
additional details that can help you find the bug and fix it.
You can omit the argument with the filename: python -m pytest -v
In this case, pytest scans the current directory and runs tests
in all files whose names start with test_. You can use a path to
a directory and pytest finds tests in it.
Try to change the add function (or its test) and see what happens
if a test fails.
You do not usually write tests in the same file with the regular code. Typically, you write tests in another file. This way, your code is easier to read, and it makes it possible to distribute only the code, without the tests, to someone who is interested only in executing the program.
Split the test_addition.py file: Move the add function to a new module addition.py.
In the test_addition.py file, keep only the test.
To the test_addition.py file, add from addition import add to the top
so the test can call the tested function.
The test should pass again.
Automated tests have to run "unattended". They are usually executed automatically and the failures are reported automatically (e.g. by email) and the code that passes all tests can be automatically released (installed to a system where it runs or is made available to customers).
What does this mean to us?
The input function will not work in tests. There is no-one who can reply.
This can make your work harder sometimes. Let's look at a more complex project: 1D (one-dimensional) tic-tac-toe.
If you do not have the 1D tic-tac-toe program, the following sections are only theoretical. If you study at home, complete the 1D tic-tac-toe lesson before continuing. The homework assignment is in PyLadies projects on page 2. (the English translation is at one-dimensional tic-tac-toe)..
The structure of your 1D tic-tac-toe code could look like this:
import random # (and possibly other import statements that are needed)
def move(board, space_number, mark):
"""Returns the board with the specified mark placed in the specified position"""
...
def player_move(board):
"""Asks the player what move should be done and returns the board
with the move played.
"""
...
input('What is your move? ')
...
def tic_tac_toe_1d():
"""Starts the game
It creates an empty board and runs player_move and computer_move alternately
until the game is finished.
"""
while ...:
...
player_move(...)
...
# Start the game:
tic_tac_toe_1d()
If you import this module, Python executes all commands in it from top to bottom.
The first command, import, makes some functions and variables available.
Imports do not usually have any side-effects.
The definitions of functions (def statements and everything in them)
just define the functions (but they do not execute the functions).
Calling the tic_tac_toe_1d function starts the game.
The tic_tac_toe_1d calls the player_move() function which calls input().
If you import this module from tests, the input fails
and the module is not imported.
If you want to import such a module from elsewhere – for example, you would like
to use move in another game – the import of the module requires the user to
play 1D tic-tac-toe!
The calling of tic_tac_toe_1d is a side-effect and we need to remove it.
Yeah but you cannot start the game without it! What about it?
You can create a new module.
Name it game.py and put just this call into it:
import tic_tac_toe
tic_tac_toe.tic_tac_toe_1d()
You cannot test this module because it calls input indirectly.
But you can execute it if you want to play.
Since you do not have tests for this module, it should be very simple:
one import and one statement.
You can import the original module from tests or other modules.
A test for the original module could look like this:
import tic_tac_toe
def test_move_to_empty_space():
board = tic_tac_toe.computer_move('--------------------')
assert len(board) == 20
assert board.count('x') == 1
assert board.count('-') == 19
The tests that verify that the program works correctly under correct conditions are called positive tests. But you can test what your program does under unexpected conditions.
The tests that check the behavior in case of "invalid" input
are called negative tests.
They can check for a specific negative result (for example
that a call like is_number_even(7) returns False),
or that a "reasonable" exception is raised.
For example, the computer_move function should raise an error
(for example ValueError) when the board is full.
It is much better to raise an exception than doing nothing and silently letting the program get stuck. You can use such function in a more complex program and be sure that you will get an understandable error when it is called under bad conditions. Then you can easily fix it.
Use the with statement and the raises function
to test that your code raises the expected exception.
The raises function is imported from the pytest module.
We will explain the with statement later.
You just need to know that it checks that the block of code below
raises the specified exception:
import pytest
import tic_tac_toe
def test_move_failure():
with pytest.raises(ValueError):
tic_tac_toe.computer_move('oxoxoxoxoxoxoxoxoxox')
{
"data": {
"sessionMaterial": {
"id": "session-material:2018/pyladies-en-prague:tests:2",
"title": "Testing",
"html": "\n \n \n\n <h1>Testing</h1>\n<p>Programming is not just about writing code. \nIt is important to verify that the code does what it should \n(and about fixing it if needed).\nThe process of verification that the program works as expected is called <em>testing</em>.</p>\n<p>You have probably tested your programs by trying to execute them, \nentered some input data and looked if the results are correct.\nIt is harder to do it for bigger programs. \nBigger programs have more possibilities what to do and it is harder \nto verify that all possibilities do what they should.</p>\n<p>That is why developers write code that verifies their program \ninstead of testing their programs manually.</p>\n<p><em>Automated tests</em> are functions that check that the program works correctly.\nYou can execute the tests anytime and verify that the code works.\nThe main benefit is that you can change the code in the future\nand let the tests verify that the change does not break existing functionalitye.</p>\n<h2>Installing the pytest library</h2>\n<p>Up to now, we have used only the modules that come installed with Python, \nfor example, modules such as <code>math</code> or <code>turtle</code>.\nThere are many more <em>libraries</em> that are not included in Python\nbut you can install them to your Python environment and use them.</p>\n<p>The library for testing in Python is called <code>unittest</code>.\nIt is quite difficult to use this library so we will use a better one.\nWe will install the library <code>pytest</code> which is easy to use and very popular.</p>\n<p>You install libraries into your active virtual environment.\nWe have learned how to create and activate a virtual environment\nin the lesson about <a href=\"/2018/pyladies-en-prague/beginners/install/\">Python installation</a>.\nMake sure that you have activated a virtual environment.</p>\n<p>Submit the following command. (It is a command-line command, \njust as <code>cd</code> or <code>mkdir</code>; do not enter it into the Python console.)</p>\n<div class=\"highlight\"><pre><span></span><span class=\"gp\">(venv)$ </span>python -m pip install pytest\n</pre></div><div class=\"admonition note\"><p class=\"admonition-title\">What does Pip do?</p>\n<p><code>python -m pip</code> calls Python and tells it to execute the\n<code>pip</code> module. This module can install and uninstall libraries. \n(Similarly, when you created a virtual environment, you used the\ncommand <code>python -m venv</code> – the <code>venv</code> module can create virtual environments.)\nAnd the arguments <code>install pytest</code> tell Pip to install <code>pytest</code>.</p>\n<p>You can display the help for the Pip module using the command\n<code>python -m pip --help</code>.</p>\n</div><div class=\"admonition warning\"><p class=\"admonition-title\">For Windows users</p>\n<p>If you use Windows, it is important to run Python programs using\n<code>python program.py</code>, and not just <code>program.py</code>.\nAlthough we always show <code>python</code> in our lessons, \nit could work without it so far.\nIf you do not use the command <code>python</code> in the beginning, the program \ncould start in a different Python and different virtual environment, \nwhere the <code>pytest</code> module might not have been installed.</p>\n</div><h2>Writing tests</h2>\n<p>We will show testing thtough a very simple example.\nThere is a function <code>add</code> that can add two numbers.\nThere is another function that tests if the \n<code>add</code> function returns correct results for specific numbers.</p>\n<p>Make a copy of the code into a file named <code>test_addition.py</code>\nin a new empty directory.</p>\n<p>The naming of files and test functions is important for <code>pytest</code> (with default settings). \nIt is important for names of files containing tests and test functions\nto start with <code>test_</code>.</p>\n<div class=\"highlight\"><pre><span></span><span class=\"k\">def</span> <span class=\"nf\">add</span><span class=\"p\">(</span><span class=\"n\">a</span><span class=\"p\">,</span> <span class=\"n\">b</span><span class=\"p\">):</span>\n <span class=\"k\">return</span> <span class=\"n\">a</span> <span class=\"o\">+</span> <span class=\"n\">b</span>\n\n<span class=\"k\">def</span> <span class=\"nf\">test_add</span><span class=\"p\">():</span>\n <span class=\"k\">assert</span> <span class=\"n\">add</span><span class=\"p\">(</span><span class=\"mi\">1</span><span class=\"p\">,</span> <span class=\"mi\">2</span><span class=\"p\">)</span> <span class=\"o\">==</span> <span class=\"mi\">3</span>\n</pre></div><p>What does the test function do?</p>\n<p>The <code>assert</code> statement evaluates the expression that follows it.\nIf the result is not true then it raises an exception \nand it makes the test fail.\nYou can imagine that <code>assert a == b</code> does following:</p>\n<div class=\"highlight\"><pre><span></span><span class=\"k\">if</span> <span class=\"n\">a</span> <span class=\"o\">!=</span> <span class=\"n\">b</span><span class=\"p\">:</span>\n <span class=\"k\">raise</span> <span class=\"ne\">AssertionError</span><span class=\"p\">(</span><span class=\"s1\">'Test failed!'</span><span class=\"p\">)</span>\n</pre></div><div class=\"admonition note\"><p>Do not use <code>assert</code> outside of test functions for now.\nFor "regular" code, the <code>assert</code> has functionality that\nwe will not explain now.</p>\n</div><h2>Running tests</h2>\n<p>You execute tests with the command <code>python -m pytest -v</code>, \nfollowed by the name of the file containing the tests.\nBy using this command you are telling: <strong>Python</strong>: \nExecute the <strong>m</strong>odule named <strong>pytest</strong>,\nin <strong>v</strong>erbose mode, for the given file.</p>\n<div class=\"highlight\"><pre><code>$ python -m pytest -v test_addition.py\n<span style=\"font-weight: bold\">============= test session starts =============</span>\nplatform darwin -- Python 3.6.5, pytest-3.9.1, py-1.7.0, pluggy-0.8.0 -- \nrootdir: learn, inifile: \n<span style=\"font-weight: bold\">collecting ...</span> collected 1 items\n\ntest_addition.py::test_add <span style=\"color: #00aa00\">PASSED</span>\n\n<span style=\"color: #00aa00\">============= 1 passed in 0.01 seconds =============</span></code></pre></div><p>This command scans the given file and calls all functions that start\nwith <code>test_</code>. It checks that they do not raise any exceptions, \nfor example, an exception raised by <code>assert</code>.\nIf an exception occurs, <code>pytest</code> shows a red message with\nadditional details that can help you find the bug and fix it.</p>\n<div class=\"admonition note\"><p>You can omit the argument with the filename: <code>python -m pytest -v</code>\nIn this case, <code>pytest</code> scans the current directory and runs tests\nin all files whose names start with <code>test_</code>. You can use a path to \na directory and <code>pytest</code> finds tests in it.</p>\n</div><p>Try to change the <code>add</code> function (or its test) and see what happens\nif a test fails.</p>\n<h2>Test modules</h2>\n<p>You do not usually write tests in the same file with the regular code.\nTypically, you write tests in another file.\nThis way, your code is easier to read, and it makes it possible to distribute \nonly the code, without the tests, to someone who is interested only in executing the program.</p>\n<p>Split the <code>test_addition.py</code> file: Move the <code>add</code> function to a new module <code>addition.py</code>.\nIn the <code>test_addition.py</code> file, keep only the test.\nTo the <code>test_addition.py</code> file, add <code>from addition import add</code> to the top\nso the test can call the tested function.</p>\n<p>The test should pass again.</p>\n<h2>Executable modules</h2>\n<p>Automated tests have to run "unattended".\nThey are usually executed automatically and the failures are reported\nautomatically (e.g. by email) and the code that passes all tests can\nbe automatically released (installed to a system where it runs \nor is made available to customers).</p>\n<p>What does this mean to us?\nThe <code>input</code> function will not work in tests. There is no-one who can reply.</p>\n<p>This can make your work harder sometimes. Let's look at a more complex project: 1D (one-dimensional) tic-tac-toe.</p>\n<div class=\"admonition note\"><p>If you do not have the 1D tic-tac-toe program, the following sections are only theoretical.\nIf you study at home, complete the 1D tic-tac-toe lesson before continuing.\nThe homework assignment is in <a href=\"http://pyladies.cz/v1/s004-strings/handout/handout4.pdf\">PyLadies projects</a>\non page 2. (the English translation is at <a href=\"/2018/pyladies-en-prague/beginners-en/tictactoe/\">one-dimensional tic-tac-toe</a>)..</p>\n</div><p>The structure of your 1D tic-tac-toe code could look like this:</p>\n<div class=\"highlight\"><pre><span></span><span class=\"kn\">import</span> <span class=\"nn\">random</span> <span class=\"c1\"># (and possibly other import statements that are needed)</span>\n\n<span class=\"k\">def</span> <span class=\"nf\">move</span><span class=\"p\">(</span><span class=\"n\">board</span><span class=\"p\">,</span> <span class=\"n\">space_number</span><span class=\"p\">,</span> <span class=\"n\">mark</span><span class=\"p\">):</span>\n <span class=\"sd\">"""Returns the board with the specified mark placed in the specified position"""</span>\n <span class=\"o\">...</span>\n\n<span class=\"k\">def</span> <span class=\"nf\">player_move</span><span class=\"p\">(</span><span class=\"n\">board</span><span class=\"p\">):</span>\n <span class=\"sd\">"""Asks the player what move should be done and returns the board</span>\n<span class=\"sd\"> with the move played.</span>\n<span class=\"sd\"> """</span>\n <span class=\"o\">...</span>\n <span class=\"nb\">input</span><span class=\"p\">(</span><span class=\"s1\">'What is your move? '</span><span class=\"p\">)</span>\n <span class=\"o\">...</span>\n\n<span class=\"k\">def</span> <span class=\"nf\">tic_tac_toe_1d</span><span class=\"p\">():</span>\n <span class=\"sd\">"""Starts the game</span>\n\n<span class=\"sd\"> It creates an empty board and runs player_move and computer_move alternately</span>\n<span class=\"sd\"> until the game is finished.</span>\n<span class=\"sd\"> """</span>\n <span class=\"k\">while</span> <span class=\"o\">...</span><span class=\"p\">:</span>\n <span class=\"o\">...</span>\n <span class=\"n\">player_move</span><span class=\"p\">(</span><span class=\"o\">...</span><span class=\"p\">)</span>\n <span class=\"o\">...</span>\n\n<span class=\"c1\"># Start the game:</span>\n<span class=\"n\">tic_tac_toe_1d</span><span class=\"p\">()</span>\n</pre></div><p>If you import this module, Python executes all commands in it \nfrom top to bottom.</p>\n<p>The first command, <code>import</code>, makes some functions and variables available.\nImports do not usually have any side-effects.</p>\n<p>The definitions of functions (<code>def</code> statements and everything in them) \njust define the functions (but they do not execute the functions).</p>\n<p>Calling the <code>tic_tac_toe_1d</code> function starts the game.\nThe <code>tic_tac_toe_1d</code> calls the <code>player_move()</code> function which calls <code>input()</code>.</p>\n<p>If you import this module from tests, the <code>input</code> fails \nand the module is not imported.</p>\n<div class=\"admonition note\"><p>If you want to import such a module from elsewhere – for example, you would like\nto use <code>move</code> in another game – the import of the module requires the user to \nplay 1D tic-tac-toe!</p>\n</div><p>The calling of <code>tic_tac_toe_1d</code> is a side-effect and we need to remove it.\nYeah but you cannot start the game without it! What about it?</p>\n<p>You can create a new module.\nName it <code>game.py</code> and put just this call into it:</p>\n<div class=\"highlight\"><pre><span></span><span class=\"kn\">import</span> <span class=\"nn\">tic_tac_toe</span>\n\n<span class=\"n\">tic_tac_toe</span><span class=\"o\">.</span><span class=\"n\">tic_tac_toe_1d</span><span class=\"p\">()</span>\n</pre></div><p>You cannot test this module because it calls <code>input</code> indirectly.\nBut you can execute it if you want to play.\nSince you do not have tests for this module, it should be very simple: \none import and one statement.</p>\n<p>You can import the original module from tests or other modules.</p>\n<p>A test for the original module could look like this:</p>\n<div class=\"highlight\"><pre><span></span><span class=\"kn\">import</span> <span class=\"nn\">tic_tac_toe</span>\n\n<span class=\"k\">def</span> <span class=\"nf\">test_move_to_empty_space</span><span class=\"p\">():</span>\n <span class=\"n\">board</span> <span class=\"o\">=</span> <span class=\"n\">tic_tac_toe</span><span class=\"o\">.</span><span class=\"n\">computer_move</span><span class=\"p\">(</span><span class=\"s1\">'--------------------'</span><span class=\"p\">)</span>\n <span class=\"k\">assert</span> <span class=\"nb\">len</span><span class=\"p\">(</span><span class=\"n\">board</span><span class=\"p\">)</span> <span class=\"o\">==</span> <span class=\"mi\">20</span>\n <span class=\"k\">assert</span> <span class=\"n\">board</span><span class=\"o\">.</span><span class=\"n\">count</span><span class=\"p\">(</span><span class=\"s1\">'x'</span><span class=\"p\">)</span> <span class=\"o\">==</span> <span class=\"mi\">1</span>\n <span class=\"k\">assert</span> <span class=\"n\">board</span><span class=\"o\">.</span><span class=\"n\">count</span><span class=\"p\">(</span><span class=\"s1\">'-'</span><span class=\"p\">)</span> <span class=\"o\">==</span> <span class=\"mi\">19</span>\n</pre></div><h2>Positive and negative tests</h2>\n<p>The tests that verify that the program works correctly \nunder correct conditions are called <em>positive tests</em>.\nBut you can test what your program does under unexpected conditions.</p>\n<p>The tests that check the behavior in case of "invalid" input\nare called <em>negative tests</em>.\nThey can check for a specific negative result (for example \nthat a call like <code>is_number_even(7)</code> returns <code>False</code>), \nor that a "reasonable" exception is raised.</p>\n<p>For example, the <code>computer_move</code> function should raise an error \n(for example <code>ValueError</code>) when the board is full.</p>\n<div class=\"admonition note\"><p>It is much better to raise an exception than doing nothing \nand silently letting the program get stuck.\nYou can use such function in a more complex program \nand be sure that you will get an understandable error\nwhen it is called under bad conditions. \nThen you can easily fix it.</p>\n</div><p>Use the <code>with</code> statement and the <code>raises</code> function \nto test that your code raises the expected exception.\nThe <code>raises</code> function is imported from the <code>pytest</code> module.\nWe will explain the <code>with</code> statement later.\nYou just need to know that it checks that the block of code below \nraises the specified exception:</p>\n<div class=\"highlight\"><pre><span></span><span class=\"kn\">import</span> <span class=\"nn\">pytest</span>\n\n<span class=\"kn\">import</span> <span class=\"nn\">tic_tac_toe</span>\n\n<span class=\"k\">def</span> <span class=\"nf\">test_move_failure</span><span class=\"p\">():</span>\n <span class=\"k\">with</span> <span class=\"n\">pytest</span><span class=\"o\">.</span><span class=\"n\">raises</span><span class=\"p\">(</span><span class=\"ne\">ValueError</span><span class=\"p\">):</span>\n <span class=\"n\">tic_tac_toe</span><span class=\"o\">.</span><span class=\"n\">computer_move</span><span class=\"p\">(</span><span class=\"s1\">'oxoxoxoxoxoxoxoxoxox'</span><span class=\"p\">)</span>\n</pre></div>\n\n\n "
}
}
}