This README.md describes how to run the experiments involving these case studies. The following steps assume you have installed pytest and the probtest plugin installed.
We consider two implementations of the skip list data structure: one without errors (presumably) and one with injected bugs.
To run the test suite for the error-free implementation of the skip list data structure, one can run it using probtest:
cd case_studies/skip_list
pytest --probtest --Pbug 0.25 --epsilon 0.05
To change the number of inserted nodes and skip list parameters, one must manually change these (in lines 32-34 in test_skip_list.py).
We describe below how to perform mutation testing and inject bugs in the implementation as done in the paper.
To perform mutation testing of the skip list implementation using mutmut, one must first install mutmut:
pip install mutmut
Then the mutation testing can be run (note that it is important to change directory as shown):
cd case_studies/skip_list
mutmut run
Mutmut is set up to run pytest with probtest. The specification provided to probtest can be changed in the setup.cfg file:
[mutmut]
paths_to_mutate=src/
backup=False
runner=pytest --probtest --Pbug 0.25 --epsilon 0.05
tests_dir=tests/
Be aware that mutmut makes changes to the script when it injects mutants. If suddenly stopped, mutmut may have failed to reverse them back to the original version. For more details, see the documentation of mutmut.
To run the test suite with the injected bugs as in the paper, one must first in test_skip_list.py import the buggy version of the skip list instead of the correct (in lines 22-23):
# sys.path.insert(1, './src')
sys.path.insert(1, './src_bugs')
Then, in the buggy implemenetation, one can inject the 8 bugs one by one by changing the value of the variable bug in line 14:
global bug; bug=1
Finally, the test suite can be run as usual:
cd case_studies/skip_list
pytest skip_list --probtest --Pbug 0.25 --epsilon 0.05
To run the experiments considered in the paper, one must run the test suite using probtest by providing the probability of errors occurring, for example:
pytest case_studies/frozen_lake --probtest --Pbug 0.1
To change the number of episodes of training, one must change it manually in the source code (line 47 in test_frozen_laky.py).