Functional testing a multilanguage Flask website with Pytest

Testing is hard work. It is totally different from creating or modifying functionality. I know, I also developed computer hardware, integrated circuits, hardware test systems. I wrote tests for CPUs, computer products, developed test systems. With hardware you cannot make mistakes. Mistakes may be the end of your company.

Welcome to the wonderful world of software testing

With software many things are different. Mistakes are tolerated. I do not blame small companies. But big tech giants like Google, Apple, Facebook, Microsoft should be blamed. Most software is complex or at least uses third-party libraries or connect to remote systems. Testing an application has become a complex task.

Why I choose to write functional tests (first)

There are many difference types of tests for software, to mention the most well-known with developers:

• Unit test
• Integration test
• Functional test
• End-to-end test

Unit tests are easy to write but even if they all pass this does not guarantee that my Flask application is working. Integration tests are also not that difficult. If you write libraries these tests make perfect sense. Just pull in the classes and write some tests. But there is no guarantee that my application will be working when only doing these tests.

With limited time how can I make sure that my Flask CMS/Blog application is actually working? I decided to go for the functional tests and end-to-end tests. This is much more work but at least I know that the visitor of the website sees the right things.

Tools for testing

I added a 'testing' option to my docker setup. This is like my 'development' option, with code outside the container, but with a few extra packages,

• Flake8
• Pytest
• Pytest-cov
• Pytest-flake8

Pytest is nice test framework if you want to test your Flask application. When you install Pytest-flake8 you can also run Flake8 from Pytest:

python3 -m pytest app_frontend/blueprints/demo  --flake8 -v

But I prefer to run flake8 by itself, in this case ignoring long lines:

flake8 --ignore=E501 app_frontend/blueprints/demo

Test setup

Starting with Pytest is not very difficult. I created a tests_app_frontend directory in the project directory:

.
|-- alembic
|-- app_admin
|-- app_frontend
|   |-- blueprints
|   |-- templates
|   |-- translations
|   |-- __init__.py
|   `-- ...
|-- shared
|-- tests_frontend
|   |-- helpers
|   |   |-- helpers_cleanup.py
|   |   |-- helpers_globals.py
|   |   `-- helpers_utils.py
|   |-- language_logs
|   |-- language_logs_ok
|   |-- conftest.py
|   |-- pytest.ini
|   |-- ...
|   |-- test_ay_account.py
|   |-- ...
|   |-- test_an_auth_login.py
|   `-- ...

The pytest.ini file:

[pytest]
norecursedirs=tests_frontend/helpers

cache_dir=/home/flask/project/tests_frontend/cache
console_output_style=progress

log_file=/home/flask/project/tests_frontend/testresults_frontend.log
log_file_level=DEBUG
log_file_format=%(asctime)s %(levelname)s %(message)s
log_file_date_format=%Y-%m-%d %H:%M:%S

log_level=DEBUG
python_files=test_*.py

flake8-max-line-length=200

The helpers directory contains functions that can make your life much more easy, in fact after writing a few tests, I was looking at minimizing my test code, and spend a lot of time writing these helper functions.

Cleanup functions in helpers_cleanup.py are part of the test and run before the actual test, examples:

def delete_test_contact_forms_from_db(client=None, lang_code=None, dbg=False):
    ...
    
def delete_test_contact_form_emails_from_db(client=None, lang_code=None, dbg=False):
    ...

Some of the globals stored in helpers.globals:

# all test emails have the same email_host
email_host = 'test_email_host.test'

# test user present in the database
my_test_user_email = 'mytestuser@mytestuser.mytestuser'
my_test_user_password = 'nio7FKCYUHTgd765edfglfsSDgn'
my_test_user_new_password = 'LUI76tlfvutjlFLJFgf'

Some of the helper functions in helper_utils.py:

def get_attr_from_html_page_by_elem_id(rv, elem_id, attr_name, form_prefix=None, dbg=False):
    ...

def get_unique_test_email():
    ...

def page_has_tag_with_text(rv, tag, text, dbg=False):
    ...

def client_get(client, url, follow_redirects=False, dbg=False):
    ...

def client_post(client, url, form_field2values=None, form_prefix=None, follow_redirects=False, dbg=False):
    ...

def my_test_user_restore_password(dbg=False):
    ...

def my_test_user_login(client, lang_code=None, email=None, password=None, check_message_logged_in=True, dbg=False):
    ...

def my_test_user_logout(client, lang_code=None, check_message_logged_out=True, dbg=False):
    ...

def get_auth_login_url(client=None, lang_code=None, dbg=False):
    ...

def get_auth_logout_url(client=None, lang_code=None, dbg=False):
    ...

I added a client_get() and client_post() function which behave almost identical to client.get() and client.post() but also return the location of page after redirects. The client_post() function also has an optional form_prefix parameter that can be used when the form is prefixed, see WTForms documentation.

The function my_test_user_login() can be used when not authenticated to log in the test user. All helper files have an optional dbg (debug) argument. When developing tests you often want to see what is happening, setting dbg=True gives me all details, also in the log file.

Test file names, function names and authentication

In Pytest you can use 'mark' to group tests but I did not use this yet. Instead I have two main categories. Tests when the user is not authenticated, an = not authenticated, and tests when the user is authenticated, ay = authenticated. Test files and tests start with the same string. Test file:

test_an_auth_login.py

has tests like:

def test_an_auth_login_enable_disable(client_lang_code_an):
    client, lang_code = client_lang_code_an
    ...

And the test file:

test_ay_account.py

has tests like:

def test_ay_account_change_password_to_new_password(client_lang_code_ay):
    client, lang_code = client_lang_code_ay
    ...

The parameters 'client_lang_code_an' and 'client_lang_code_ay' are Tuples that contain the (test) client and the language code.
They are unpacked in the test function. There is a package Pytest-cases that avoids the unpacking but I did not want to use this.

Starting tests with fixtures

I want the tests to run one-by-one for all available languages, or just one or more languages I specify on the command line. This means the fixture scope must be 'function'. Example of ouput from Pytest:

tests_frontend/test_ay_account.py::test_ay_account_change_password_logout_login[de] PASSED
tests_frontend/test_ay_account.py::test_ay_account_change_password_logout_login[en] PASSED
tests_frontend/test_ay_account.py::test_ay_account_change_password_logout_login[es] PASSED
tests_frontend/test_ay_account.py::test_ay_account_change_password_logout_login[fr] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_email_too_many_attempts[de] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_email_too_many_attempts[en] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_email_too_many_attempts[es] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_email_too_many_attempts[fr] PASSED

You might argue that running all tests for language X first, then running all tests for language Y, is faster. You are right, see also below 'changing the Pytest scope', but I wanted the option to keep thing together when running multiple new tests. Here is the main part of conftest.py:

from app_frontend import create_app


def pytest_addoption(parser):
    parser.addoption('--lc', action='append', default=[],
        help='lc (one or more possible)')


def pytest_generate_tests(metafunc):
    default_opts = ['de', 'en', 'es']
	lc_opts = metafunc.config.getoption('lc') or default_opts
    if 'client_lang_code_an' in metafunc.fixturenames:
        metafunc.parametrize('client_lang_code_an', lc_opts, indirect=True)
    if 'client_lang_code_ay' in metafunc.fixturenames:
        metafunc.parametrize('client_lang_code_ay', lc_opts, indirect=True)


@pytest.fixture(scope='function')
def client_lang_code_an(request):

    lang_code = request.param
    project_config = os.getenv('PROJECT_CONFIG')
    flask_app = create_app(project_config)

    with flask_app.test_client() as client:
        with flask_app.app_context():
            rv = client.get(get_language_root_url(lang_code))
            yield (client, lang_code)


@pytest.fixture(scope='function')
def client_lang_code_ay(request):

    lang_code = request.param
    project_config = os.getenv('PROJECT_CONFIG')
    flask_app = create_app(project_config)

    with flask_app.test_client() as client:
        with flask_app.app_context():
            rv = client.get(get_language_root_url(lang_code))
            my_test_user_restore_password()
            my_test_user_login(client, lang_code, change_to_language=True)
            yield (client, lang_code)
            my_test_user_logout(client)

Not really much special here. Tests that must run with an authenticated user have the 'client_lang_code_ay' parameter. This refers to the fixture with this name. In this case, the client is logged in before the test is called.

If I do not specify the '--lc' on the command line all languages are used for a test. I can select one or more languages by specifying e.g. '--lc de' on the command line. In fact later I also added some that allows me to type '--lc=fr,en,de'.

Database

Because this is a functional test of a CMS/Blog I need some data in the database. Before a test, I must cleanup some data to prevent interference with the test. I must write this cleanup code myself. This is extra work but not so bad because I can use parts of this code also in the application later. Like when removing a user.

Some tests

When not authenticated the situation is not very complex. Here is a test that checks the error message with a too short password.

def test_an_auth_login_password_too_short(client_lang_code_an):
    client, lang_code = client_lang_code_an
    """
    TEST: submit login_form with too short password
    EXPECT: proper error message
    """

    cleanup_db_auth()

    auth_login_url = get_auth_login_url()

    auth_login_info = AuthLoginInfo()
    login_form_prefix = auth_login_info.login_form_prefix

    input_validation = InputValidationAuth()
    length_min = input_validation.password.length_min
    error_message = check_text(lang_code, input_validation.password.error_message_length_min)

    url, rv = client_post(
        client,
        auth_login_url,
        form_field2values=get_login_form_form_field2values_for_submit(
            password='x' * (length_min - 1)
        ),
        form_prefix=login_form_prefix,
        follow_redirects=True
    )
    assert page_contains_text(rv, error_message)

Before the actual test some data is removed from the database, if present, to avoid interference. I get the form prefix, minimum length and error message by pulling in some classes.

It becomes more complex when authenticated. This has to do with the fact that a registered user can set the preferred language in the account. For example, the user with German (Deutsch) as the preferred language, is on the English version of the site, and then logs in. After login, the site language is switched to German (Deutsch) automatically.

Because I want to check messages in all languages, I added some code to the my_test_user_login() function to make my life more easy. If the language, lang_code, is specified, then after a login the user is moved to the specified language version.

def test_ay_auth_try_to_login_when_loggedin_get(client_lang_code_ay):
    client, lang_code = client_lang_code_ay
    """
    TEST: logged in test user, try to login
    EXPECT: proper error message
    """

    cleanup_db_auth()

    auth_login_url = get_auth_login_url()

    auth_login_info = AuthLoginInfo()
    error_message = check_text(lang_code, auth_login_info.error_message_you_are_already_logged_in)

    rv = client.get(auth_login_url, follow_redirects=True)
    assert page_contains_text(rv, error_message), 'not found, error_message = {}'.format(error_message)

Test time

To time a test you can simply put the time command in front of the Pytest command, for example:

time python3 -m pytest --cov=shared/blueprints/auth --cov-report term-missing  tests_frontend  -k "test_ay_auth_try_to_login_when_loggedin_get" -s -v

prints at the end to the terminal:

real	0m 16.67s
user	0m 9.65s
sys	0m 0.29s

This is the result for a single test with six languages, an average of 2.5 seconds per language.

A more easy way to see in detail the execution times of tests is by adding the --durations=N parameter to the command line. If N=0 you get the execution times for every test. With N=1 you get the execution time of the slowest test. With N=2 you get the execution time of the two slowest tests, etc. For example, to get the slowest 4 contact form tests the command is:

time python3 -m pytest --cov=shared/blueprints/auth --cov-report term-missing  tests_frontend  -k "test_an_contact" -s -vv --durations=4 --lc=fr

The result includes:

========== slowest 4 test durations ==========
14.88s call     test_an_contact_form.py::test_an_contact_form_multiple_same_ip_address[fr]
6.96s call     test_an_contact_form.py::test_an_contact_form_multiple_same_email[fr]
6.36s call     test_an_contact_form.py::test_an_contact_form_missing_form_field[fr]
2.87s call     test_an_contact_form.py::test_an_contact_form_thank_you_logged_in[fr]
========== 15 passed, 24 deselected, 1 warning in 51.92s ==========

Timings inside the fixture

To see where all this time is going I did a time test by collecting times at various stages in the authenticated fixture, client_lang_code_ay().

0.0 client_lang_code_ay
0.141031 after: flask_app = create_app(project_config)
0.141062 after: with flask_app.test_client() as client
0.141078 after: with flask_app.app_context()
0.448304 after: rv = client.get(get_language_root_url(lang_code))
0.531201 after: my_test_user_restore_password()
1.154249 after: my_test_user_login(client, lang_code)
2.553381 after: yield (client, lang_code)
2.624607 after: my_test_user_logout(client, lang_code)

The time to login is longer then you may expect, this is caused by a deliberate delay on login. Tests with the not authenticated fixture, client_lang_code_an(), take an average of 1.5 seconds, which is expected without log in.

Reducing test time by changing the Pytest scope

Because the Pytest fixture scope is 'function', we get a new test_client for every test. I used the scope='function' to see tests executed for all languages. The output of Pytest:

test1[de] PASSED
test1[en] PASSED
test1[es] PASSED
test1[fr] PASSED
test2[de] PASSED
test2[en] PASSED
test2[es] PASSED
test2[fr] PASSED
test3[de] PASSED
test3[en] PASSED
test3[es] PASSED
test3[fr] PASSED

This is very nice but also takes a lot of time.

To reduce the test time I change the fixture scope to 'session'. In this case a new test_client will be created only for every language. First all tests are executed for language 'de', then all tests are executed for language 'en', etc.. The output of Pytest:

test1[de] PASSED
test2[de] PASSED
test3[de] PASSED
test1[en] PASSED
test2[en] PASSED
test3[en] PASSED
test1[es] PASSED
test2[es] PASSED
test3[es] PASSED
test1[fr] PASSED
test2[fr] PASSED
test3[fr] PASSED

The test time is dramatically reduced this way. But only if you select a lot of, all, tests of course. In general most tests will complete very fast and only a few are responsible for 80% of the test time.

Test time for 14 (contact_form) tests using the scope='function' was 00:06:11. After changing the scope to 'session' the time was 00:04:56. In this case the reduction in test time is 20%. In another case the reduction was more than 50%.

Design for test

The more data we can pull in from our application, the more we can automate testing. For example, what is the prefix of the login form? What are the error messages when trying to log in when already logged in? What are the minimum and maximum number of characters for a password? This data is present in the application, how do we get it out?

For a number of classes I added an 'Info' class that is the parent of the actual class.

class AuthLoginInfo:

    def __init__(self):

        self.error_message_you_are_already_logged_in = _('You are already logged in.')
        self.error_message_login_is_temporary_suspended = _('Log in is temporary suspended.')
        ...
        self.message_you_are_logged_in = _('You are logged in.')

        self.login_form_prefix = self.login_form_id


class AuthLogin(AuthLoginInfo):

    def __init__(self):
        super().__init__()

    def login(self):

        if already_logged_in():
            flash(self.error_message_you_are_already_logged_in, 'error')
            return redirect(already_logged_in_redirect_url())
        ...

I still do not know if this can be done better, but it is workable.

Automating language checking

In the above tests we do not know if the messages are actually correct and translated for every language. We must manually check the texts but can try to minimize our work.

I created a function check_text() that does three things:

• Store the text in a dictionary
• Check if the same text is already in a dictionary
• Store the text in a language log file

The names of these language log files are the test names. A test is run for different languages. If the text is already in the dictionary then probably something is wrong, like the text is not translated. In this case I call 'assert' in the function check_text(). This shows an error so we can investigate what is wrong.

If all is well I check the language log file to look at the texts. The tab-separated fields of lines in this file are test_name, text_id, lang_code, text:

test_ay_auth_try_to_login_when_loggedin_get	default	de	Sie sind bereits eingeloggt.
test_ay_auth_try_to_login_when_loggedin_get	default	en	You are already logged in.
test_ay_auth_try_to_login_when_loggedin_get	default	es	Ya está conectado.
test_ay_auth_try_to_login_when_loggedin_get	default	fr	Vous êtes déjà connecté.
test_ay_auth_try_to_login_when_loggedin_get	default	nl	Je bent al ingelogd.
test_ay_auth_try_to_login_when_loggedin_get	default	ru	Вы уже вошли в систему.

I do not know Russian and only a little bit Spanish and French, but I think this looks fine.

More about the check_text() function

I can specify a text_id when I want to check more then one text in a test function. If we do not specify this then 'default' is used.
I also added a flag for texts we do not want to check (automatically). The name of the test, the caller, is extracted from the stack. The check_text() function first lines:

def check_text(lang_code, text, text_id=None, do_not_check=False, dbg=False):
    test_name = inspect.stack()[1][3]
    ...

We have done a lot but we still must manually check texts every time after a test run. This is still far too much work, we have verified the texts, translations, in the language log file and do not want to do this again!

To avoid this I created a directory 'language_logs_ok' and copy the language log files that we approved here. But still too much work, we do not want to copy files manually!

With some extra code we can check at the end of a test if this ok_file exists. If it does not exist, we can ask to approve the texts and if answered Yes we copy the language log file also to the directory 'language_logs_ok'. If it exists we can compare the texts of the current test to the approved texts and signal an error if they they do not match.

Pytest has hooks that can be used to add functionality before the start of the tests and after the tests finished. Here I only use the latter one:

def pytest_sessionfinish(session, exitstatus):
    ....

I added the code and now the result is:

test_an_auth_login_password_too_long: NO ok file yet WARNING
texts:
test_an_auth_login_password_too_long	default	de	Das Passwort muss zwischen 6 und 40 Zeichen lang sein.
test_an_auth_login_password_too_long	default	en	Password must be between 6 and 40 characters long.
test_an_auth_login_password_too_long	default	es	La contraseña debe tener entre 6 y 40 caracteres.
test_an_auth_login_password_too_long	default	fr	Le mot de passe doit comporter entre 6 et 40 caractères.
test_an_auth_login_password_too_long	default	nl	Het wachtwoord moet tussen 6 en 40 tekens lang zijn.
test_an_auth_login_password_too_long	default	ru	Пароль должен иметь длину от 6 до 40 символов.
Do you approve these texts (create an ok file)? [yN]

If I approve these texts, the ok_file is created. Then on the next run, after the Pytest test results the following message is shown:

test_an_auth_login_password_too_long: language match with ok_file PASSED

This not only takes away a lot of stress but also a lot of time!

Another test with multiple texts

I am using the WTForms package. I can pull the form into the test, extract the form field labels and check that they are on the page. By automating form field label checks we eliminate missed translations, or wrong translations.

The login form:

class AuthLoginForm(FlaskForm):

    email = StringField(
        _l('Email'),
        filters=[strip_whitespace],
        validators=[InputRequired()])

    password = PasswordField(
        _l('Password'),
        render_kw={'autocomplete': 'off'},
        filters=[strip_whitespace],
        validators=[InputRequired()])

    remember = BooleanField(_l('Remember me'))

    submit = SubmitField(_l('Log in'))

The test:

def test_an_auth_login_form_labels(client_lang_code_an):
    client, lang_code = client_lang_code_an
    """
    TEST: check login form labels
    EXPECT: proper error message
    """
    fname = get_fname()

    cleanup_db_auth()

    auth_login_url = get_auth_login_url()

    auth_login_info = AuthLoginInfo()
    auth_login_form_prefix = auth_login_info.login_form_prefix

    auth_login_form = AuthLoginForm(prefix=auth_login_form_prefix)

    # select page
    location, rv = client_get(client, auth_login_url, follow_redirects=True)

    assert page_contains_form_field_label(rv, 'email', auth_login_form, lang_code, form_prefix=auth_login_form_prefix)
    assert page_contains_form_field_label(rv, 'password', auth_login_form, lang_code, form_prefix=auth_login_form_prefix)
    assert page_contains_form_field_label(rv, 'remember', auth_login_form, lang_code, form_prefix=auth_login_form_prefix)
    assert page_contains_form_field_label(rv, 'submit', auth_login_form, lang_code, form_prefix=auth_login_form_prefix)

And the result after a succesful test:

tests_frontend/test_an_auth_login.py::test_an_auth_login_form_labels[de] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_form_labels[en] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_form_labels[es] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_form_labels[fr] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_form_labels[nl] PASSED
tests_frontend/test_an_auth_login.py::test_an_auth_login_form_labels[ru] PASSED
test_an_auth_login_form_labels: language NO ok file yet WARNING
texts:
test_an_auth_login_form_labels	email	de	E-Mail
test_an_auth_login_form_labels	email	en	Email
test_an_auth_login_form_labels	email	es	Correo electrónico
test_an_auth_login_form_labels	email	fr	Courriel
test_an_auth_login_form_labels	email	nl	E-mail
test_an_auth_login_form_labels	email	ru	Электронная почта
test_an_auth_login_form_labels	password	de	Passwort
test_an_auth_login_form_labels	password	en	Password
test_an_auth_login_form_labels	password	es	Contraseña
test_an_auth_login_form_labels	password	fr	Mot de passe
test_an_auth_login_form_labels	password	nl	Wachtwoord
test_an_auth_login_form_labels	password	ru	Пароль
test_an_auth_login_form_labels	remember	de	Erinnern Sie sich an mich
test_an_auth_login_form_labels	remember	en	Remember me
test_an_auth_login_form_labels	remember	es	Recuérdame
test_an_auth_login_form_labels	remember	fr	Se souvenir de moi
test_an_auth_login_form_labels	remember	nl	Herinner mij
test_an_auth_login_form_labels	remember	ru	Помните меня
test_an_auth_login_form_labels	submit	de	Einloggen
test_an_auth_login_form_labels	submit	en	Log in
test_an_auth_login_form_labels	submit	es	Iniciar sesión
test_an_auth_login_form_labels	submit	fr	Ouvrir une session
test_an_auth_login_form_labels	submit	nl	Aanmelden
test_an_auth_login_form_labels	submit	ru	Войти
Do you approve these texts (create an ok file)? [yN] 

Test coverage with Pytest-cov

I also installed Pytest-cov to get details about what amount of the code was used during a test. Again, this is very worthwile when testing a library, package. You get the line numbers of the code that was not used in the tests. You should get 100% coverage!

When doing functional tests, it is initially very difficult to reach this 100% because we are testing parts of the application. For example, if we have an input validation class, when starting with our functional tests, we may skip many input validation methods.

Sometimes, I add extra code to re-check if a value is correct. This is because I want to make sure I did not make a mistake somewhere, and also sometimes I do not trust a third-party package I am using. This code remains untested. I need to add hooks to make this code testable ... one day.

Parallel testing

When I can run tests parallel, the batch finishes faster. I already talked about deliberate delays that are inserted on log in. What a waste of test time. But I cannot just run all test at the same time. Some tests modify the test_user, others may interfere with email sending limitations, etc. I must carefully select which tests are indepent from each other. The Pytest-xdist package can help. I will look into this one of these days.

Problems

When starting with my first Pytest fixture running the Flask test_client I got the following error: 

AssertionError: a localeselector function is already registered

Flask-Babel with Pytest gives AssertionError: a localeselector function is already registered. With the first parameter ('de') no problems but with the next ('en', 'es', ...) the above error message is shown. This happens when a new test_client is created.

My workaround, which appears to work fine is using a global variable that indicates if the localeselector was decorated already. If not, I decorate it myself.

    # removed @babel.localeselector
    def get_locale():
        do something

    global babel_localeselector_already_registered
    if not babel_localeselector_already_registered:
        get_locale = babel.localeselector(get_locale)
        babel_localeselector_already_registered = True

Summary

This is my first implementation of mainly functional tests for this multilanguage Flask CMS/Blog website. It took quite some time to setup and write helper functions. But these are essential to writing compact tests. And I am certain I will have to write new helper functions for new test categories.

I do not want to hardcode texts and parameters into the tests. Texts and parameters may change during the lifetime of a product. I changed some code and added 'Info' classes so I can pull in the texts and parameters I want to test.

Tests with texts require interaction and must be manually approved. After approval I can run the same tests and the texts will be checked automatically against the approved texts. If texts change, an ERROR is presented and we will be asked again to approve these changes.

I also implemented some 30+ tests with Selenium. But I decided to focus on the Pytest + Flask first.

There is still much to be desired but first let me write some more code and tests ...

Links / credits

Create and import helper functions in tests without creating packages in test directory using py.test
https://stackoverflow.com/questions/33508060/create-and-import-helper-functions-in-tests-without-creating-packages-in-test-di

how to testing flask-login? #40
https://github.com/pytest-dev/pytest-flask/issues/40

Pytest - How to override fixture parameter list from command line?
https://stackoverflow.com/questions/51992562/pytest-how-to-override-fixture-parameter-list-from-command-line

pytest cheat sheet
https://gist.github.com/kwmiebach/3fd49612ef7a52b5ce3a

Python Pytest assertion error reporting
https://code-maven.com/pytest-assert-error-reporting

Testing Flask Applications
https://flask.palletsprojects.com/en/1.1.x/testing/