wireshark/doc/wsdg_src/wsdg_tests.adoc

// WSDG Chapter Tests

[#ChapterTests]
== Wireshark Tests

The Wireshark sources include a collection of Python scripts that test
the features of Wireshark, TShark, Dumpcap, and other programs that
accompany Wireshark. These are located in the `test` directory of the
Wireshark source tree.

The command line options of Wireshark and its companion command line
tools are numerous. These tests help to ensure that we don't introduce
bugs as Wireshark grows and evolves.

[#TestsQuickStart]
=== Quick Start

The recommended steps to prepare for and run tests with a UN*X toolchain:

* Install two Python packages, pytest: `pip install pytest pytest-xdist`
* Build programs (“wireshark”, “tshark”, etc.): `ninja`
* Build additional programs for the “unittests” suite: `ninja test-programs`
* Run tests in the build directory: `pytest`

Replace `ninja` by `make` as needed.

If building with <<#ChWindowsBuild,Microsoft Visual Studio>> the analogous steps are:

* Install pytest Python packages: `python -m pip install pytest pytest-xdist`
* Build programs: `msbuild /m /p:Configuration=RelWithDebInfo Wireshark.sln`
* Build test-programs: `msbuild /m /p:Configuration=RelWithDebInfo test-programs.vcxproj`
* Run tests: `python -m pytest`

TIP: Depending on your PATH, you may need to run the pytest module as a
script from your Python interpreter, e.g, `python -m pytest` or
`python3 -m pytest` instead of `pytest`.

The test suite will attempt to test as much as possible and skip tests
when its dependencies are not satisfied. For example, packet capture
tests require a Loopback interface and capture privileges. To avoid
capture tests, pass the `--disable-capture` option.

List available tests with `pytest --collectonly`. Enable verbose output
with `pytest --verbose`. For more details, see <<ChTestsRun>>.

You can also run the "ninja test" target instead of invoking pytest
directly. This will automatically build the test programs dependency,
so it may be preferred for that reason.

[#ChTestsStructure]
=== Test suite structure

The following sections describes how the test suite is organized.

[#TestCoverage]
==== Test Coverage And Availability

The testing framework can run programs and check their stdout, stderr,
and exit codes. It cannot interact with the Wireshark UI. Tests cover
capture, command line options, decryption, file format support and
conversion, Lua scripting, and other functionality.

Available tests depend on the libraries with which Wireshark was built.
For example, some decryption tests depend on a minimum version of
Libgcrypt and Lua tests depend on Lua.

Capture tests depend on the permissions of the user running the test
script. We assume that the test user has capture permissions on Windows
and macOS and capture tests are enabled by default on those platforms.

TIP: Build the "test-capture" target on Linux (using sudo) to set dumpcap
permissions and enable capture tests.

If a feature is unavailable, the test will be skipped. For example, if
an old version of Libgcrypt is in use, then some decryption tests will
be skipped while other tests can still run to completion.

[#TestsLayout]
==== Suites, Cases, and Tests

The test suite uses pytest as a test runner. Tests are organized according to
suites, cases, and individual tests. Suites correspond to Python modules
that match the pattern “suite_*.py”. Cases correspond to one or more
classes in each module, and case class methods matching the pattern
”test_*” correspond to individual tests. For example, the invalid
capture filter test in the TShark capture command line options test case
in the command line options suite has the ID
“suite_clopts.py::TestTsharkCaptureClopts::test_tshark_invalid_capfilter”.

[#TestsPytest]
==== pytest fixtures

A test has typically additional dependencies, like the path to an
executable, the path to a capture file, a configuration directory, the
availability of an optional library, and so on.

https://pytest.org/[pytest] is a test framework which has full
parallelization support (test-level instead of just suite-level),
provides nice test reports, and allows
https://docs.pytest.org/en/latest/fixture.html[modular fixtures].

A fixture is a function decorated with `@pytest.fixture` and can
either call `pytest.skip("reason")` to skip tests that depend on the
fixture, or return/yield a value.
Test functions (and other fixture functions) can receive the fixture
value by using the name of the fixture function as function parameters.
Common fixtures are available in `fixtures_ws.py` and includes
`cmd_tshark` for the path to the `tshark` executable and `capture_file`
for a factory function that produces the path to a capture file.

[#ChTestsRun]
=== Listing And Running Tests

Tests are run with https://pytest.org/[pytest]. Pytest features versus the
"unittest" standard library module include finer
test selection, full parallelism, nicer test execution summaries, better output
in case of failures (containing the contents of variables) and the ability to
open the PDB debugger on failing tests.

To get started, install pytest 3.0 or newer and
https://pypi.org/project/pytest-xdist/[pytest-xdist]:

[source,sh]
----
# Install required packages on Ubuntu 18.04 or Debian jessie-backports
$ sudo apt install python3-pytest python3-pytest-xdist

# Install required packages on other systems
$ pip install pytest pytest-xdist
----

Run `pytest` in the Wireshark build directory, Wireshark binaries are assumed to
be present in the `run` subdirectory (or `run\RelWithDebInfo` on Windows).

[source,sh]
----
# Run all tests
$ cd /path/to/wireshark/build
$ pytest

# Run all except capture tests
$ pytest --disable-capture

# Run all tests with "decryption" in its name
$ pytest -k decryption

# Run all tests with an explicit path to the Wireshark executables
$ pytest --program-path /path/to/wireshark/build/run
----

To list tests without actually executing them, use the `--collect-only` option:

[source,sh]
----
# List all tests
$ pytest --collect-only

# List only tests containing both "dfilter" and "tvb"
$ pytest --collect-only -k "dfilter and tvb"
----

The test suite will fail tests when programs are missing. When only a
subset of programs are built or when some programs are disabled, then
the test suite can be instructed to skip instead of fail tests:

[source,sh]
----
# Run tests when libpcap support is disabled (-DENABLE_PCAP=OFF)
$ pytest --skip-missing-programs dumpcap,rawshark

# Run tests and ignore all tests with missing program dependencies
$ pytest --skip-missing-programs all
----

To open a Python debugger (PDB) on failing tests, use the `--pdb` option and
disable parallelism with the `-n0` option:

[source,sh]
----
# Run decryption tests sequentially and open a debugger on failing tests
$ pytest -n0 --pdb -k decryption
----

[#ChTestsDevelop]
=== Adding Or Modifying Tests

Tests must be in a Python module whose name matches “suite_*.py”. The
module must contain one or more subclasses with a name starting with
"Test" something, for example "class TestDissectionHttp2:". Each test case
method whose name starts with “test_” constitutes an individual test.

Success or failure conditions are signalled using regular assertions
with the "assert" Python keyword.

Test dependencies (such as programs, directories, or the environment
variables) are injected through method parameters. Commonly used
fixtures include `cmd_tshark` and `capture_file`.

Processes (tshark, capinfos, etc.) are run using the "subprocess" Python module,
or the Wireshark `subprocesstest` module with some convenience functions.
Possible functions include `subprocesstest.run()`, `subprocesstest.check_run()`
or creating `subprocess.Popen` object if the utility functions are not sufficient for some reason.
Usually this is only required if two-way communication is performed with
the child process. `subprocesstest.check_run()` is exactly the same as
calling `subprocesstest.run()` with `check=True` as an argument, only
a bit more expressive.

Check the documentation for the Python subprocess module for a full description
of the arguments available to the `subprocesstest.run()` convenience wrapper
and the `subprocess.Popen` object.

All of the current tests run one or more of Wireshark's suite of
executables and either check their return code or their output. A
simple example is “suite_clopts.py::TestBasicClopts::test_existing_file”,
which reads a capture file using TShark and checks its exit code.

[source,python]
----
import subprocesstest
import pytest

class TestBasicClopts:
    def test_existing_file(self, cmd_tshark, capture_file, test_env):
        subprocess.check_run((cmd_tshark, '-r', capture_file('dhcp.pcap')), env=test_env)
----

Output can be checked using `assert subprocesstest.grep_output()`,
`assert subprocesstest.count_output()` or any other `assert` statement.
`subprocesstest.check_run()` also asserts that the child process returns
the value 0 as exit code.

[source,python]
----
import subprocesstest
import pytest

class TestDecrypt80211:
    def test_80211_wpa_psk(self, cmd_tshark, capture_file, test_env):
        tshark_proc = subprocesstest.run((cmd_tshark,
                '-o', 'wlan.enable_decryption: TRUE',
                '-Tfields',
                '-e', 'http.request.uri',
                '-r', capture_file('wpa-Induction.pcap.gz'),
                '-Y', 'http',
            ), capture_output=True, env=test_env)
        assert 'favicon.ico' in tshark_proc.stdout
----

Tests can be run in parallel. This means that any files you create must
be unique for each test. Filenames based on the current test name are
generated using fixtures such as "capture_file" and "result_file". By default
pytest generates paths in the system's temporary directory and the last three
pytest runs are kept. Temporary files from older runs are automatically deleted.

[#ChTestsExternal]
=== External Tests

You can create your own Python test files outside of the Wireshark source tree.
To include your tests when running the Wireshark test suite, simply add the
directory containing your test files to the `pytest` command line. Note that
filenames must match the same conventions as discussed above.

In order for your tests to have access to the Wireshark test fixtures, you will
need this line in each test file:

[source,python]
----
from fixtures_ws import *
----

[#ChTestsExtFixtures]
==== Custom Fixtures

You may wish to define your own test fixtures -- for example, a fixture similar
to `capture_file` but which gives the path to a file in your external test
directory. Here is an example Python file containing such a fixture. It presumes
a subdirectory named `extra_captures` which exists in the same directory, and
which contains your extra capture files.

[source,python]
----
# my_fixtures.py
# To use in your own tests, import like so:
#   from my_fixtures import *

from pathlib import Path
import pytest

@pytest.fixture(scope='session')
def extra_file():
    def resolver(filename):
        return Path(__file__).parent.joinpath("extra_captures", filename)
    return resolver
----

NOTE: If you give your fixture the same name as an existing Wireshark fixture,
any tests using your fixture library will lose access to the Wireshark fixture
of the same name. This can lead to confusing behavior and is not recommended.