7.1. Testing

7.1.1. Overview

Extensive testing is done on the functionality, documentation and website of optrace with tox and pytest. The tox configuration can be found in Section Section 7.1.12. Test can be run locally or through GitHub actions. The workflows are located in .github/workflows/, while GitHub Actions shows the last runs.

7.1.2. Functionality Testing

The goal of the functionality testing is to check typical features as well as edge cases. Not only should the code “run”, but also produce correct results. This is enforced by many automated test cases.

Some examples include:

  • a nearly ideal lens produces a distinct focal point at the correct focal distance

  • a growing number of rays decreases sampling noise

  • user-built surfaces modeling a spherical surface behave the same as the in-built type

  • sampling of directions, color, positions produces the correct distribution on both the source and destination

  • GUI interactions execute the expected actions

  • images are correctly loaded and saved

  • typical phenomena and aberrations of geometrical optics can be reproduced (spherical and chromatic aberration, vignetting, dispersion, astigmatism, coma, …)

  • image formation by PSF convolution and raytracing produce comparable results (in cases where both are applicable)

  • simple .zmx and .agf files are handled and imported correctly

  • … many more

Some edge cases include:

  • normalizing a black image does not lead to divisions by zero

  • requesting billions of rays prints an error message instead of crashing the machine

  • collisions of lenses are detected

  • moving geometry elements outside of the setup bounding box is not possible

  • check of value validity (no negative powers, wavelengths outside the visible range, …)

  • plotting/loading/tracing of a single pixel image

  • an optical setup without lenses, filters or apertures is “traced” correctly

  • … many more

Test files can be found under tests/, starting with filename test_ Testing is performed with pytest. To ensure that it is performed on a “clean” and defined environment, it is run by tox, which creates a virtual python environment where the library is installed and tested. With its default call, tox calls the testing environment that includes all test cases. To lower RAM usage, tox calls test groups instead of all tests at once.

The workflow file tox_test.yml runs the functionality tests on a defined system and python version as GitHub action. By default, this is done on pushed code changes, but the action can also be run manually.

7.1.3. Coverage Testing

coverage checks if all code lines and branches were executed while testing. It wraps around pytest inside the tox test environment and reports on executed and missing lines. The output gets represented as html or as text output (inside the terminal or GitHub action output).

7.1.4. Benchmark Testing

Tracing performance mainly depends on the number of surfaces and rays as well as the complexity of the surfaces. Spherical surfaces are easier to trace then complex user defined ones. For systems with a low number of surfaces the ray generation step at the source becomes dominant.

The benchmarking file tests/benchmark.py is an adapted version of the Microscope example. It includes 57 light-interacting surfaces (one ray source, one aperture, 27 lenses with front and back surface each, one outline surface of the bounding box). Detectors are excluded as they don’t interact with the rays unless image rendering is invoked. Turning off the polarization calculation (with Raytracer.no_pol) leads to a significant speedup. The number of used threads is controlled by setting an environment variable as described in section Section 4.1.4.

The raytracing on my system (Arch Linux 6.15.8, i7-1360P notebook CPU, 16GB RAM, Python 3.13) results in:

Table 7.1 Performance Comparison. Result values are in seconds / surface / million rays.

Cores

Without Polarization

With Polarization

1

0.148

0.218

2

0.082

0.115

4

0.053

0.085

8

0.045

0.082

12

0.043

0.078

16

0.046

0.073

Performance plateaus at 8-16 cores for the “without polarization” case, but still seems to improve for the second case. Possible reasons are thermal throttling or other bottlenecks. On my machine, 8 cores is a suitable compromise between CPU usage and performance.

7.1.5. Documentation Testing

The workflow file doc_test.yml runs multiple test environments. This includes:

  • tox -e docsbuildcheck: Testing of correct documentation building while not permitting warnings

  • tox -e linkcheck: Testing of all documentation links (currently deactivated, due to GitHub rate limits)

  • tox -e doctest: Testing of most documentation code examples (using doctest)

7.1.6. Python Version Testing

Testing of the compatibility with multiple python versions is performed with the pyver_comp.yml workflow. It executes a subset of tests (tox -e fast) for multiple python main versions in an Ubuntu runner.

7.1.7. Installation Testing

Weekly tests that try to install the package and execute some quick tests. Ensures that requirements and available PyPI packages are up-to-date and no changes are needed. The workflow is located in install_test.yml.

7.1.8. Platform Testing

Platform testing is performed with the os_comp.yml workflow. It executes the os tox environment (run with tox -e os), which runs a subset of all available tests. These tests include relevant, possibly platform-dependent cases, including:

  • installation

  • loading and saving of files

  • handling of filenames and paths

  • loading sensitivity and image presets (which also are files inside the library)

  • opening the GUI and plots

  • enforcing a specific float bit width

  • detecting the number of cores and multithreading

The workflow is executed with the current python version and both the latest macOS and Windows runners. Linux is not included, as all other workflows already run on Ubuntu.

7.1.9. Update Action Dependencies

The action dependencies inside the GitHub Actions are kept up-to-date with the Dependabot. It runs weekly and an pull request is opened if there are newer versions of an action available.

7.1.10. Website tests

Documentation website tests are done with the website_test.yml <https://github.com/drocheam/optrace/blob/main/.github/workflows/website_test.yml> workflow that runs the tests/test_website.sh and tests/test_connections.sh bash scripts. The test checks that the site is reachable, includes most important parts (index, impressum, robots, sitemap), does not set cookies and does not load any external resources. The two last points are required for the best privacy and GDPR compliance. Runs are triggered either manually, weekly or after the GitHub pages deployment.

7.1.11. Manual Tests

Unfortunately, not all tests can be automated. The following test cases need to be handled manually:

Checking that optrace.plots.block actually pauses the program execution

This can’t be tested automatically, as it would halt testing. (Maybe there is a way by using multithreading or multiprocessing and a timer?)

Correct and visually pleasing formatting inside plots and the GUI

Can only be tested by a human viewer.

Usage in Python Notebooks or inline IDEs such as Spyder

  • Spyder Installation:
  • Testing:

    • make sure pyplot windows are displayed correctly (plot viewer) with enough dpi in the interface

    • make sure this is also the case for plots generated by the GUI

7.1.12. Tox configuration in tox.toml

 1env_list = ["py{311,312,313,314}"]
 2requires = ["tox-ignore-env-name-mismatch"]
 3skip_missing_interpreters = true
 4
 5[env_run_base]
 6description = "Main testing environment"
 7pass_env = ["HOME", "DISPLAY", "WAYLAND_DISPLAY", "QT_QPA_PLATFORM", "XAUTHORITY", "GITHUB_ACTIONS", "PYTHON_CPU_COUNT",
 8            "XDG_SESSION_TYPE", "XDG_SESSION_DESKTOP"]
 9env_dir = "{toxworkdir}{/}env_static"
10runner = "ignore_env_name_mismatch"
11deps = ["pylint"]
12extras = ["tests", "docs"]
13commands = [
14    ["coverage", "run", "-m", 
15        "pytest", {replace="posargs", extend=true}, "tests{/}test_geometry.py", "tests{/}test_misc.py", 
16        "tests{/}test_image.py", "tests{/}test_refraction_index.py", "tests{/}test_scope.py", 
17        "tests{/}test_spectrum.py", "tests{/}test_surface.py", "tests{/}test_tma.py"
18    ],
19    ["coverage", "run", "-m", 
20        "pytest", {replace="posargs", extend=true}, "tests{/}test_tracer.py", "tests{/}test_tracer_hurb.py", 
21        "tests{/}test_tracer_special.py"
22    ],
23    ["coverage", "run", "-m", 
24        "pytest", {replace="posargs", extend=true}, "tests{/}test_color.py", "tests{/}test_convolve.py", 
25        "tests{/}test_load.py"
26    ],
27    ["coverage", "run", "-m", 
28        "pytest", {replace="posargs", extend=true}, "tests{/}test_plots.py"
29    ],
30    ["coverage", "run", "-m", 
31        "pytest", {replace="posargs", extend=true}, "tests{/}test_examples.py"
32    ],
33    ["coverage", "run", "-m",
34        "pytest", {replace="posargs", extend=true}, "-m", "gui1", "tests{/}test_gui.py"
35    ],
36    ["coverage", "run", "-m", 
37        "pytest", {replace="posargs", extend=true}, "-m", "gui2", "tests{/}test_gui.py"
38    ],
39    ["coverage", "run", "-m", 
40        "pytest", {replace="posargs", extend=true}, "-m", "gui3", "tests{/}test_gui.py"
41    ],
42    ["coverage", "combine", "--quiet"],
43    ["coverage", "report", "--show-missing"],
44    ["coverage", "erase"]
45]
46
47[env.os]
48description = "Testing of system dependent functionality"
49commands = [["pytest", "-m", "os", {replace="posargs", extend=true}]]
50
51[env.fast]
52description = "Don't test slower tests (> 10 seconds)"
53commands = [
54    ["pytest", "-m", "not slow", "--ignore=tests{/}test_examples.py", "--ignore=tests{/}test_gui.py", 
55        {replace="posargs", extend=true}],
56    ["pytest", "-m", "not slow", {replace="posargs", extend=true}, "tests{/}test_examples.py"],
57    ["pytest", "-m", "not slow", {replace="posargs", extend=true}, "tests{/}test_gui.py"],
58]
59
60[env.install]
61description = "Quick testing of installation"
62commands = [["pytest", "-m", "install", {replace="posargs", extend=true}]]
63
64[env.linkcheck]
65description = "Check all links in software and documentation"
66commands = [["sphinx-build", "-M", "linkcheck", "docs{/}source", "docs{/}build", "-j", "auto"]]
67
68[env.docsbuildcheck]
69description = "Nitpicky documentation building, turn warnings into errors"
70commands = [["sphinx-build", "-M", "html", "docs{/}source", "docs{/}build{/}htmlcheck", "-n", "-W"]]
71
72[env.doctest]
73description = "Test documentation doctest snippets"
74commands = [["sphinx-build", "-M", "doctest", "docs{/}source", "docs{/}build", "-j", "auto"]]
75
76[env.docs]
77description = "Create documentation and update structure and changelog"
78allowlist_externals = ["{/}usr{/}bin{/}bash"]
79commands = [
80    ["bash", "docs{/}generate_changelog.sh"],
81    ["bash", "docs{/}generate_bib_structure.sh"],
82    ["sphinx-build", "-M", "html", "docs{/}source", "docs{/}build", "-n"]
83]
84
85[env.pylint]
86description = "Run pylint with excluded unnecessary rules"
87skip_install = true
88commands = [
89    ["pylint", "optrace", "-j", "0", "--max-line-length", "120", "--variable-naming-style", "any",
90    "--disable", "W0612,W0613,W0106,W0212,E1101,R0902,R0904,R0911,R0912,R0913,R0914,R0915,R0916,C0301,C0302",
91    "--good-names-rgxs", "^[_a-z][_a-z0-9]?$"]
92]

The pytest configuration is located in the pyproject.toml in Section Section 7.3.2.

Notes

  • parallelized testing with pytest-xdist could be possible, but we wouldn’t gain much from it, as the heaviest tasks are already multithreaded

  • pytest-xvfb uses xvfb as headless-display. Use the option --no-xvfb to actually see the plots/windows.

  • using pytest-timeout for adding timeouts

  • using pytest-qt to handle exceptions in slots and virtual methods

  • using pytest-random-order for running tests in a random order

  • tox-ignore-env-name-mismatch is required so multiple tox environments are able to use the same Python virtualenv.

  • when GUI tests fail on wayland, first run xhost +

  • some tests are excluded in GitHub actions, as there issues with the headless displays