7.1. Testing

7.1.1. Functionality Testing

The goal of the functionality testing is to check both typical features as well as edge cases. Not only should the code “run”, but also produce correct results. This is guaranteed by many automated test cases.

Some examples include:

• a nearly ideal lens produces a distinct focal point at the correct focal distance

• a growing number of rays decreases sampling noise

• user-built surfaces modeling a spherical surface behaving the same as the in-built type

• sampling of directions, color, positions produces the correct distribution on both the source and destination

• GUI interactions execute the expected actions

• images are correctly loaded and saved

• typical phenomena and aberrations of geometrical optics can be reproduced (spherical and chromatic aberration, vignetting, dispersion, astigmatism, coma, …)

• image formation by convolving produces correct results even the images must be scaled or interpolated

• PSF convolution and raytracing produce comparable results

• simple .zmx and .agf files are handled and imported correctly

• … many more

Some edge cases include:

• normalizing a black image does not lead to divisions by zero

• requesting billions of rays prints an error message instead of crashing the PC due to too much requested RAM

• collisions of lenses are detected

• moving geometry elements outside of the setup bounding box is not possible

• check of value validity (no negative powers, wavelengths outside the visible range, …)

• plotting/loading/tracing of a single pixel image

• an optical setup without lenses, filters or apertures is traced correctly

• … many more

Test files can be found under tests/, starting with filename test_ Testing is performed with pytest. To ensure that it is performed on a “clean” and defined environment, it is run by tox, this creates a virtual python environment where the library is installed and tested. With its default call, tox calls the testing environment that includes all test cases. To lower RAM usage, tox calls test groups instead of all tests at once.

The workflow file tox_test.yml runs the functionality tests on a defined system and python version as github actions. By default this is done on pushed code changes, but the action can also be run manually.

7.1.2. Coverage Testing

coverage is used to check if all code lines and branches were tested. It wraps around pytest inside the tox test environment and reports on executed and missing lines. The output gets represented as both html or as text output (inside the terminal or github action output).

7.1.3. Benchmark Testing

Tracing performance depends on the number of surfaces and rays as well as the complexity of the surfaces. Spherical surfaces are easier to trace then complex user defined ones. For systems with a low number of surfaces the ray generation at the source becomes dominant.

The benchmarking file tests/benchmark.py is an adapted version of the Microscope example. It includes 57 light-interacting surfaces (one ray source, one aperture, 27 lenses with front and back surface each, one outline surface of the bounding box). Detectors are excluded as they don’t interact with the rays unless image rendering is executed. Turning off the polarization calculation (with Raytracer.no_pol) leads to a significant speedup.

The raytracing on my system (Arch Linux 6.13, i7-1360P notebook CPU, 16GB RAM, Python 3.13) results in:

Table 7.1 Performance Comparison. Result values are in seconds / surface / million rays.

Cores	Without Polarization	With Polarization
1	0.262	0.417
2	0.122	0.171
4	0.069	0.125
8	0.059	0.108
16	0.057	0.101

So there is not much gain in using 16 over 8 cores.

7.1.4. Test Cases

• doctest for documentation strings in some non-class functions

• test cases are handled with unittest, testing is however typically done by pytest that loads and executed the created TestSuites

• TraceGUI has a debug method, which runs a separate thread, from which actions are executed:

  • Note that each UI actions needs to be run in the main thread, for this TraceGUI provides _do_in_main and _set_in_main methods

  • generally we want to do an action, after the old one has finished. For this TraceGUI._wait_for_idle is implemented.

7.1.5. Documentation Testing

The workflow file doc_test.yml runs multiple test environments. This includes:

• tox -e docsbuildcheck: Testing of correct documentation building while not permitting warning

• tox -e linkcheck: Testing of all documentation links

• tox -e doctest: Testing of most documentation code examples (using doctest)

7.1.6. Python Version Testing

Testing of the compatibility with multiple python versions is done with the pyver_comp.yml workflow. It executes a subset of tests (tox -e fast) for multiple python main versions in an Ubuntu runner.

7.1.7. Installation Testing

Weekly tests that try to install the package and execute some quick tests. Ensures that requirements and available PyPI packages are up-to-date or need changes. The workflow is located in install_test.yml.

7.1.8. Platform Testing

Platform testing is done with the os_comp.yml workflow. It executes the os tox environment, which runs a subset of all available tests. These tests include relevant possibly platform-dependent cases, including:

• installation

• loading and saving of files

• handling of filenames and paths

• loading sensitivity and image presets (which also are files inside the library)

• opening the gui and plots

• enforcing a specific float bit width

• detecting the number of cores and multithreading

The workflow is executed with the current python version and both the latest macOC and Windows runners. Linux is not included, as all other workflows already run on Ubuntu.

7.1.9. Github Workflows

• see .github/workflows/

• Actions: Main testing, OS Compatibility, Older Python Version Compatibility

• all get run on a push the repository or can get run manually

7.1.10. Manual Tests

Unfortunately, not all tests can be automated. The following test cases need to be handled manually:

Checking that optrace.plots.block actually pauses the program execution

This can’t be tested automatically, as it would halt testing. Maybe there is a way by using multithreading or multiprocessing and a timer? What about coverage information in such cases?

Correct and nice formatting inside plots and the GUI

Can only be tested by a human viewer.

Usage in Python Notebooks or inline IDEs such as Spyder

• Installation:

  • see https://docs.spyder-ide.org/current/installation.html

  • python3 -m venv spyder-env

  • source spyder-env/bin/activate

• Testing:

  • make sure pyplot windows are displayed correctly (plot viewer) with enough dpi

  • make sure this is also the case for plots generated by the GUI

7.1.11. Tox configuration in tox.toml

env_list = ["py{310,311,312,313}"]
requires = ["tox-ignore-env-name-mismatch"]
skip_missing_interpreters = true

[env_run_base]
description = "Main testing environment"
pass_env = ["HOME", "DISPLAY", "WAYLAND_DISPLAY", "QT_QPA_PLATFORM", "XAUTHORITY", "GITHUB_ACTIONS", "PYTHON_CPU_COUNT",
            "XDG_SESSION_TYPE", "XDG_SESSION_DESKTOP"]
env_dir = "{toxworkdir}{/}env_static"
runner = "ignore_env_name_mismatch"
deps = ["pylint"]
extras = ["tests", "docs"]
commands = [
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "tests{/}test_geometry.py", "tests{/}test_misc.py", 
        "tests{/}test_image.py", "tests{/}test_refraction_index.py", "tests{/}test_scope.py", 
        "tests{/}test_spectrum.py", "tests{/}test_surface.py", "tests{/}test_tma.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "tests{/}test_tracer.py", "tests{/}test_tracer_hurb.py", 
        "tests{/}test_tracer_special.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "tests{/}test_color.py", "tests{/}test_convolve.py", 
        "tests{/}test_load.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "tests{/}test_plots.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "tests{/}test_examples.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m",
        "pytest", {replace="posargs", extend=true}, "-m", "gui1", "tests{/}test_gui.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "-m", "gui2", "tests{/}test_gui.py"
    ],
    ["coverage", "run", "-p", "--source=optrace", "--branch", "-m", 
        "pytest", {replace="posargs", extend=true}, "-m", "gui3", "tests{/}test_gui.py"
    ],
    ["coverage", "combine"],
    ["coverage", "report", "--show-missing"],
    ["coverage", "erase"]
]

[env.os]
description = "Testing of system dependent functionality"
commands = [["pytest", "-m", "os", {replace="posargs", extend=true}]]

[env.fast]
description = "Don't test slower tests (> 10 seconds)"
commands = [
    ["pytest", "-m", "not slow", "--ignore=tests{/}test_examples.py", "--ignore=tests{/}test_gui.py", 
        {replace="posargs", extend=true}],
    ["pytest", "-m", "not slow", {replace="posargs", extend=true}, "tests{/}test_examples.py"],
    ["pytest", "-m", "not slow", {replace="posargs", extend=true}, "tests{/}test_gui.py"],
]

[env.install]
description = "Quick testing of installation"
commands = [["pytest", "-v", "-m", "install", {replace="posargs", extend=true}]]

[env.linkcheck]
description = "Check all links in software and documentation"
commands = [["sphinx-build", "-M", "linkcheck", "docs{/}source", "docs{/}build", "-j", "auto"]]

[env.docsbuildcheck]
description = "Nitpicky documentation building, turn warnings into errors"
commands = [["sphinx-build", "-M", "html", "docs{/}source", "docs{/}build{/}htmlcheck", "-n", "-W"]]

[env.doctest]
description = "Test documentation doctest snippets"
commands = [["sphinx-build", "-M", "doctest", "docs{/}source", "docs{/}build", "-j", "auto"]]

[env.docs]
description = "Create documentation and update structure and changelog"
allowlist_externals = ["{/}usr{/}bin{/}bash"]
commands = [
    ["bash", "docs{/}generate_changelog.sh"],
    ["bash", "docs{/}generate_bib_structure.sh"],
    ["sphinx-build", "-M", "html", "docs{/}source", "docs{/}build", "-n"]
]

[env.pylint]
description = "Run pylint with excluded unnecessary rules"
skip_install = true
commands = [
    ["pylint", "optrace", "-j", "0", "--max-line-length", "120", "--variable-naming-style", "any",
    "--disable", "W0612,W0613,W0106,W0212,E1101,R0902,R0904,R0911,R0912,R0913,R0914,R0915,R0916,C0301,C0302",
    "--good-names-rgxs", "^[_a-z][_a-z0-9]?$"]
]

Notes

• tox-ignore-env-name-mismatch allows us to reuse the tox env for all actions

• parallelized testing with pytest-xdist could be possible, but we wouldn’t gain much from it as the heaviest tasks are already multithreaded

• pytest-xvfb uses xvfb as headless-display. Use the option --no-xvfb to actually see the plots/windows.

• when GUI tests fail on wayland, first run xhost +