Engagement Report: A Test Suite That Depended on the Wrong Python

Sixth sample engagement report, on Tit for Tat, newsroom-focused forensic attribution and adversarial security tooling: chain-of-custody hashing, ASN profiling, canary and zero-width-character detection. This repo has the strongest test setup of the ones covered so far, a real integration suite that starts an actual local server and runs the tool against it, rather than mocking the target.

Scope and Objective

Target: tests/test_integration.py, which starts evasion/defender_server.py as a live local HTTP server and runs three of the tool's own commands against it: audit, forensic --stealth, and canary-scan.

Objective: confirm the suite runs end to end tonight, in a clean environment, and produces three real passes against a real (if local) target, not a result inherited from however the repo's own development machine happened to be configured.

Finding: The Test Depended on an Unpinned Interpreter

All three tests failed on the first run, every one with the identical error: ModuleNotFoundError: No module named 'requests', even though requests was installed and importable in the Python environment actually running pytest. The cause: each test shells out with subprocess.run(["python3", ...]), a literal string naming whatever binary resolves first on the system PATH. That is not necessarily the same interpreter running the test suite itself, and on this machine it wasn't: python3 on PATH pointed at a system Python with no project dependencies installed, while the dependencies were correctly installed into the virtual environment actually running pytest.

This is a reproducibility gap, not a one-off local quirk. A test suite that passes or fails depending on which python3 happens to be first on the invoker's PATH, independent of the environment the suite claims to be testing, will behave inconsistently across machines, CI runners, and contributors, each showing a different result for the same code. The fix: replace the literal string "python3" with sys.executable in all four subprocess calls across the suite, which is Python's own documented way to guarantee a subprocess uses the exact interpreter currently running, dependencies and all.

Verification

Re-run immediately after the fix, same machine, same clean environment:

tests/test_integration.py::test_audit_against_defender PASSED
tests/test_integration.py::test_stealth_forensic_against_defender PASSED
tests/test_integration.py::test_canary_scan_against_defender PASSED
3 passed in 3.48s

All three tests spin up the real local defender server, point the real CLI at it over real HTTP, and tear the server down cleanly at the end, generating a report file in the process. Nothing here was mocked; this is the setup CLAUDE.md's own notes described as "spins up an actual local defender server" and it held up under a fresh run tonight, once the interpreter-selection bug was out of the way.

Mitigations and Next Steps

1. Never shell out to a bare interpreter name in a test. sys.executable costs nothing and removes an entire class of "works on my machine" failure. This is a five-minute fix with an outsized blast radius if missed: a contributor or a CI runner with a different default python3 gets a false negative that has nothing to do with the code under test.
2. This is also a CI-readiness flag. A suite this solid (real server, real HTTP, real teardown) is worth running on every commit. The interpreter bug would have shown up immediately in CI with a consistent, controlled environment; it only surfaced here because of a local PATH quirk, which argues for getting this suite into CI sooner rather than later.
3. No findings against the tool's actual detection logic. This pass was entirely about test-harness correctness; the audit, forensic-stealth, and canary-scan commands all completed successfully against the real defender server once the environment issue was resolved. That's a real pass, not a vacuous one, but it's also not a claim about detection accuracy against adversarial input, which is a separate, deeper pass this repo's test suite doesn't yet cover.

---

GhostInThePrompt.com // The suite failed on a string, not on the tool. Fixed the string.