[ConnectionBroker] Unicon CLI log files silently deleted after test run

[oboehmer]

Summary

The ConnectionBroker logs the path of the Unicon CLI log file at INFO level:

Unicon CLI log will be written to: .../pyats_results/sd-dc-c8kv-01-cli-4700779846.log

But this file is not present after the test run completes — it is silently deleted before the user has a chance to inspect it. The file is only retained when running with NAC_TEST_DEBUG=true.

Root Cause

The Unicon CLI log files are written to {output_dir}/pyats_results/ during the test run (via connection_broker.py). At the start of report generation, MultiArchiveReportGenerator.generate_reports_from_archives() unconditionally wipes the entire pyats_results/ directory:

nac_test/pyats_core/reporting/multi_archive_generator.py, lines 98–101:

# Clean and prepare results directory
if self.pyats_results_dir.exists():
    shutil.rmtree(self.pyats_results_dir)
self.pyats_results_dir.mkdir(parents=True)

This happens before any archive is extracted and has no DEBUG_MODE or NAC_TEST_PYATS_KEEP_REPORT_DATA guard — unlike the other cleanup steps in the same file (JSONL files) and in orchestrator.py (archive files after extraction).

Impact

• The INFO-level log message referencing the CLI log path creates a false expectation: users are told where the log will be, but the file is gone by the time they look.
• Connection failures, authentication issues, and Unicon state machine errors can only be diagnosed by re-running with NAC_TEST_DEBUG=true.
• This is particularly painful for intermittent or environment-specific failures.

Expected Behavior

The Unicon CLI log files should be retained in the output directory after the test run, at any log level. The shutil.rmtree cleanup should either:

• Move the CLI log files out before wiping, then move them back, or
• Respect DEBUG_MODE / NAC_TEST_PYATS_KEEP_REPORT_DATA the same way the other cleanup steps do.

Steps to Reproduce

1. Run a D2D test against any device (reachable or not)
2. Observe the INFO log: Unicon CLI log will be written to: .../pyats_results/<hostname>-cli-<timestamp>.log
3. After the run completes, check the output directory — the *-cli-*.log file is absent
4. Re-run with NAC_TEST_DEBUG=true — the file is now present

References

• Unicon CLI log file creation: connection_broker.py ~lines 325–340
• Unconditional shutil.rmtree: multi_archive_generator.py lines 98–101
• DEBUG-guarded JSONL cleanup (same file): lines 155–164
• DEBUG-guarded archive cleanup: orchestrator.py lines 836–850

[oboehmer]

Related to #538 — the missing DEBUG_MODE guard on the shutil.rmtree is part of the broader inconsistency that #538 aims to consolidate. Fixing this issue correctly means applying the same shared should_keep_debug_files() utility that #538 proposes, so the two should ideally be addressed together or sequentially.