study_runner

agentevac.analysis.study_runner

End-to-end calibration study orchestrator.

study_runner.py is the top-level entry point for running a complete parameter calibration study. It chains three stages:

1. **Experiment sweep** (``experiments.run_parameter_sweep``):
   Spawns one headless ``agentevac.simulation.main`` subprocess per grid cell and collects
   the metrics JSON and replay JSONL for each case.

2. **Calibration fit** (``calibration.fit_agent_parameters``):
   Scores each run against the reference metrics and ranks candidates by
   goodness of fit.

3. **Report export**:
   Writes three output artifacts to a timestamped study directory:
       - ``experiments/experiment_results.json``   : Full sweep results.
       - ``experiments/experiment_results.csv``    : Flat CSV summary.
       - ``calibration_report.json``               : Ranked calibration results.
       - ``study_report.json``                     : Top-level study summary.

Each call to run_study creates a fresh timestamped directory under output_dir so studies do not overwrite each other.

run_study(*, reference_path, script_path='agentevac/simulation/main.py', python_executable=None, output_dir='outputs/studies', sumo_binary='sumo', run_mode='record', timeout_s=None, sigma_values=None, delay_values=None, trust_values=None, scenario_values=None, messaging_enabled=True, weights=None, top_k=5, map_name=None)

Run a complete parameter calibration study: sweep → fit → report.

Creates a timestamped study directory, runs all grid cases, fits against the reference, and writes summary reports.

Parameters:

Name	Type	Description	Default
reference_path	str	Path to the reference metrics JSON (target behaviour).	required
script_path	str	Path to agentevac/simulation/main.py.	'agentevac/simulation/main.py'
python_executable	Optional[str]	Python interpreter; defaults to sys.executable.	None
output_dir	str	Base directory for all study outputs.	'outputs/studies'
sumo_binary	str	SUMO binary name ("sumo" for headless).	'sumo'
run_mode	str	"record" or "replay".	'record'
timeout_s	Optional[float]	Per-case subprocess timeout in seconds.	None
sigma_values	Optional[List[float]]	INFO_SIGMA values to sweep.	None
delay_values	Optional[List[float]]	INFO_DELAY_S values to sweep.	None
trust_values	Optional[List[float]]	DEFAULT_THETA_TRUST values to sweep.	None
scenario_values	Optional[List[str]]	Scenario mode strings to sweep.	None
messaging_enabled	bool	Whether inter-agent messaging is active for all cases.	True
weights	Optional[Dict[str, float]]	Per-metric calibration weight overrides.	None
top_k	int	Number of top-ranked candidates to include in the calibration report.	5
map_name	Optional[str]	Map config directory name (e.g., "lytton" or "halifax").	None

Returns:

Type	Description
Dict[str, Any]	A study summary dict containing timing information, case counts, file paths,
Dict[str, Any]	and the best-fit parameter configuration.

Source code in agentevac/analysis/study_runner.py

def run_study(
    *,
    reference_path: str,
    script_path: str = "agentevac/simulation/main.py",
    python_executable: Optional[str] = None,
    output_dir: str = "outputs/studies",
    sumo_binary: str = "sumo",
    run_mode: str = "record",
    timeout_s: Optional[float] = None,
    sigma_values: Optional[List[float]] = None,
    delay_values: Optional[List[float]] = None,
    trust_values: Optional[List[float]] = None,
    scenario_values: Optional[List[str]] = None,
    messaging_enabled: bool = True,
    weights: Optional[Dict[str, float]] = None,
    top_k: int = 5,
    map_name: Optional[str] = None,
) -> Dict[str, Any]:
    """Run a complete parameter calibration study: sweep → fit → report.

    Creates a timestamped study directory, runs all grid cases, fits against the
    reference, and writes summary reports.

    Args:
        reference_path: Path to the reference metrics JSON (target behaviour).
        script_path: Path to ``agentevac/simulation/main.py``.
        python_executable: Python interpreter; defaults to ``sys.executable``.
        output_dir: Base directory for all study outputs.
        sumo_binary: SUMO binary name (``"sumo"`` for headless).
        run_mode: ``"record"`` or ``"replay"``.
        timeout_s: Per-case subprocess timeout in seconds.
        sigma_values: ``INFO_SIGMA`` values to sweep.
        delay_values: ``INFO_DELAY_S`` values to sweep.
        trust_values: ``DEFAULT_THETA_TRUST`` values to sweep.
        scenario_values: Scenario mode strings to sweep.
        messaging_enabled: Whether inter-agent messaging is active for all cases.
        weights: Per-metric calibration weight overrides.
        top_k: Number of top-ranked candidates to include in the calibration report.
        map_name: Map config directory name (e.g., ``"lytton"`` or ``"halifax"``).

    Returns:
        A study summary dict containing timing information, case counts, file paths,
        and the best-fit parameter configuration.
    """
    study_dir = _timestamped_study_dir(output_dir)
    experiments_dir = str(Path(study_dir) / "experiments")
    reference = load_reference_scenario(reference_path)

    grid = build_experiment_grid(
        sigma_values=sigma_values,
        delay_values=delay_values,
        trust_values=trust_values,
        scenario_modes=scenario_values,
        base_overrides={
            "messaging_enabled": bool(messaging_enabled),
        },
    )

    started_at = time.time()
    results = run_parameter_sweep(
        grid,
        script_path=script_path,
        python_executable=python_executable,
        output_dir=experiments_dir,
        sumo_binary=sumo_binary,
        run_mode=run_mode,
        timeout_s=timeout_s,
        map_name=map_name,
    )
    finished_experiments_at = time.time()

    exported = export_experiment_results(results, output_dir=experiments_dir)
    fit = fit_agent_parameters(
        {
            "weights": dict(weights or {}),
            "top_k": int(top_k),
        },
        reference=reference,
        experiments_results=results,
    )
    calibration_report = {
        "mode": "batch_fit",
        "reference_path": reference_path,
        "results_json": exported["json"],
        "fit": fit,
    }
    calibration_path = export_calibration_report(
        calibration_report,
        str(Path(study_dir) / "calibration_report.json"),
    )

    failed_cases = sum(1 for row in results if row.get("status") != "ok")
    completed_at = time.time()
    summary = {
        "mode": "experiment_and_calibration",
        "study_dir": study_dir,
        "reference_path": reference_path,
        "started_at_epoch_s": round(started_at, 3),
        "experiments_completed_at_epoch_s": round(finished_experiments_at, 3),
        "completed_at_epoch_s": round(completed_at, 3),
        "durations_s": {
            "experiments": round(finished_experiments_at - started_at, 3),
            "total": round(completed_at - started_at, 3),
        },
        "experiment": {
            "case_count": len(results),
            "failed_cases": failed_cases,
            "results_json": exported["json"],
            "results_csv": exported["csv"],
        },
        "calibration": {
            "report_path": calibration_path,
            "candidate_count": fit.get("candidate_count", 0),
            "best_case": fit.get("best_case"),
        },
    }

    study_report_path = Path(study_dir) / "study_report.json"
    with open(study_report_path, "w", encoding="utf-8") as fh:
        json.dump(summary, fh, ensure_ascii=False, indent=2, sort_keys=True)
        fh.write("\n")

    summary["study_report_path"] = str(study_report_path)
    return summary