Metadata-Version: 2.5
Name: agentclash-evals
Version: 0.2.0a1
Summary: Local pre-deploy agent eval assertions for CI/CD
Author: AgentClash
License-Expression: MIT
License-File: LICENSE
Keywords: agent,eval,llm,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: pytest==8.4.2
Provides-Extra: dev
Requires-Dist: jsonschema<5,>=4; extra == 'dev'
Requires-Dist: pytest==8.4.2; extra == 'dev'
Description-Content-Type: text/markdown

# AgentClash local evaluation pilot

This maintained Python integration preserves the upstream `assert_agent`,
`evaluate`, result models, and deterministic metrics. See UPSTREAM.md and LICENSE.
Version 0.2.0a1 adds an actual-target refund fixture and complete version-2
pytest session reports. It is an unreleased source pilot, not the SDK included
by installing the published AgentClash npm CLI 2.2.0.

From a source checkout, install in your project environment:

```bash
python3 -m pip install ./packages/agentclash-evals
```

Use a regression bundle produced by the matching ChatGPT compiler. Make your
own Python callable importable and select it explicitly:

```bash
export AGENTCLASH_EVAL_TARGET=your_agent:run
export AGENTCLASH_TARGET_REVISION=your_exact_revision
python3 -m agentclash_eval.runner --tests tests/evaltest --manifest suite-manifest.json --out agentclash-results
```

The target signature is `run(prompt, tools) -> {"outcome": ..., "message": ...}`.
Supported outcomes are `refunded`, `clarify`, and `error`. Each case has a fresh
in-memory refund service; turns in a case share its ledger. Wire the target's
tool boundary to `tools.lookup_charges()`, `tools.refund(charge_id, amount_cents)`,
`tools.get_subscription()`, and `tools.cancel_subscription()`. There is no
production fallback. This runs trusted local Python: it is not a sandbox and
does not intercept arbitrary network access performed by your target.

The sample `agentclash_eval.reference_targets:correct` is available only when
explicitly selected. Selecting a sample is evidence about that sample, not your
agent. The corresponding `faulty_original` and `faulty_double` are negative
controls. Missing targets and unsupported async callables are errors.

Generated tests call `run_refund_case(case, suite=suite, target=your_adapter)`.
It captures actual invocations, every turn, tool calls, final ledger, and
subscription state, and records both passing and failing business checks.
For custom pytest business assertions use `record_check` from
`agentclash_eval.recording` to attach expected/actual evidence, then assert its
returned boolean. Fixture-only tests do not establish actual-agent coverage.
`record_check` captures diagnostics; it does not automatically satisfy the
bounded refund contract. Complete runner/CLI/MCP coverage requires the six
maintained final refund checks plus observed target/tool/state evidence.

JSON and JUnit are always written. Reports bind run ID, suite/file hashes,
required cases, target revision, and SDK version. Missing/skipped/expected-fail
required cases and empty or unobserved executions cannot pass. Cost stays
unknown; target latency is measured separately from scoring/runner duration.
Reports are user-supplied evidence, not cryptographic attestation against a
malicious local target that can modify Python or output files.

Exit codes: 0 complete pass; 1 business assertion failure; 2 incomplete or
configuration/collection error; 3 target/runtime error or deadline; 4 internal
runner/report error. `--run-id`, `--target-revision`, and `--timeout-seconds`
(default 60, maximum 3600) are supported. A fresh subprocess group enforces
POSIX deadlines and cleans up descendants in that process group. Targets that
deliberately detach subprocesses are outside this trusted-code pilot contract.
Native Windows execution is
explicitly blocked pending verified process-tree cleanup; use WSL. Linux is
verified here; macOS requires the native release acceptance matrix.

For development:

```bash
python3 -m pip install -e '.[dev]'
python3 -m pytest
python3 -m build --wheel
```

Local bundle/report contracts differ from hosted challenge packs. The runner
does not call the AgentClash API, require an account, or upload reports. Review
and sanitize reports before deliberately sharing them for comparison.
