harbor — AGENTS.md

# CLAUDE.md - Harbor Framework

> **Breaking changes**: See [CHANGELOG.md](CHANGELOG.md) for recent breaking changes to the agent and environment APIs and migration guidance.

## Project Overview

Harbor is a framework for evaluating and optimizing AI agents and language models. It provides:

- **Agent Evaluation**: Run evaluations of arbitrary agents (Claude Code, OpenHands, Codex CLI, Aider, etc.) against benchmark tasks
- **Benchmark Support**: Interface with standard benchmarks (SWE-Bench, Terminal-Bench, Aider Polyglot, etc.)
- **Parallel Execution**: Conduct experiments in thousands of environments in parallel via providers like Daytona and Modal
- **RL Optimization**: Generate rollouts for reinforcement learning optimization

## Quick Start Commands

```bash
# Install
uv tool install harbor

# Run a benchmark
harbor run --dataset [email protected] --agent claude-code --model anthropic/claude-opus-4-1 --n-concurrent 4

# Pass environment variables to the agent
harbor run --dataset [email protected] --agent claude-code --model anthropic/claude-opus-4-1 \
  --ae AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  --ae AWS_REGION=us-east-1

# List available datasets
harbor datasets list

# Get help
harbor run --help
```

## Repository Structure

This is a monorepo containing the Harbor CLI, documentation website, and results viewer.

```
harbor/
├── src/harbor/           # Main CLI source code
│   ├── agents/           # Agent implementations
│   │   ├── base.py       # BaseAgent abstract class
│   │   ├── factory.py    # Agent factory for instantiation
│   │   ├── installed/    # Built-in agent implementations
│   │   ├── terminus_2/   # Terminus agent implementation
│   │   ├── oracle.py     # Oracle agent (for testing)
│   │   └── nop.py        # No-op agent
│   ├── cli/              # Command-line interface (Typer-based)
│   │   ├── main.py       # Main CLI entry point
│   │   ├── jobs.py       # Job management commands
│   │   ├── datasets.py   # Dataset commands
│   │   ├── trials.py     # Trial management
│   │   ├── tasks.py      # Task management
│   │   ├── traces.py     # Trace viewing
│   │   ├── sweeps.py     # Parameter sweeps
│   │   ├── adapters.py   # Adapter commands
│   │   ├── adapter_wizard.py  # Interactive adapter creation
│   │   ├── publish.py    # Package publishing
│   │   ├── analyze.py    # Analysis commands
│   │   ├── cache.py      # Cache management
│   │   ├── view.py       # Results viewing
│   │   ├── admin/        # Admin commands
│   │   ├── annotator/    # Annotation tools
│   │   ├── quality_checker/  # Quality verification
│   │   ├── template-adapter/ # Adapter templates
│   │   ├── template-metric/  # Metric templates
│   │   └── template-task/    # Task templates
│   ├── environments/     # Execution environments
│   │   ├── base.py       # BaseEnvironment abstract class
│   │   ├── factory.py    # Environment factory
│   │   ├── docker/       # Local Docker environment
│   │   ├── daytona.py    # Daytona cloud environment
│   │   ├── e2b.py        # E2B environment
│   │   ├── modal.py      # Modal environment
│   │   ├── runloop.py    # Runloop environment
│   │   ├── apple_container.py  # Apple container environment
│   │   ├── gke.py        # Google Kubernetes Engine
│   │   ├── openshift.py  # Red Hat Openshift environment
│   │   └── novita.py     # Novita AI Sandbox environment
│   ├── models/           # Pydantic data models
│   │   ├── agent/        # Agent context and metadata
│   │   ├── job/          # Job configuration and results
│   │   ├── task/         # Task configuration
│   │   ├── trial/        # Trial configuration and results
│   │   ├── metric/       # Metric definitions
│   │   ├── package/      # Package registry models
│   │   ├── trajectories/ # ATIF trajectory format
│   │   ├── verifier/     # Verification results
│   │   └── registry.py   # Dataset registry models
│   ├── orchestrators/    # Trial orchestration
│   ├── verifier/         # Test verification system
│   ├── inspect/          # Inspection utilities
│   ├── analyze/          # Analysis backend (LLM-powered)
│   ├── auth/             # Authentication (OAuth callback server)
│   ├── publisher/        # Package publishing and registry DB
│   ├── storage/          # Storage backends (Supabase)
│   ├── db/               # Database types
│   ├── llms/             # LLM integrations (LiteLLM)
│   ├── dataset/          # Dataset handling
│   ├── registry/         # Dataset registry
│   ├── tasks/            # Task utilities
│   ├── trial/            # Trial utilities
│   ├── metrics/          # Metrics collection
│   ├── mappers/          # Data mappers
│   ├── viewer/           # Results viewer UI
│   └── utils/            # Utility functions
├── adapters/             # Benchmark adapters (convert external datasets)
├── apps/
│   └── viewer/           # Results viewer web app (React Router, Vite)
├── docs/                 # Documentation website (Next.js, Fumadocs)
├── examples/             # Example configurations and tasks
│   ├── tasks/            # Example task definitions
│   ├── agents/           # Agent configuration examples
│   ├── configs/          # Job configuration examples
│   ├── datasets/         # Dataset examples
│   ├── metrics/          # Custom metrics examples
│   ├── prompts/          # Prompt templates
│   └── training/         # Training examples
├── rfcs/                 # RFC specifications
├── scripts/              # Utility scripts
├── skills/               # Claude Code skills
├── tests/                # Test suite
│   ├── unit/             # Unit tests
│   ├── integration/      # Integration tests
│   ├── runtime/          # Runtime tests (may need Docker)
│   └── golden/           # Golden file tests
├── dataset/              # Local dataset storage (jobs/)
├── jobs/                 # Job output storage
└── trials/               # Trial output storage
```

## Key Concepts

### Tasks

A task is a unit of evaluation defined in a directory with:
- `task.toml` - Configuration (timeouts, resources, metadata)
- `instruction.md` - Natural language task description for the agent
- `environment/` - Dockerfile or environment definition
- `tests/` - Verification scripts (test.sh writes reward to `/logs/verifier/reward.txt`)
- `solution/` (optional) - Reference solution

### Agents

Agents implement `BaseAgent` (in `src/harbor/agents/base.py`):
```python
class BaseAgent(ABC):
    SUPPORTS_ATIF: bool = False  # Set True if agent supports trajectory format
    SUPPORTS_WINDOWS: bool = False  # Set True if agent can run in Windows containers

    @staticmethod
    @abstractmethod
    def name() -> str: ...
    @abstractmethod
    def version(self) -> str | None: ...
    @abstractmethod
    async def setup(self, environment: BaseEnvironment) -> None: ...
    @abstractmethod
    async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None: ...
```

Built-in agents:
- **Installed agents**: `claude-code`, `copilot-cli`, `openhands`, `openhands-sdk`, `aider`, `codex`, `goose`, `gemini-cli`, `hermes`, `qwen-coder`, `opencode`, `cursor-cli`, `cline-cli`, `mini-swe-agent`, `swe-agent`, `kimi-cli`, `rovodev-cli`, `trae-agent`, `deerflow`
- **Internal agents**: `terminus`, `terminus-1`, `terminus-2` (Terminus agent variants)
- **Utility agents**: `oracle` (for testing), `nop` (no-operation)

### Environments

Environments implement `BaseEnvironment` (in `src/harbor/environments/base.py`):
- **docker** - Local Docker execution (default)
- **daytona** - Daytona cloud
- **e2b** - E2B sandbox
- **modal** - Modal cloud
- **runloop** - Runloop environment
- **apple_container** - Apple container environment
- **gke** - Google Kubernetes Engine
- **Openshift** - Red Hat Openshift Container Platform
- **novita** - Novita AI Agent Sandbox environment

### Trials and Jobs

- **Trial**: Single execution of an agent on a task
- **Job**: Collection of trials (multiple agents × tasks × attempts)

## Development Setup

```bash
# Clone and setup
git clone https://github.com/harbor-framework/harbor.git
cd harbor

# Install dependencies (Python 3.12+ required)
uv sync --all-extras --dev

# Run tests
uv run pytest tests/

# Run with coverage
uv run pytest tests/ --cov=src/harbor --cov-report=term-missing
```

## Testing

### Test Markers
```python
@pytest.mark.unit           # Fast, no external dependencies
@pytest.mark.integration    # Requires external services (may be mocked)
@pytest.mark.runtime        # May need Docker
@pytest.mark.asyncio        # Async tests (auto mode enabled)
```

### Running Tests

**When verifying changes, only run `uv run pytest tests/unit/` unless the change specifically affects integration-tested code and integration tests are necessary.**

Do not test CLI help panels. Typer/Rich help output changes with terminal width,
colors, and platform; test command behavior, parser wiring, or callback effects instead.

```bash
# Unit tests (default for verifying changes)
uv run pytest tests/unit/

# All tests (only when needed)
uv run pytest tests/

# Specific marker
uv run pytest -m unit

# With verbose output
uv run pytest -v --tb=short
```

## Code Style and Linting

- **Formatter**: Ruff (format on changed files in CI)
- **Linter**: Ruff (check with `--fix`)
- **Type checker**: ty (run via `uv run ty check`)
- **Imports**: First-party imports from `harbor` (configured in pyproject.toml)
- **File I/O**: Prefer `Path.write_text()` / `Path.write_bytes()` / `Path.read_text()` over `with open(...)` whenever possible
- **Internal invariants**: Prefer explicit `if` checks that raise clear errors over `assert`; runtime guards must not disappear under optimized Python execution
- **Async concurrency**: Always prefer `asyncio.TaskGroup` over `asyncio.gather`
- **Logging**: Prefer `logger.debug` by default. Only use `logger.info` or higher when the information is critical for the user to see at runtime

```bash
# Format code
uv run ruff format .

# Lint and fix
uv run ruff check --fix .

# Type check
uv run ty check
```

Always run `uv run ruff check --fix .`, `uv run ruff format .`, and `uv run ty check` after making any code changes.

## CI/CD Workflows

Located in `.github/workflows/`:
- `pytest.yml` - Runs tests on PR/push to main
- `ruff-format.yml` - Checks formatting on PRs
- `ty.yml` - Type checking
- `claude.yml` - Claude-related workflows
- `claude-code-review.yml` - Code review automation
- `sync-registry.yml` - Syncs dataset registry
- `adapter-review.yml` - Adapter review automation
- `check-registry-format.yml` - Validates registry format
- `pr-labeler.yml` - Auto-labels PRs
- `update-parity-summary.yml` - Updates benchmark parity summary

## Key Patterns

### Pydantic Models
All configuration and data models use Pydantic v2:
```python
from pydantic import BaseModel, Field

class MyConfig(BaseModel):
    name: str
    timeout_sec: float = 60.0
    kwargs: dict[str, Any] = Field(default_factory=dict)
```

### Async Operations
Environment and agent operations are async:
```python
async def run_trial():
    await environment.start(force_build=False)
    await agent.setup(environment)
    await agent.run(instruction, environment, context)
    result = await verifier.verify()
    await environment.stop(delete=True)
```

### Lazy Imports
The main `__init__.py` uses lazy imports to avoid loading heavy dependencies at import time.

## Adapters

Adapters convert external benchmark datasets to Harbor task format:
```
adapters/{benchmark-name}/
├── adapter.py       # Main conversion logic
├── run_adapter.py   # CLI for running the adapter
├── README.md        # Documentation
└── template/        # Task template files
```

Supported adapters (50+):
- **SWE-Bench family**: `swebench`, `swebenchpro`, `swebench_multilingual`, `swesmith`, `swtbench`, `multi-swe-bench`, `swelancer`
- **Code generation**: `aider_polyglot`, `autocodebench`, `compilebench`, `livecodebench`, `humanevalfix`, `evoeval`, `deveval`, `bigcodebench_hard`, `crustbench`, `ds1000`, `quixbugs`
- **Research/ML**: `mlgym-bench`, `ml_dev_bench`, `replicationbench`, `codepde`, `kumo`
- **Reasoning/QA**: `aime`, `gpqa-diamond`, `usaco`, `ineqmath`, `simpleqa`, `mmmlu`, `reasoning-gym`, `satbench`
- **Data/SQL**: `bird_bench`, `spider2-dbt`, `spreadsheetbench-verified`
- **Domain-specific**: `financeagent`, `medagentbench`, `labbench`, `lawbench`, `pixiu`, `bixbench`
- **Agents/Tools**: `gaia`, `bfcl`, `dabstep`, `dacode`, `featurebench`, `strongreject`, `rexbench`
- **Multimodal**: `mmau`
- **Other**: `sldbench`, `adebench`, `algotune`, `arc_agi_2`, `qcircuitbench`

## Environment Variables

Common environment variables:
- `ANTHROPIC_API_KEY` - For Claude-based agents
- `OPENAI_API_KEY` - For OpenAI-based agents
- `DAYTONA_API_KEY` - For Daytona cloud execution
- Model provider keys as needed

To pass arbitrary environment variables to an agent at runtime, use `--ae` / `--agent-env`:
```bash
harbor run ... --ae AWS_REGION=us-east-1 --ae CUSTOM_VAR=value
```

## Common Tasks for AI Assistants

### Adding a New Agent
1. Create `src/harbor/agents/installed/{agent_name}.py`
2. Extend `BaseInstalledAgent` or `BaseAgent`
3. Register in `AgentName` enum (`src/harbor/models/agent/name.py`)
4. If the agent supports Windows containers, set `SUPPORTS_WINDOWS = True`

### Adding a New Environment Type
1. Create `src/harbor/environments/{env_name}.py`
2. Extend `BaseEnvironment`
3. Register in `EnvironmentType` enum
4. Update `environments/factory.py`

### Creating a New Adapter
1. Create directory `adapters/{benchmark_name}/`
2. Implement `adapter.py` with dataset loading and task generation
3. Create `run_adapter.py` CLI entry point
4. Add README.md with usage instructions

### Modifying the CLI
The CLI uses Typer and is structured in `src/harbor/cli/`:
- Add new command groups as `{name}_app = Typer()`
- Register in `main.py` with `app.add_typer()`

## File Naming Conventions

- Python files: `snake_case.py`
- Test files: `test_{module_name}.py`
- Config files: `task.toml`, `config.json`
- Markdown: `README.md`, `instruction.md`

## Important Notes

- Python 3.12+ is required
- Use `uv` for package management
- For Supabase work, prefer the Supabase CLI over the Supabase MCP for remote database inspection or mutation.
- Supabase/PostgREST queries that may return more than 1,000 rows must paginate explicitly with `.range(...)` or an equivalent keyset/limit loop; do not rely on the default response size.
- Async/await patterns are used throughout for I/O operations
- All models use Pydantic v2 for validation and serialization
- The verifier writes reward to `/logs/verifier/reward.txt` or `/logs/verifier/reward.json`
- Agent trajectories follow the ATIF format (Agent Trajectory Interchange Format)
- It's often convenient to test changes using `harbor run -t hello-world/hello-world -e daytona`