Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# PyAthena Development Guide for AI Assistants

## Project Overview

PyAthena is a Python DB API 2.0 (PEP 249) compliant client for Amazon Athena. See `pyproject.toml` for Python version support and dependencies.

## Rules and Constraints

### Git Workflow

- **NEVER** commit directly to `master` — always create a feature branch and PR
- Create PRs as drafts: `gh pr create --draft`

### Import Rules

- **NEVER** use runtime imports (inside functions, methods, or conditional blocks)
- All imports must be at the top of the file, after the license header
- Exception: the existing codebase uses runtime imports for optional dependencies (`pyarrow`, `pandas`, etc.) in source code. For new code, use `TYPE_CHECKING` instead when possible

### Code Quality — Always Run Before Committing

```bash
just format # Auto-fix formatting and imports
just lint # Lint + format check + mypy
```

### Testing

```bash
# ALWAYS run `just lint` first — tests will fail if lint doesn't pass
just test pyathena # Unit tests (runs lint first)
just test sqla # SQLAlchemy dialect tests
just test sqla-async # SQLAlchemy async dialect tests
```

Tests require AWS environment variables. Use a `.env` file (gitignored):

```bash
AWS_DEFAULT_REGION=<region>
AWS_ATHENA_S3_STAGING_DIR=s3://<bucket>/<path>/
AWS_ATHENA_WORKGROUP=<workgroup>
AWS_ATHENA_SPARK_WORKGROUP=<spark-workgroup>
```

```bash
export $(cat .env | xargs) && uv run pytest tests/pyathena/test_file.py -v
```

- Tests mirror source structure under `tests/pyathena/`
- Use pytest fixtures from `conftest.py`
- New features require tests; changes to SQLAlchemy dialects must pass `just test sqla`

#### Test Conventions

- **Class-based tests** for integration tests that use fixtures (cursors, engines): `class TestCursor:` with methods like `def test_fetchone(self, cursor):`
- **Standalone functions** for unit tests of pure logic (converters, parsers, utils): `def test_to_struct_json_formats(input_value, expected):`
- Test file naming mirrors source: `pyathena/parser.py` → `tests/pyathena/test_parser.py`
- **Fixtures**: Cursor/engine fixtures are defined in `conftest.py` and injected by name (e.g., `cursor`, `engine`, `async_cursor`). Use `indirect=True` parametrization to pass connection options:

```python
@pytest.mark.parametrize("engine", [{"driver": "rest"}], indirect=True)
def test_query(self, engine):
engine, conn = engine
```

- **Parametrize** with `@pytest.mark.parametrize(("input", "expected"), [...])` for data-driven tests
- **Integration tests** (need AWS) use cursor/engine fixtures with real Athena queries; **unit tests** (no AWS) call functions directly with test data

### Markdown Lint

`docs/**/*.md` and project-root `*.md` files are linted with [markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2). The config lives at `.markdownlint-cli2.jsonc`. CI runs lint + Sphinx build on PRs that touch docs (`.github/workflows/docs-lint.yaml`).

`markdownlint-cli2` is pinned in `.mise.toml`, so [`mise`](https://mise.jdx.dev/) installs the exact version used in CI. Run locally:

```bash
mise install # one-time: installs markdownlint-cli2
just docs lint # check
just docs format # auto-fix what's possible
just docs build # build the Sphinx site under docs/_build/html
```

## Architecture — Key Design Decisions

These are non-obvious conventions that can't be discovered by reading code alone.

### PEP 249 Compliance

All cursor types must implement: `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `close()`. New cursor features must follow the DB API 2.0 specification.

### Cursor Module Pattern

Each cursor type lives in its own subpackage (`pandas/`, `arrow/`, `polars/`, `s3fs/`, `spark/`) with a consistent structure: `cursor.py`, `async_cursor.py`, `converter.py`, `result_set.py`. When adding features, consider impact on all cursor types.

### Filesystem (fsspec) Compatibility

`pyathena/filesystem/s3.py` implements fsspec's `AbstractFileSystem`. When modifying:

- Match `s3fs` library behavior where possible (users migrate from it)
- Use `delimiter="/"` in S3 API calls to minimize requests
- Handle edge cases: empty paths, trailing slashes, bucket-only paths

### Version Management

Versions are derived from git tags via `hatch-vcs` — never edit `pyathena/_version.py` manually.

### Google-style Docstrings

Use Google-style docstrings for public methods. See existing code for examples.
108 changes: 0 additions & 108 deletions CLAUDE.md

This file was deleted.

1 change: 1 addition & 0 deletions CLAUDE.md