(iceberg-python) branch main updated: Add basic assumptions to AGENTS.md (#3552)

[fo...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

Fokko pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git


The following commit(s) were added to refs/heads/main by this push:
     new 768a29d50 Add basic assumptions to AGENTS.md (#3552)
768a29d50 is described below

commit 768a29d50a1c749e3b3455747cda151217654960
Author: Alex Stephen <13...@users.noreply.github.com>
AuthorDate: Sat Jul 18 13:29:05 2026 -0700

    Add basic assumptions to AGENTS.md (#3552)
    
    <!--
    Thanks for opening a pull request!
    -->
    
    <!-- In the case this PR will resolve an issue, please replace
    ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
    <!-- Closes #${GITHUB_ISSUE_ID} -->
    
    # Rationale for this change
    Our AGENTS.md just talks about the security model. We get a non-trivial
    amount of AI activity on this repo and our AGENTS.md should reflect some
    basic assumptions we want for those PRs.
    
    A lot of this was inspired by [Iceberg
    AGENTS.md](https://github.com/apache/iceberg/blob/main/AGENTS.md).
    
    This is just meant to be an initial version with some thoughts that I've
    seen and some of the basic architecture (as seen in the Iceberg
    AGENTS.md)
    
    ## Are these changes tested?
    Just Markdown.
    
    ## Are there any user-facing changes?
    No.
    
    <!-- In the case of user-facing changes, please add the changelog label.
    -->
    
    ---------
    
    Co-authored-by: Fokko Driesprong <fo...@apache.org>
---
 AGENTS.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 67 insertions(+), 3 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 2606a579b..1f7b18a54 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -17,10 +17,9 @@
   under the License.
 -->
 
-# Apache Iceberg Python — Agent Instructions
+# PyIceberg — Agent Instructions
 
-This file provides repository-specific guidance for automated agents working
-in this repository.
+Project conventions, architecture, and coding patterns synthesized from PyIceberg developers.
 
 ## Security Model
 
@@ -28,3 +27,68 @@ When assessing potential vulnerabilities or calibrating automated security
 findings, use [`SECURITY-THREAT-MODEL.md`](SECURITY-THREAT-MODEL.md) as the
 authoritative detailed description of this repository's security boundaries,
 trust assumptions, and non-boundaries.
+
+## Architecture
+
+PyIceberg is a pure-Python library — it has no separate engine modules. The code
+lives under `pyiceberg/`, organized by concern rather than by engine:
+
+- **`schema.py`, `types.py`, `transforms.py`, `partitioning.py`, `conversions.py`**: The table spec core — schemas, types, partition specs, and value conversions. Must stay engine- and catalog-agnostic.
+- **`table/`**: Table abstraction, metadata (`metadata.py`), snapshots, refs, sort orders, and the transaction/update machinery (`table/update/`). The commit path lives here.
+- **`catalog/`**: Catalog implementations — `rest/`, `hive`, `glue`, `dynamodb`, `sql`, `bigquery_metastore`, `memory`, `noop`. New catalogs subclass the base `Catalog` in `catalog/__init__.py`. Catalog-specific assumptions must not leak into `table/` or the spec core.
+- **`io/`**: The `FileIO` abstraction over storage. `pyarrow.py` and `fsspec.py` are the two backends. Never hard-code a storage SDK where `FileIO` exists.
+- **`expressions/`**: The expression DSL and its visitors (predicate binding, projection, evaluation).
+- **`avro/`, `manifest.py`**: Manifest and Avro read/write — performance-sensitive, partly accelerated by Cython.
+- **`cli/`**: The `pyiceberg` command-line interface (Click + Rich).
+- **`utils/`**: Shared helpers (`deprecated.py`, `concurrent.py`, `config.py`, `bin_packing.py`, `singleton.py`, etc.). Check here before writing new utility code.
+
+## Coding Conventions
+
+### Style & Typing
+
+- Formatting and linting are enforced by `ruff` via `prek` (pre-commit): line length **130**, double-quoted strings, isort with `pyiceberg`/`tests` as first-party. Run `make lint` — ruff autofixes most issues.
+- Full type annotations are required (`mypy` runs in strict mode: `disallow_untyped_defs`, `no_implicit_optional`, `warn_unused_ignores`). Avoid Any types.
+- Docstrings follow the project's pydocstyle config; one-line summaries on public functions. No personal pronouns in comments.
+- Use domain-specific exceptions from pyiceberg/exceptions.py for Iceberg-specific error conditions. For invalid arguments/values, ValueError is acceptable. Don't raise bare Exception.
+- Comments should be succinct and follow the same style of comments found in the rest of the codebase.
+
+### Dependencies
+
+- Large/integration libraries must be **optional extras** in `pyproject.toml`, not core `dependencies`.
+
+## Testing
+
+- Bias towards adding tests to existing files, rather than creating new files.
+- Use existing test fixtures when possible.
+- We have a strong bias towards integration testing over mocks. Mocks should be avoided whenever possible and should only be used if similar, existing tests are using mocks.
+
+## Commands
+
+- **Install / set up dev env:** `make install` (installs `uv`, syncs all extras, builds Cython, installs pre-commit hooks)
+- **Run unit tests:** `make test`
+- **Run a subset:** `make test PYTEST_ARGS="-v -k <pattern>"`
+- **Integration tests (Spark/Docker):** `make test-integration` (rebuild with `make test-integration-rebuild`)
+- **Cloud storage suites:** `make test-s3` / `make test-adls` / `make test-gcs`
+- **Lint & format:** `make lint`
+- **Docs preview / build:** `make docs-serve` / `make docs-build`
+- **Clean build artifacts:** `make clean`
+- **Use a specific Python:** prefix with `PYTHON=3.12`, e.g. `PYTHON=3.12 make install`
+
+## PR & Commit Conventions
+
+- Ensure that there are no existing PRs for this feature before beginning development.
+- One concern per PR. Keep unrelated formatting/import churn out of feature PRs.
+- Keep the first version of a PR minimal; defer optimizations and edge cases to follow-ups.
+- Commit messages explain the *what* and *why*, not line-by-line implementation. Be as succinct as possible.
+- The Apache License header is required on every new source file (enforced by `./dev/check-license`).
+- Run `make lint` and `make test` before pushing; CI runs both plus the lockfile check.
+
+## Boundaries
+
+- **Never** add a hard (non-optional) dependency without discussion — keep heavy/integration libraries as optional extras.
+- **Never** edit `uv.lock` by hand — regenerate it with `uv lock`.
+- **Never** reach for a storage SDK (`boto3`, `s3fs`, `gcsfs`, `adlfs`) outside its `FileIO` backend module.
+- **Never** break the public API or remove a public method without a `@deprecated` cycle.
+- **Never** leak catalog- or engine-specific assumptions into the spec core (`schema.py`, `types.py`, `table/metadata.py`).
+- **Never** commit secrets, credentials, or cloud tokens (integration tests can delete catalog data — never point them at production).
+- **Ask first** before adding any third-party dependency or promoting internal (`_`-prefixed) APIs to public.