You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Developer1010x (via GitHub)" <gi...@apache.org> on 2026/09/11 18:22:14 UTC

[PR] feat: run examples/*.py in CI and fix the examples that could not run [datafusion-python]

Developer1010x opened a new pull request, #1732:
URL: https://github.com/apache/datafusion-python/pull/1732

   # Which issue does this PR close?
   
   Closes #1728.
   
   # Rationale for this change
   
   Nothing in CI ran the top-level `examples/*.py`, so nothing noticed when they
   stopped working. Two consequences were already in the tree, and both are the
   first thing a new user copies from.
   
   # What changes are included in this PR?
   
   **`examples/csv-read-options.py` no longer crashes.** It read `data.csv` and
   `data.csv.gz`, neither of which is in the repository, so it raised on the first
   read. It now writes what it needs into a temporary directory: a comma-separated
   CSV, a pipe-separated and single-quoted one so the builder-pattern example
   actually exercises the delimiter and quote it sets, and a gzipped one with a
   comment line so the advanced-options example exercises `with_comment` and
   `with_file_compression_type`. Every example now calls `df.show()`, so the script
   demonstrates the options instead of only naming them.
   
   **Nine scripts printed nothing.** `export.py`, `import.py`, `python-udaf.py`,
   `python-udf.py`, `query-pyarrow-data.py`, `sql-to-pandas.py`,
   `sql-using-python-udaf.py`, `sql-using-python-udf.py` and `substrait.py` ended
   in a bare `assert`. They now print their results; the asserts are unchanged.
   
   **`substrait.py` only worked from the repository root.** It read
   `./testing/data/csv/aggregate_test_100.csv`, so it failed from anywhere else. It
   now resolves that path relative to `__file__`.
   
   **The CI job.** A `Run Python examples` step, gated to `abi3` like the FFI and
   TPC-H steps, running from the repository root against the wheel built earlier in
   the job. Placement and content are both load-bearing:
   
   - It runs **after** the TPC-H generation step, because
     `python-udf-comparisons.py` reads `examples/tpch/data/lineitem.parquet`. The
     issue did not list this one; it shows up as soon as you actually run them.
   - It installs `pandas` and `polars`, which are neither runtime nor dev
     dependencies but are what `import.py` and `export.py` demonstrate converting
     to and from.
   - It relies on the `git submodule update --init` already done by the test step
     for `substrait.py`.
   - It skips five, each for a stated reason in the workflow: `sql-parquet.py`,
     `dataframe-parquet.py` and `sql-to-pandas.py` need the NYC taxi Parquet file
     the README says to download by hand; `ray_pickle_expr.py` needs a Ray cluster;
     `sql-parquet-s3.py` needs network access and AWS credentials.
   - A failing example does not stop the loop. All of them run and the step reports
     every failure at once, so one broken script does not hide the next.
   
   **`examples/README.md`** gains a short "Running the examples" section with a
   table of the per-example prerequisites, and drops the links to
   `sql-on-polars.py`, `sql-on-pandas.py` and `sql-on-cudf.py`, which are not in
   the repository.
   
   ## Verification
   
   All 14 non-skipped examples were run from the repository root against a
   `datafusion` 54.0.0 wheel, with the TPC-H data generated and the `testing`
   submodule initialised. Every one exits 0 and prints output. `ruff check`,
   `ruff format --check` and `codespell` pass, and `shellcheck` is clean on the new
   step's script.
   
   ## One thing found on the way, not fixed here
   
   `CsvReadOptions.with_null_regex` appears to have no effect at read time. With
   `datafusion` 54.0.0:
   
   ```python
   from pathlib import Path
   from datafusion import CsvReadOptions, SessionContext
   
   p = Path("probe.csv")
   p.write_text("id,name,value\n1,alice,10\n2,N/A,20\n3,carol,30\n")
   ctx = SessionContext()
   options = CsvReadOptions().with_has_header(True).with_null_regex(r"^(null|NULL|N/A)$")
   ctx.read_csv(p, options=options).show()
   ```
   
   `N/A` comes back as the literal string rather than NULL, and if the same value
   sits in a column inferred as `Int64` the read fails outright with
   `Parser error: Error while parsing value 'N/A' as type 'Int64'`. The option is
   plumbed through `crates/core/src/options.rs` into DataFusion's `CsvReadOptions`,
   so this looks like it is below this crate. To keep this PR runnable and honest,
   the advanced-options example keeps `with_null_regex` set and puts the `N/A` in a
   string column. Happy to open a separate issue for it.
   
   # Are there any user-facing changes?
   
   No API changes. The examples print their results now, and
   `examples/csv-read-options.py` no longer requires data files that were never
   shipped.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run examples/*.py in CI and fix the examples that could not run [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed pull request #1732: feat: run examples/*.py in CI and fix the examples that could not run
URL: https://github.com/apache/datafusion-python/pull/1732


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run examples/*.py in CI and fix the examples that could not run [datafusion-python]

Posted by "Developer1010x (via GitHub)" <gi...@apache.org>.
Developer1010x commented on PR #1732:
URL: https://github.com/apache/datafusion-python/pull/1732#issuecomment-5664440308

   Fair question, and I would rather answer it straight than dance around it: this was written with an AI agent. The commit carries a `Co-Authored-By: Claude` trailer for that reason, so you are not wrong about any of the three.
   
   The distinction I would offer is not "mine is not AI" but that it was not one pass over the issue text. Against #1736, which is the closest of the three:
   
   - **Coverage.** My step runs *after* the tpchgen-cli step, so `python-udf-comparisons.py` is exercised rather than skipped. It reads `examples/tpch/data/lineitem.parquet`, which that step creates. #1736 places the job before it and skips the example for that reason. 14 examples run instead of 13. The issue does not mention this dependency, it only shows up once you actually run them.
   - **`substrait.py`** reads `./testing/data/csv/aggregate_test_100.csv`, so it only works when the working directory is the repository root. I resolve it relative to `__file__`. Both approaches pass in CI; only one works when a user runs the file directly.
   - **Example 3 of `csv-read-options.py`** configures `delimiter="|"` and `quote="'"`. #1730 and #1736 point it at the comma-separated fixture, so it runs green and prints a single-column table, demonstrating the opposite of what it sets. I write a pipe-delimited file for that example and a gzipped one with a comment line for the advanced-options example.
   - **`examples/README.md`** links `sql-on-polars.py`, `sql-on-pandas.py` and `sql-on-cudf.py`. None of the three are in the repository. Removed, and replaced with a table of what each example needs.
   
   One point where #1736 is better than mine: it gates on `matrix.python-version == '3.12'`, which runs the job once. I followed the neighbouring `matrix.wheel-tag == 'abi3'` convention, and that matches five matrix rows, so my job runs five times where once is enough. Their gate is the correct one and I am happy to switch to it.
   
   On "a single output of dumping the issue into an agent": the thing that argues against that, for whatever it is worth, is what running the examples turned up. `CsvReadOptions.with_null_regex` is accepted and silently ignored (#1735). That traced to `datafusion-datasource-csv`, where `CsvSource::builder` never passes the regex to the `ReaderBuilder` that parses the rows, so it applies during schema inference only: apache/datafusion#25213, fix in apache/datafusion#25254. Chasing that turned up a second, unrelated defect on the same path, where an invalid pattern panics the query task instead of returning an error: apache/datafusion#25260, fix in apache/datafusion#25261. None of that is in issue #1728.
   
   Closing all three is a perfectly reasonable call and I will not take it badly. If it is more useful, I am glad to cut this down to just the CI job, with their gate, so there is a smaller change for someone to review properly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run examples/*.py in CI and fix the examples that could not run [datafusion-python]

Posted by "Developer1010x (via GitHub)" <gi...@apache.org>.
Developer1010x commented on PR #1732:
URL: https://github.com/apache/datafusion-python/pull/1732#issuecomment-5642796519

   Filed the `null_regex` finding mentioned above as #1735 — it turned out to be upstream in `datafusion-datasource-csv`, where the reader's `ReaderBuilder` never gets `with_null_regex`. Nothing in this PR depends on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run examples/*.py in CI and fix the examples that could not run [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1732:
URL: https://github.com/apache/datafusion-python/pull/1732#issuecomment-5664674098

   The response above also appears generated. Closing in accordance with our parent project's AI usage guidance.
   
   https://datafusion.apache.org/contributor-guide/index.html#ai-assisted-contributions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run examples/*.py in CI and fix the examples that could not run [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1732:
URL: https://github.com/apache/datafusion-python/pull/1732#issuecomment-5664384845

   We have 3 PRs all addressing the same issue and all 3 appear to be just AI generated results.
   
   - https://github.com/apache/datafusion-python/pull/1730
   - https://github.com/apache/datafusion-python/pull/1732
   - https://github.com/apache/datafusion-python/pull/1736
   
   I am tempted to just close all 3. If any of the 3 of you can suggest why one should be taken over another, I'd be interested to hear your thoughts. These mostly appear to be a single output of dumping the issue into an agent.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org