You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Mohit-25-tech (via GitHub)" <gi...@apache.org> on 2026/10/10 19:12:28 UTC
[I] [Python][Dataset] Mixed FileInfo and path sources can crash Python in dataset() [arrow]
Mohit-25-tech opened a new issue, #53033:
URL: https://github.com/apache/arrow/issues/53033
### Describe the bug, including details regarding any error messages, version, and platform.
### Describe the bug
`pyarrow.dataset.dataset()` behaves inconsistently when its source list contains both filesystem paths and `pyarrow.fs.FileInfo` objects.
Both homogeneous input types work correctly. However, mixed inputs behave differently depending on their order:
- `[path, FileInfo]` raises a normal `TypeError`.
- `[FileInfo, path]` terminates the Python process.
The crash occurs instead of a safe Python exception.
### Reproduction
```python
import pathlib
import subprocess
import sys
import tempfile
import textwrap
import pyarrow as pa
import pyarrow.parquet as pq
child_code = textwrap.dedent("""
import sys
import pyarrow.dataset as ds
import pyarrow.fs as fs
path, case = sys.argv[1:3]
info = fs.LocalFileSystem().get_file_info(path)
sources = {
"paths_only": [path, path],
"infos_only": [info, info],
"path_first": [path, info],
"info_first": [info, path],
}[case]
print("Starting:", case, flush=True)
try:
result = ds.dataset(
sources, format="parquet"
).to_table().to_pydict()
print("Result:", result, flush=True)
except Exception as exc:
print("Exception:", type(exc).__name__, str(exc), flush=True)
""")
print("Python:", sys.version.split()[0])
print("PyArrow:", pa.__version__)
print("Platform:", sys.platform)
with tempfile.TemporaryDirectory() as directory:
path = pathlib.Path(directory) / "tiny.parquet"
pq.write_table(pa.table({"x": [1]}), path)
for case in ("paths_only", "infos_only", "path_first", "info_first"):
run = subprocess.run(
[sys.executable, "-X", "faulthandler", "-c",
child_code, str(path), case],
capture_output=True,
text=True,
timeout=30,
)
print(case, "returncode:", run.returncode)
print("stdout:", run.stdout.strip())
print("stderr:", run.stderr.strip())
```
### Actual behavior
Observed with PyArrow 23.0.1 on Kaggle/Linux:
```text
paths_only: returncode 0
infos_only: returncode 0
path_first: TypeError: not a path-like object
info_first: returncode -11
Fatal Python error: Segmentation fault
```
The same four-case assessment was repeated with PyArrow 26.0.0, where both homogeneous cases succeeded, path-first raised `TypeError`, and FileInfo-first terminated the subprocess.
The behavior was also reproduced locally on Windows with PyArrow 16.1.0 and 24.0.0.
### Expected behavior
Mixing file paths and `FileInfo` objects should either:
1. Be supported consistently in both orders; or
2. Raise a clear Python exception in both orders.
Invalid or unsupported source combinations should not terminate the interpreter.
### Possible root cause
In `python/pyarrow/dataset.py`, the public `dataset()` function accepts lists containing either paths or `FileInfo` objects, but the downstream `_filesystem_dataset()` logic dispatches based on the first element.
When the first element is `FileInfo`, the native `FileSystemDatasetFactory` receives a mixed list. Its Cython path assumes the remaining entries are also `FileInfo` objects, which appears to cause the native crash.
A possible minimal fix is to reject mixed lists safely before entering native code. Alternatively, mixed inputs could be normalized if supporting them is the intended API contract.
I'd appreciate maintainer guidance on the preferred behavior.
### Environment
- Kaggle/Linux, Python 3.13.15, PyArrow 23.0.1
- Kaggle/Linux, PyArrow 26.0.0
- Windows, PyArrow 16.1.0 and 24.0.0
The investigation and initial report drafting were AI-assisted. I independently ran and reviewed the Kaggle reproductions.
### Component(s)
Python, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Parquet][Python][Dataset] Mixed FileInfo and path sources can crash Python in dataset() [arrow]
Posted by "Mohit-25-tech (via GitHub)" <gi...@apache.org>.
Mohit-25-tech commented on issue #53033:
URL: https://github.com/apache/arrow/issues/53033#issuecomment-6101496272
Hi maintainers,
I've submitted a PR addressing this issue.
**Root cause:** The dataset construction logic handles mixed `FileInfo` and path inputs based on the first element's type. This can pass a mixed list into the Cython factory, where an unchecked `FileInfo` cast causes a segmentation fault.
**Fix implemented:**
- Normalize path entries in mixed lists to `FileInfo` objects.
- Preserve existing `FileInfo` metadata, source ordering, and duplicates.
- Keep homogeneous input behavior unchanged.
- Add defensive Cython type validation to prevent unsafe casts.
- Add regression tests covering mixed inputs, tuples, duplicates, subtree filesystems, and invalid paths.
**Verification:**
- Independently reproduced the original segmentation fault using PyArrow 26.0.0 on Kaggle/Linux.
- Verified that all four input combinations work using the patched Python wrapper with PyArrow 26.0.0.
- 25 focused local checks passed.
- Flake8, Python syntax, autopep8, and `git diff --check` passed.
The modified Cython extension has not yet been compiled locally, so full integration validation remains pending in CI.
**PR:** #53034
I'd appreciate a review of the implementation and would be happy to make any necessary adjustments.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org