You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/02 12:03:28 UTC

Re: [PR] [python] Add array predicates to PredicateBuilder [paimon]

JingsongLi commented on PR #10175:
URL: https://github.com/apache/paimon/pull/10175#issuecomment-5951915740

   Reviewed head `6f4b683234542e086714d2e3c15218f356334d7f`. The read-plan array filtering use case has end-to-end value, and the earlier non-ARRAY-field validation finding is fixed. I found one additional numeric-array correctness issue.
   
   **[P2] Compare array elements using the declared element type** (`predicate.py:516,535,553`). The new testers all use Python list membership, but the builder accepts every `ArrayType` and discards the element type when constructing the predicate. For a persisted `ARRAY<FLOAT>` written from `[0.1]`, `array_contains("floats", 0.1)` returns no rows: the stored float32 becomes Python `0.10000000149011612`, while the literal remains a double-precision Python float. `arrays_overlap` and `array_contains_all` have the same false negative. There are also direct `ARRAY<DOUBLE>` differences from Java's element comparator: a NaN literal misses a stored NaN, and either zero literal matches both `-0.0` and `+0.0`, whereas Java `Float/Double.compareTo` matches NaN and distinguishes signed zero.
   
   I reproduced these through actual table writes, commits, reloads and reads in Parquet, Avro, ORC and row formats, projecting only the ID column. Please retain the element type, normalize literals accordingly, and use the same typed comparison contract as Java; alternatively reject unsupported element types explicitly. Add numeric-array tests for all three methods. This finding is scoped to the newly accepted array-predicate paths, not the existing scalar Python predicates.
   
   Validation: 86 Python array/predicate/read/projection tests passed; 22 Java PredicateBuilder tests passed on JDK 8 with normal Maven checks. Across all four formats, 24 additional string-array controls passed for exact row IDs, projection, compound OR, empty literal lists and nulls; the numeric mismatches above reproduced in every format. Configured flake8, Python 3.6 grammar and diff checks passed; current-head CI is green. The no-file-pruning limitation is now accurately described in the PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org