Re: [I] ArrowScan to_table fails if the data is mixed between dict-encoded strings and plain strings. [iceberg-python]

["hzmsh (via GitHub)" <gi...@apache.org>]

hzmsh commented on issue #3260:
URL: https://github.com/apache/iceberg-python/issues/3260#issuecomment-6198820440

   This still reproduces on `main` (b32392dc) and 0.12.0. A common way to hit it is a pandas `Categorical` column, followed by a normal append:
   
   ```python
   import pandas as pd, pyarrow as pa
   
   # table has a string column `status`
   tbl.append(pa.Table.from_pandas(pd.DataFrame({"id": [1, 2], "status": pd.Categorical(["new", "done"])}), preserve_index=False))
   tbl.append(pa.Table.from_pandas(pd.DataFrame({"id": [3], "status": ["new"]}), preserve_index=False))
   
   tbl.scan().to_arrow()   # ArrowTypeError: Unable to merge: Field status has incompatible types: large_string vs dictionary<values=string, indices=int8, ...>
   tbl.scan().to_pandas()  # same
   tbl.scan().to_polars()  # same
   ```
   
   `to_arrow_batch_reader()` works because it casts each batch to the table schema, but `to_table` concatenates the per-file tables as they are.
   
   @Fokko, would you be open to a fix in `ArrowScan.to_table` that decodes dictionary columns to plain strings before concatenating, unless the column is listed in `dictionary_columns`? This matches `to_arrow_batch_reader()`, and Iceberg has no dictionary type, so `string` is what the schema says. It is a small change (about 10 lines plus tests).
   
   One behavior change to flag: a table where every file was written from categorical data currently comes back as `category` in `to_pandas()`. With this fix it would come back as a plain string dtype unless the column is passed in `dictionary_columns`.
   
   The same error came up in #2250, where you asked what outcome users expect; the answer there was the type from the table schema, which is what this fix returns. Three earlier PRs for this (#3444, #3667, #3849) were closed as stale without review. If this direction works for you, I'm happy to open a PR and credit those authors.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org