You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jcklie-amazon (via GitHub)" <gi...@apache.org> on 2025/07/26 08:09:35 UTC

[I] [Python][Parquet] Extending the schema and writing it back does not update PySpark metadata [arrow]

jcklie-amazon opened a new issue, #47201:
URL: https://github.com/apache/arrow/issues/47201

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I have a use case where I need to process files that have been originally written by PySpark with pyarrow and write them back so that they can again be read by PySpark. During the processing, I need to extend the schema. 
   
   When extending the schema and writing it with `flavour=pyspark` , then reading with PySpark does not show the added columns. Using other Parquet tools like pandas, polars or random online viewers, the data is there. 
   
   My guess for what is happening after some debugging is that PySpark uses not the parquet schema itself if there is metadata called `org.apache.spark.sql.parquet.row.metadata` that pyarrow does not update. When I call `replace_schema_metadata()` before writing the table in PyArrow, then PySpark can see the new column. I would expect writing with pyarrow using the pyspark flavor would handle this.
   
   Code to reproduce:
   
   `pip install pyarrow pyspark`
   
   ```python
   from pyspark.sql import SparkSession
   import pyarrow as pa
   import pyarrow.parquet as pq
   
   spark = SparkSession.builder.getOrCreate()
   df = spark.createDataFrame(
       [
           (1, "foo"),
           (2, "bar"),
       ],
       ["id", "label"]  # add your column names here
   )
   df.write.mode("overwrite").parquet("table.parquet")
   table = pq.read_table("table.parquet")
   
   result = []
   for batch in table.to_batches():
       d = batch.to_pydict()
       d["value"] = d["label"] * 2
   
   schema = table.schema.append(pa.field("value", pa.string()))
   enriched_table = table.from_pylist(result, schema=schema)
   pq.write_table(enriched_table, 'enriched_table.parquet',  flavor="spark")
   
   print("# Pyarrow reading enriched")
   print()
   enriched_table2 = pq.read_table('enriched_table.parquet')
   print(enriched_table2.schema)
   print(enriched_table2.schema.metadata)
   
   print("# Pyspark reading enriched")
   print()
   df = spark.read.parquet('enriched_table.parquet')
   df.printSchema()
   print(df.select("value"))
   ```
   
   Output:
   
   ```
   # Pyarrow reading enriched
   
   id: int64
   label: string
   value: string
   -- schema metadata --
   org.apache.spark.version: '3.5.5'
   org.apache.spark.sql.parquet.row.metadata: '{"type":"struct","fields":[{"' + 120
   {b'org.apache.spark.version': b'3.5.5', b'org.apache.spark.sql.parquet.row.metadata': b'{"type":"struct","fields":[{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"label","type":"string","nullable":true,"metadata":{}}]}'}
   # Pyspark reading enriched
   
   root
    |-- id: long (nullable = true)
    |-- label: string (nullable = true)
   
   Traceback (most recent call last):
   
   pyspark.errors.exceptions.captured.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `value` cannot be resolved. Did you mean one of the following? [`id`, `label`].;
   'Project ['value]
   +- Relation [id#6L,label#7] parquet
   
   Process finished with exit code 1
   ```
   
   This use case might look a bit convoluted, but in the whole context of the app, it makes sense. 
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][Parquet] Extending the schema and writing it back does not update Spark schema metadata [arrow]

Posted by "rmnskb (via GitHub)" <gi...@apache.org>.
rmnskb commented on issue #47201:
URL: https://github.com/apache/arrow/issues/47201#issuecomment-3146443182

   Hey @jcklie-amazon, thanks for raising the issue, I will investigate it, and will come back with the conclusions. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python][Parquet] Extending the schema and writing it back does not update Spark schema metadata [arrow]

Posted by "rmnskb (via GitHub)" <gi...@apache.org>.
rmnskb commented on issue #47201:
URL: https://github.com/apache/arrow/issues/47201#issuecomment-3146442559

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org