[I] Pyarrow seems to drop logical_type and converted_type in pyarrow.parquet.write_metadata leading to RuntimeError: AppendRowGroups requires equal schemas. error [arrow]

["cpaulik (via GitHub)" <gi...@apache.org>]

cpaulik opened a new issue, #44525:
URL: https://github.com/apache/arrow/issues/44525

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   I'm trying to write a `_metadata` file from a set of existing parquet files written by duckdb.
   
   This is the minimal example I could come up with that reproduces the issue
   
   ```python
   import duckdb
   import pyarrow.parquet as pq
   
   fname = 'parquet_file.parquet'
   con = duckdb.connect()
   
   con.sql(("CREATE TABLE t1 (num INTEGER);"
           "INSERT INTO t1 VALUES (1);"
           "INSERT INTO t1 VALUES (2);"))
   con.sql(f"COPY t1 TO '{fname}' (FORMAT PARQUET)")
   
   
   metadata = pq.read_metadata(fname)
   schema = pq.read_schema(fname)
   metadata_path = "_metadata"
   
   pq.write_metadata(
       schema, metadata_path,
       metadata_collector=[metadata, metadata])
   ```
   
   leads to 
   
   ```
   Traceback (most recent call last):
     File "/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/pyarrow_error.py", line 17, in <module>
       pq.write_metadata(
     File "/Users/christoph/debugging/2024-09-27-parquet-speed/parquet-tests/.venv/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 2247, in write_metadata
       metadata.append_row_groups(m)
     File "pyarrow/_parquet.pyx", line 1047, in pyarrow._parquet.FileMetaData.append_row_groups
   RuntimeError: AppendRowGroups requires equal schemas.
   The two columns with index 0 differ.
   column descriptor = {
     name: num,
     path: num,
     physical_type: INT32,
     converted_type: NONE,
     logical_type: None,
     max_definition_level: 1,
     max_repetition_level: 0,
   }
   column descriptor = {
     name: num,
     path: num,
     physical_type: INT32,
     converted_type: INT_32,
     logical_type: Int(bitWidth=32, isSigned=true),
     max_definition_level: 1,
     max_repetition_level: 0,
   }
   ```
   
   Package versions
   
   ```
   "duckdb==1.1.2",
   "pyarrow>=17.0.0"
   ```
   
   ### Component(s)
   
   Parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python][Parquet] Pyarrow seems to drop logical_type and converted_type in pyarrow.parquet.write_metadata leading to RuntimeError: AppendRowGroups requires equal schemas. error [arrow]

["mapleFU (via GitHub)" <gi...@apache.org>]

mapleFU commented on issue #44525:
URL: https://github.com/apache/arrow/issues/44525#issuecomment-2436823746

   ```
     physical_type: INT32,
     converted_type: NONE,
     logical_type: None,
   ```
   
   This is possible when the type is just int32 🤔


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org