[GitHub] [arrow] lidavidm commented on issue #13535: Arrow python client read and write, did not output the expected metadata

[GitBox <gi...@apache.org>]

lidavidm commented on issue #13535:
URL: https://github.com/apache/arrow/issues/13535#issuecomment-1177580038

   The metadata is actually part of the schema:
   
      ```
       metadata : dict or Mapping, default None
           Optional metadata for the schema (if schema not passed).
      ```
   
   So when the writer is created with an explicit schema, the metadata is dropped.
   
   For example, this works:
   
   ```python
   import tempfile
   import pyarrow as pa
   print("PyArrow version:", pa.__version__)
   
   table = pa.table([[1, 2, 3, 4]], names=["ints"], metadata={"foo": "bar"})
   print("Before")
   print(table.to_string(show_metadata=True))
   print()
   
   with tempfile.NamedTemporaryFile() as path:
       with pa.ipc.new_file(path, schema=table.schema) as sink:
           sink.write_table(table)
   
       with pa.ipc.open_file(path) as reader:
           table2 = reader.read_all()
           print("After")
           print(table2.to_string(show_metadata=True))
   ```
   
   ```
   PyArrow version: 8.0.0
   Before
   pyarrow.Table
   ints: int64
   -- schema metadata --
   foo: 'bar'
   
   After
   pyarrow.Table
   ints: int64
   -- schema metadata --
   foo: 'bar'
   ```
   
   But if you give it a schema without the metadata, you lose the data:
   
   ```python
   import tempfile
   import pyarrow as pa
   print("PyArrow version:", pa.__version__)
   
   table = pa.table([[1, 2, 3, 4]], names=["ints"], metadata={"foo": "bar"})
   print("Before")
   print(table.to_string(show_metadata=True))
   print()
   
   with tempfile.NamedTemporaryFile() as path:
       # Explicit schema here!
       with pa.ipc.new_file(path, schema=pa.schema([("ints", pa.int64())])) as sink:
           sink.write_table(table)
   
       with pa.ipc.open_file(path) as reader:
           table2 = reader.read_all()
           print("After")
           print(table2.to_string(show_metadata=True))
   ```
   
   ```
   PyArrow version: 8.0.0
   Before
   pyarrow.Table
   ints: int64
   -- schema metadata --
   foo: 'bar'
   
   After
   pyarrow.Table
   ints: int64
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org