You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/07 05:05:25 UTC
[GitHub] [arrow] liusitan opened a new issue, #13535: Arrow python client read and write, did not output the expected metadata
liusitan opened a new issue, #13535:
URL: https://github.com/apache/arrow/issues/13535
This is the code for reading.
preader.py
```python
import pyarrow as pa
# f = '/tmp/fib.arrow'
f = 'arraydata.arrow'
with pa.ipc.open_file(f) as reader:
t = reader.read_all()
print(t.to_string(show_metadata=True, preview_cols=2))
This is the code for writing
pwriter.py
```
```python
import numpy as np
import pyarrow as pa
arr = pa.array(np.arange(8))
schema = pa.schema([
pa.field('nums', arr.type)
])
n_legs = pa.array([2, 4, 5, 100])
t = pa.Table.from_arrays([n_legs], names=["nums"],metadata = {"loc":"san diego"})
with pa.OSFile('arraydata.arrow', 'wb') as sink:
with pa.ipc.new_file(sink, schema=schema) as writer:
# batch = pa.record_batch([arr], schema=schema)
writer.write_table(t)
```
weirdly, when I run the python writer first, and reader second, it does not print out the metadata stored, which is {"loc":"san diego"}
I am using macOS, if it matters:>
<img width="498" alt="image" src="https://user-images.githubusercontent.com/20542539/177695553-f070a949-5656-4fbf-801c-2fae39397bee.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liusitan closed issue #13535: Arrow python client read and write, did not output the expected metadata
Posted by GitBox <gi...@apache.org>.
liusitan closed issue #13535: Arrow python client read and write, did not output the expected metadata
URL: https://github.com/apache/arrow/issues/13535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on issue #13535: Arrow python client read and write, did not output the expected metadata
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #13535:
URL: https://github.com/apache/arrow/issues/13535#issuecomment-1177580038
The metadata is actually part of the schema:
```
metadata : dict or Mapping, default None
Optional metadata for the schema (if schema not passed).
```
So when the writer is created with an explicit schema, the metadata is dropped.
For example, this works:
```python
import tempfile
import pyarrow as pa
print("PyArrow version:", pa.__version__)
table = pa.table([[1, 2, 3, 4]], names=["ints"], metadata={"foo": "bar"})
print("Before")
print(table.to_string(show_metadata=True))
print()
with tempfile.NamedTemporaryFile() as path:
with pa.ipc.new_file(path, schema=table.schema) as sink:
sink.write_table(table)
with pa.ipc.open_file(path) as reader:
table2 = reader.read_all()
print("After")
print(table2.to_string(show_metadata=True))
```
```
PyArrow version: 8.0.0
Before
pyarrow.Table
ints: int64
-- schema metadata --
foo: 'bar'
After
pyarrow.Table
ints: int64
-- schema metadata --
foo: 'bar'
```
But if you give it a schema without the metadata, you lose the data:
```python
import tempfile
import pyarrow as pa
print("PyArrow version:", pa.__version__)
table = pa.table([[1, 2, 3, 4]], names=["ints"], metadata={"foo": "bar"})
print("Before")
print(table.to_string(show_metadata=True))
print()
with tempfile.NamedTemporaryFile() as path:
# Explicit schema here!
with pa.ipc.new_file(path, schema=pa.schema([("ints", pa.int64())])) as sink:
sink.write_table(table)
with pa.ipc.open_file(path) as reader:
table2 = reader.read_all()
print("After")
print(table2.to_string(show_metadata=True))
```
```
PyArrow version: 8.0.0
Before
pyarrow.Table
ints: int64
-- schema metadata --
foo: 'bar'
After
pyarrow.Table
ints: int64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liusitan commented on issue #13535: Arrow python client read and write, did not output the expected metadata
Posted by GitBox <gi...@apache.org>.
liusitan commented on issue #13535:
URL: https://github.com/apache/arrow/issues/13535#issuecomment-1177778455
hi David, thanks for your response, problems solved !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liusitan commented on issue #13535: Arrow python client read and write, did not output the expected metadata
Posted by GitBox <gi...@apache.org>.
liusitan commented on issue #13535:
URL: https://github.com/apache/arrow/issues/13535#issuecomment-1177778740
hi David, thanks for your response, problems solved !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org