You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "deep826 (via GitHub)" <gi...@apache.org> on 2024/05/22 12:52:08 UTC
[I] pyarrow.fs.HadoopFileSystem Usage Problems [arrow]
deep826 opened a new issue, #41777:
URL: https://github.com/apache/arrow/issues/41777
### Describe the usage question you have. Please include as many useful details as possible.
hi, I use pyarrow.fs.HadoopFileSystem client to interact with hdfs. And I write some bytes to a file in hdfs, then download it to local filesystem. I read the file using python native api: read, but the result is false. When I use pyarrow hdfs client to read the file in hdfs, the result is right. I'm confused..... here are some sudo code snippet.
`a = 1000 `
`b = 64`
`with hdfs_client.open_output_stream(path) as f:`
` f.write(a.to_bytes(8, sys.byteorder))`
` f.write(b.to_bytes(4, sys.byteorder))`
here, i write 12 bytes to file: path, then i download it from hdfs to local_path and read these bytes as follows:
`with open(local_path, 'rb') as f:`
` bs = f.read(12)`
` a = int.from_bytes(bs[0:8], sys.byteorder)`
` b = int.from_bytes(bs[8:12], sys.byteorder)`
` print(f"a: {a}, b: {b}")`
The print result is: a: 559903, b: 3158573824, the expected values should be: a: 1000, b: 64.
So What's the problem.....
### Component(s)
C, C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow.fs.HadoopFileSystem Usage Problems [arrow]
Posted by "deep826 (via GitHub)" <gi...@apache.org>.
deep826 commented on issue #41777:
URL: https://github.com/apache/arrow/issues/41777#issuecomment-2126339498
> I would try to only print the bytes and inspect that as what I see in the example can give unexpected output (different byte order in hdfs, int bit width, etc.).
I also printed the bytes of two reading ways.
1. reading file with python native api:
`with open(local_path, 'rb') as f:`
` bs = f.read(12)`
` print(f"a in bytes: {bs[0:8]}")`
` print(f"b in bytes: {bs[8:12]}")`
the result is:
`a in bytes: b'\x1f\x8b\x08\x00\x00\x00\x00\x00'`
`b in bytes: b'\x00\x03D\xbc'`
2. reading file with pyarrow hdfs client:
`with hdfs_client.open_input_stream(hdfs_path) as f:`
` bs = f.read(12)`
` print(f"a in bytes: {bs[0:8]}")`
` print(f"b in bytes: {bs[8:12]}")`
the result is:
`a in bytes: b'\xe8\x03\x00\x00\x00\x00\x00\x00'`
`b in bytes: b'@\x00\x00\x00'`
the result of the second way is right.
byte order in my machine is 'little endian', but I use 'big endian' to parse those bytes, the result is alse wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow.fs.HadoopFileSystem Usage Problems [arrow]
Posted by "deep826 (via GitHub)" <gi...@apache.org>.
deep826 closed issue #41777: pyarrow.fs.HadoopFileSystem Usage Problems
URL: https://github.com/apache/arrow/issues/41777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] pyarrow.fs.HadoopFileSystem Usage Problems [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #41777:
URL: https://github.com/apache/arrow/issues/41777#issuecomment-2125085506
I would try to only print the bytes and inspect that as what I see in the example can give unexpected output (different byte order in hdfs, int bit width, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org