You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "frankliee (via GitHub)" <gi...@apache.org> on 2023/09/08 02:01:40 UTC
[GitHub] [iceberg] frankliee opened a new pull request, #8524: Python: Fix pyarrow hdfs support
frankliee opened a new pull request, #8524:
URL: https://github.com/apache/iceberg/pull/8524
I have tested pyiceberg on our hdfs cluster, and this pr fixed some bugs.
Changed.
1. Using optional parameters to create hdfs client, similar to gcs's style.
2. Fix `PyArrowFileIO.parse_location` for hdfs style, which cannot be resolved by HadoopFileSystem.
Previous behavior： parse_location("hdfs://127.0.0.1:9000/root/abc") -> "hdfs", "127.0.0.1:9000/root/abc"
New behavior: parse_location("hdfs://127.0.0.1:9000/root/abc") -> "hdfs", "hdfs://127.0.0.1:9000/root/abc"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #8524: Python: Fix pyarrow hdfs support
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8524:
URL: https://github.com/apache/iceberg/pull/8524#discussion_r1320656534
##########
python/tests/io/test_pyarrow.py:
##########
@@ -1527,3 +1527,19 @@ def test_writing_avro_file_gcs(generated_manifest_entry_file: str, pyarrow_filei
assert b1 == b2 # Check that bytes of read from local avro file match bytes written to s3
pyarrow_fileio_gcs.delete(f"gs://warehouse/{filename}")
+
+
+def test_parse_hdfs_location() -> None:
Review Comment:
Ah, I see. The other arrow file systems require the scheme to be removed, but for hdfs you can access both local files and on hdfs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #8524: Python: Fix pyarrow hdfs support
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #8524:
URL: https://github.com/apache/iceberg/pull/8524#issuecomment-1712647216
Thanks for fixing this @frankliee 👍🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko merged pull request #8524: Python: Fix pyarrow hdfs support
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #8524:
URL: https://github.com/apache/iceberg/pull/8524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org