Re: [I] pyarrow.fs.FileSystem.from_uri does not refer to HDFS core-site.xml config file when resolving namenode for HDFS URL [arrow]

["pstrzelczak (via GitHub)" <gi...@apache.org>]

pstrzelczak commented on issue #42050:
URL: https://github.com/apache/arrow/issues/42050#issuecomment-3443745587

   @pitrou @kszucs @westonpace @kou could you check if this can be fixed easily?
   
   With this issue, using some pyarrow API requires either opening the file upfront and passing file like object (e.g. read_csv or open_csv) or passing `filesystem` parameter explicitly:
   
   ```
   >>> import pyarrow.parquet as pq
   >>> pq.read_metadata('hdfs:///foo.parquet')
   log4j:WARN No appenders could be found for logger (org.apache.hadoop.fs.FileSystem).
   log4j:WARN Please initialize the log4j system properly.
   log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
   hdfsBuilderConnect(forceNewInstance=1, nn=hdfs://, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
   URISyntaxException: Expected authority at index 7: hdfs://java.lang.IllegalArgumentException: Expected authority at index 7: hdfs://
   	at java.base/java.net.URI.create(URI.java:906)
   Caused by: java.net.URISyntaxException: Expected authority at index 7: hdfs://
   	at java.base/java.net.URI$Parser.fail(URI.java:2976)
   	at java.base/java.net.URI$Parser.failExpecting(URI.java:2982)
   	at java.base/java.net.URI$Parser.parseHierarchical(URI.java:3226)
   	at java.base/java.net.URI$Parser.parse(URI.java:3177)
   	at java.base/java.net.URI.<init>(URI.java:623)
   	at java.base/java.net.URI.create(URI.java:904)
   /arrow/cpp/src/arrow/status.cc:168: Failed to disconnect hdfs client: IOError: HDFS hdfsFS::Disconnect failed. Detail: [errno 9] Bad file descriptor
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/usr/local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 2341, in read_metadata
       filesystem, where = _resolve_filesystem_and_path(where, filesystem)
     File "/usr/local/lib/python3.10/site-packages/pyarrow/fs.py", line 174, in _resolve_filesystem_and_path
       filesystem, path = FileSystem.from_uri(path)
     File "pyarrow/_fs.pyx", line 502, in pyarrow._fs.FileSystem.from_uri
     File "pyarrow/_fs.pyx", line 457, in pyarrow._fs.FileSystem._native_from_uri
     File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
   OSError: HDFS connection failed
   >>> import fsspec
   >>> pq.read_metadata('hdfs:///foo.parquet', filesystem=fsspec.filesystem('hdfs'))
   <pyarrow._parquet.FileMetaData object at 0x798066b93bf0>
     created_by: parquet-cpp-arrow version 21.0.0
     num_columns: 6
     num_rows: 150
     num_row_groups: 1
     format_version: 2.6
     serialized_size: 1418
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org