You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "luocan17 (via GitHub)" <gi...@apache.org> on 2024/02/20 06:36:26 UTC

[I] PyArrowFileIO.parse_location(...) return error path for hdfs location. [iceberg-python]

luocan17 opened a new issue, #449:
URL: https://github.com/apache/iceberg-python/issues/449

   ### Apache Iceberg version
   
   None
   
   ### Please describe the bug 🐞
   
   `parse_location` method in PyArrowFileIO return error `path` as follow.
   
   ![image](https://github.com/apache/iceberg-python/assets/20294680/ecfad2c4-e4ca-48ff-a4c0-d6feec0dd899)
   
   ![image](https://github.com/apache/iceberg-python/assets/20294680/ce9549e5-c833-446f-b3eb-aa244b2fa31a)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `PyArrowFileIO.parse_location` return error `path` for hdfs location. [iceberg-python]

Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #449:
URL: https://github.com/apache/iceberg-python/issues/449#issuecomment-2083068661

   Hi @luocan17 thank you for raising this issue. I have a PR up to attempt a fix. Could I ask for your review on: https://github.com/apache/iceberg-python/pull/668


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `PyArrowFileIO.parse_location` return error `path` for hdfs location. [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #449: `PyArrowFileIO.parse_location` return error `path` for hdfs location.
URL: https://github.com/apache/iceberg-python/issues/449


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `PyArrowFileIO.parse_location` return error `path` for hdfs location. [iceberg-python]

Posted by "j7nhai (via GitHub)" <gi...@apache.org>.
j7nhai commented on issue #449:
URL: https://github.com/apache/iceberg-python/issues/449#issuecomment-1966417221

   how to fix this bug? I change 'location' to 'uri.path' but still get some errors.
   (unable to get root cause for java.lang.NoClassDefFoundError)
   (unable to get stack trace for java.lang.NoClassDefFoundError)
   hdfsBuilderConnect(forceNewInstance=1, nn=hdfs://xxxxxx, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
   (unable to get root cause for java.lang.NoClassDefFoundError)
   (unable to get stack trace for java.lang.NoClassDefFoundError)
   /arrow/cpp/src/arrow/status.cc:155: Failed to disconnect hdfs client: IOError: HDFS hdfsFS::Disconnect failed. Detail: [errno 9] Bad file descriptor
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/home/j7nhai/third-party/iceberg-python/pyiceberg/catalog/hive.py", line 312, in create_table
       self._write_metadata(metadata, io, metadata_location)
     File "/home/j7nhai/third-party/iceberg-python/pyiceberg/catalog/__init__.py", line 651, in _write_metadata
       ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
     File "/home/j7nhai/third-party/iceberg-python/pyiceberg/io/pyarrow.py", line 422, in new_output
       fs=self.fs_by_scheme(scheme, netloc),
     File "/home/j7nhai/third-party/iceberg-python/pyiceberg/io/pyarrow.py", line 361, in _initialize_fs
       return HadoopFileSystem.from_uri(f"{scheme}://{netloc}")
     File "pyarrow/_hdfs.pyx", line 134, in pyarrow._hdfs.HadoopFileSystem.from_uri
     File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
   OSError: HDFS connection failed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `PyArrowFileIO.parse_location` return error `path` for hdfs location. [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #449:
URL: https://github.com/apache/iceberg-python/issues/449#issuecomment-2085249291

   Fixed both by @syun64 and @frankliee 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org