You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Paulo Roberto Cerioni (Jira)" <ji...@apache.org> on 2019/09/05 21:39:00 UTC

[jira] [Updated] (ARROW-6469) PyArrow HDFS documentation does not mention HDFS short circuit readings

     [ https://issues.apache.org/jira/browse/ARROW-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paulo Roberto Cerioni updated ARROW-6469:
-----------------------------------------
    Description: 
Due to PyArrow using libhdfs underneath, it is expected that files read from HDFS are going to make use of short circuit readings.

However, the PyArrow documentation does not explain whether this feature is supported (and on what situations) and if that works without any configuration.

For instance, I'm interested in the use case in which we make use of short circuit feature to read some of the columns from a Parquet file located in HDFS into a dataframe.

 

  was:
Due to PyArrow using libhdfs underneath, it is expected that files reading from HDFS are going to make use of short circuit readings.

However, the PyArrow documentation does not explain whether this feature is supported (and on what situations) and if that works without any configuration.

For instance, I'm interested in the use case in which we make use of short circuit feature to read some of the columns from a Parquet file located in HDFS into a dataframe.

 


> PyArrow HDFS documentation does not mention HDFS short circuit readings
> -----------------------------------------------------------------------
>
>                 Key: ARROW-6469
>                 URL: https://issues.apache.org/jira/browse/ARROW-6469
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Paulo Roberto Cerioni
>            Priority: Minor
>              Labels: documentation
>
> Due to PyArrow using libhdfs underneath, it is expected that files read from HDFS are going to make use of short circuit readings.
> However, the PyArrow documentation does not explain whether this feature is supported (and on what situations) and if that works without any configuration.
> For instance, I'm interested in the use case in which we make use of short circuit feature to read some of the columns from a Parquet file located in HDFS into a dataframe.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)