You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2019/09/11 16:28:01 UTC
[jira] [Commented] (AIRFLOW-2697) Drop snakebite in favour of
pyarrow
[ https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927765#comment-16927765 ]
ASF GitHub Bot commented on AIRFLOW-2697:
-----------------------------------------
stale[bot] commented on pull request #3560: [AIRFLOW-2697] Drop snakebite in favour of hdfs3
URL: https://github.com/apache/airflow/pull/3560
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Drop snakebite in favour of pyarrow
> -----------------------------------
>
> Key: AIRFLOW-2697
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2697
> Project: Apache Airflow
> Issue Type: Improvement
> Affects Versions: 1.9.0
> Reporter: Julian de Ruiter
> Assignee: Julian de Ruiter
> Priority: Blocker
> Fix For: 2.0.0
>
>
> The current HdfsHook relies on the snakebite library, which is unfortunately not compatible with Python 3. To add Python 3 support for the HdfsHook requires switching to a different library for interacting with HDFS. The hdfs3 library is an attractive alternative, as it supports Python 3 and seems to be stable and relatively well supported.
> Update: hdfs3 doesn't get any updates anymore. The best library right now seems to be pyarrow: https://arrow.apache.org/docs/python/filesystems.html
> Therefore I would like to upgrade to pyarrow instead of hdfs3.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)