You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zhongjiajie (JIRA)" <ji...@apache.org> on 2019/07/26 03:38:00 UTC
[jira] [Commented] (AIRFLOW-2697) Drop snakebite in favour of
pyarrow
[ https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893291#comment-16893291 ]
zhongjiajie commented on AIRFLOW-2697:
--------------------------------------
Hi [~jrderuiter], seem you do not wok on this PR anymore. Do you mind I continue woking on you PR?
> Drop snakebite in favour of pyarrow
> -----------------------------------
>
> Key: AIRFLOW-2697
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2697
> Project: Apache Airflow
> Issue Type: Improvement
> Affects Versions: 1.9.0
> Reporter: Julian de Ruiter
> Assignee: Julian de Ruiter
> Priority: Blocker
> Fix For: 2.0.0
>
>
> The current HdfsHook relies on the snakebite library, which is unfortunately not compatible with Python 3. To add Python 3 support for the HdfsHook requires switching to a different library for interacting with HDFS. The hdfs3 library is an attractive alternative, as it supports Python 3 and seems to be stable and relatively well supported.
> Update: hdfs3 doesn't get any updates anymore. The best library right now seems to be pyarrow: https://arrow.apache.org/docs/python/filesystems.html
> Therefore I would like to upgrade to pyarrow instead of hdfs3.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)