You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2019/03/20 03:03:00 UTC
[jira] [Resolved] (ARROW-4802) [Python] Hadoop classpath discovery
broken HADOOP_HOME is a symlink
[ https://issues.apache.org/jira/browse/ARROW-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney resolved ARROW-4802.
---------------------------------
Resolution: Fixed
Issue resolved by pull request 3983
[https://github.com/apache/arrow/pull/3983]
> [Python] Hadoop classpath discovery broken HADOOP_HOME is a symlink
> -------------------------------------------------------------------
>
> Key: ARROW-4802
> URL: https://issues.apache.org/jira/browse/ARROW-4802
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Micah Kornfield
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.13.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> From [https://github.com/apache/arrow/issues/3748]:
> CLASSPATH discovery was recently changed in [{{d911850}}|https://github.com/apache/arrow/commit/d91185000945cec96abad41a230d05d3cdefff93] to resolve ARROW-2113 and ARROW-3768.
> Specifically, the logic used to find all jars under HADOOP_HOME uses the find command directly
> [arrow/python/pyarrow/hdfs.py|https://github.com/apache/arrow/blob/d91185000945cec96abad41a230d05d3cdefff93/python/pyarrow/hdfs.py#L144]
> Line 144 in [d911850|https://github.com/apache/arrow/commit/d91185000945cec96abad41a230d05d3cdefff93]
> | |find_args = ('find', os.environ['HADOOP_HOME'], '-name', '*.jar')|
> This will not work when HADOOP_HOME is a symlink, in which case '-L' needs to be passed to the find command.
> CLASSPATH can still be set explicitly, but this is a change in behavior as HADOOP_HOME symlinks worked without issue before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)