Re: hadoop file system connect problem with pyarrow

[Wes McKinney <we...@gmail.com>]

Do either of these machines have a current Hadoop installation (and is
that installation in the system path)?

On Tue, Oct 20, 2020 at 9:53 AM 황세규 <gl...@naver.com> wrote:
>
> Dear Maintainer. My name is Joseph Hwang in South Korea. I need some advice about PyArrow.
>
> I try to develop Hadoop File System client application with PyArrow. But some errors are thrown from hdfs connect.
> My IDE is Eclipse PyDev and I use 2 OS (Windows 10 64bit and CentOS 8 64bit) but there are some problems on both os.
>
> My python codes are simple.
>
>
> import pyarrow as pa
> fs = pa.hdfs.connect(host='localhost', port=9000)
>
>
> == Errors on Windows 10
>
> Traceback (most recent call last):
>   File "C:\eclipse-workspace\PythonFredProj\com\aaa\fred\hdfs3-test.py", line 14, in <module>
>     fs = pa.hdfs.connect(host='localhost', port=9000)
>   File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 208, in connect
>     fs = HadoopFileSystem(host=host, port=port, user=user,
>   File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 38, in __init__
>     _maybe_set_hadoop_classpath()
>   File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 136, in _maybe_set_hadoop_classpath
>     classpath = _hadoop_classpath_glob(hadoop_bin)
>   File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 163, in _hadoop_classpath_glob
>     return subprocess.check_output(hadoop_classpath_args)
>   File "C:\Python-3.8.3-x64\lib\subprocess.py", line 411, in check_output
>     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
>   File "C:\Python-3.8.3-x64\lib\subprocess.py", line 489, in run
>     with Popen(*popenargs, **kwargs) as process:
>   File "C:\Python-3.8.3-x64\lib\subprocess.py", line 854, in __init__
>     self._execute_child(args, executable, preexec_fn, close_fds,
>   File "C:\Python-3.8.3-x64\lib\subprocess.py", line 1307, in _execute_child
>     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
> OSError: [WinError 193] %1 is not a valid win32 application
>
> I installed Visual C++ 2015. But the above errors are still thrown.
>
>
> == Errors on CentOS 8
>
> Traceback (most recent call last):
>   File "/home/jhwang/eclipse-workspace/BigDataPythonTest/com/aaa/etl/hdfs3-test.py", line 7, in <module>
>     fs = pa.hdfs.connect(host='localhost', port=9000)
>   File "/usr/python/anaconda3/lib/python3.8/site-packages/pyarrow/hdfs.py", line 208, in connect
>     fs = HadoopFileSystem(host=host, port=port, user=user,
>   File "/usr/python/anaconda3/lib/python3.8/site-packages/pyarrow/hdfs.py", line 40, in __init__
>     self._connect(host, port, user, kerb_ticket, extra_conf)
>   File "pyarrow/io-hdfs.pxi", line 75, in pyarrow.lib.HadoopFileSystem._connect
>   File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
> OSError: Unable to load libhdfs: ./libhdfs.so: cannot open shared object file: No such file or directory
>
> In case of CentOS 8 , python cli does not throw these errors. Only on eclipse IDE. I think my PyDev configuration of Eclipse
> has some problems. kindly inform me how to fix these errors. Thanks in advance.