You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by nsemichev <ns...@gmail.com> on 2017/07/10 15:32:41 UTC
Using Python library to run HDFS with Ignite
I configured HDFS Client on Ubuntu 16.04 and I can successfully run this
command:
hdfs --config /etc/hadoop/conf/ dfs -ls /
The config parameter takes the file core-site.xml from etc/hadoop/conf/
From core-site.xml
<property>
<name>fs.default.name</name>
<value>igfs://igfs@10.200.10.1:10500</value>
</property>
Then I tried using Python hdfs3 library which uses libhdfs3.
>>> from hdfs3 import HDFileSystem
>>> hdfs = HDFileSystem(host='10.200.10.1', port=10500)
When I run this code, I get the following error message:
ConnectionError: Connection Failed: HdfsRpcException: Failed to invoke RPC
call "getFsStats" on server "10.200.10.1:10500"
I opened an issue here: https://github.com/dask/hdfs3/issues/123
It seems like it should work; has anyone tried using hdfs3 Python library or
some other Python libraries to work with Ignite?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Using-Python-library-to-run-HDFS-with-Ignite-tp14595.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Using Python library to run HDFS with Ignite
Posted by Mikhail Cherkasov <mc...@gridgain.com>.
Hi,
Did you specify:
<property>
<name>fs.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
</property>
in core-site.xml? may be hadoop find this fs implementation automatically
in lib dir, I'm note sure about this.
Any way Hadoop uses special implementation of file
system(IgniteHadoopFileSystem) to work with IGFS,
so I don't think that we have this implementation for python.
Thanks,
Mikhail.
On Mon, Jul 10, 2017 at 6:32 PM, nsemichev <ns...@gmail.com> wrote:
> I configured HDFS Client on Ubuntu 16.04 and I can successfully run this
> command:
> hdfs --config /etc/hadoop/conf/ dfs -ls /
> The config parameter takes the file core-site.xml from etc/hadoop/conf/
>
> From core-site.xml
>
> <property>
> <name>fs.default.name</name>
> <value>igfs://igfs@10.200.10.1:10500</value>
> </property>
>
> Then I tried using Python hdfs3 library which uses libhdfs3.
>
> >>> from hdfs3 import HDFileSystem
> >>> hdfs = HDFileSystem(host='10.200.10.1', port=10500)
>
> When I run this code, I get the following error message:
> ConnectionError: Connection Failed: HdfsRpcException: Failed to invoke RPC
> call "getFsStats" on server "10.200.10.1:10500"
>
> I opened an issue here: https://github.com/dask/hdfs3/issues/123
>
> It seems like it should work; has anyone tried using hdfs3 Python library
> or
> some other Python libraries to work with Ignite?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Using-Python-library-to-run-HDFS-
> with-Ignite-tp14595.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Thanks,
Mikhail.