IGFS - Python and R clients

[Sergey Semichev <ss...@gmail.com>]

My application needs to read / write data to a distributed memory FS.
The application is not Map/Reduce or Spark job. 
Let's say I need to run a docker orchestration pipeline, where each step is
a docker container, and I want to save the intermediate results of each step
and pass it to the next step. A docker container may use Java, Python, R or
C++ code to execute the logic. 

Is it possible to use Python and R to r /w directly to IGFS in primary mode
or in dual_sync mode?
Technically I can use for example ZeroMQ to implement IPC communication
between Java and Python and use IGFS Native API, but it will introduce
additional latency.

If I run IGFS as Hadoop accelerator on top of HDFS, can I use python HDFS
interfaces?
http://wesmckinney.com/blog/python-hdfs-interfaces/

Thanks



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Python-and-R-clients-tp12361.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: IGFS - Python and R clients

[nsemichev <ns...@gmail.com>]

I configured HDFS Client on Ubuntu 16.04 and I can successfully run this
command:
hdfs --config /etc/hadoop/conf/ dfs -ls /
The config parameter takes the file core-site.xml from etc/hadoop/conf/
 
From core-site.xml
 
<property>
    <name>fs.default.name</name>
    <value>igfs://igfs@10.200.10.1:10500</value>
</property>
 
Then I tried using Python hdfs3 library which uses libhdfs3.
 
>>> from hdfs3 import HDFileSystem
>>> hdfs = HDFileSystem(host='10.200.10.1', port=10500)
 
When I run this code, I get the following error message:	
ConnectionError: Connection Failed: HdfsRpcException: Failed to invoke RPC
call "getFsStats" on server "10.200.10.1:10500"

I opened an issue here: https://github.com/dask/hdfs3/issues/123

It seems like it should work; has anyone tried using hdfs3 Python library or
some other Python libraries to work with Ignite?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Python-and-R-clients-tp12361p14400.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: IGFS - Python and R clients

["Ivan V." <iv...@gridgain.com>]

As far as I understand, https://wiki.apache.org/hadoop/LibHDFS is generic
enough, and should work with any org.apache.hadoop.fs.FileSystem
implementation (IGFS , HDFS, or another). May it be an option for you?

On Wed, May 3, 2017 at 12:51 PM, dkarachentsev <dk...@gridgain.com>
wrote:

> Hi Sergey,
>
> To use IGFS from non-JVM languages you may configure it as FS for Hadoop in
> IgfsMode.PRIMARY mode. After that you can configure connectivity from any
> supported language to Hadoop and it's file system.
>
> Thanks!
> -Dmitry
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/IGFS-Python-and-R-clients-tp12361p12381.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: IGFS - Python and R clients

[dkarachentsev <dk...@gridgain.com>]

Hi Sergey,

To use IGFS from non-JVM languages you may configure it as FS for Hadoop in
IgfsMode.PRIMARY mode. After that you can configure connectivity from any
supported language to Hadoop and it's file system.

Thanks!
-Dmitry



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Python-and-R-clients-tp12361p12381.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.