IGFS Usage Question

[Chris Parker <ch...@imperva.com>]

Hello,

I am new to ignite and am having some difficulty in understanding the limitations of IGFS.  Currently, I have a project that uses Spark and parquet storing to the local file system.  What I would like to do is to use IGFS as a cache for the files that are being referenced.  I had assumed that because the doc indicates that IGFS implements a java file system, that I should be able to simply add the jars to the classpath for an existing application and then use the igfs:// scheme instead of file:// for the scheme and not have to modify the code to use any specific ignite interfaces.  Is this incorrect?  I ask because I am getting an error indicating that the provider cannot be found for igfs.

If this is not possible, is there a way that I should be configuring ignite with Spark to effectively do the same thing?  I see instructions for integration with HDFS but we are not using HDFS at this time.  My goal is to avoiding introducing new code to an existing application, but get the performance boost due to the memory cache.

Thanks in advance

RE: IGFS Usage Question

[vkulichenko <va...@gmail.com>]

If you're using java.io.File, then no, it's not possible. You will have to
write the code that starts Ignite, creates IgniteFileSystem, etc.

At the same time, all IGFS streams extend standard Java streams, so you
should be able to reuse most of you code that read and writes to files.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Usage-Question-tp11584p11632.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RE: IGFS Usage Question

[Chris Parker <ch...@imperva.com>]

Hi Val,

Thanks for this, but as I said initially, I am not using Hadoop at this time.  I simply want to treat IGFS as a separate filesystem and use it to cache a local file.  Can this be done without modification to an existing program?

Chris

-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com] 
Sent: Thursday, March 30, 2017 7:42 PM
To: user@ignite.apache.org
Subject: Re: IGFS Usage Question

Hi Chris,

Ignite provides IGFS based implementation of Hadoop file system abstraction.
So you can use run you Hadoop applications without changes and connect them directly to in-memory FS. Did you go through installation instructions [1]?

[1] https://apacheignite-fs.readme.io/docs/installing-on-apache-hadoop

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Usage-Question-tp11584p11591.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: IGFS Usage Question

[vkulichenko <va...@gmail.com>]

Hi Chris,

Ignite provides IGFS based implementation of Hadoop file system abstraction.
So you can use run you Hadoop applications without changes and connect them
directly to in-memory FS. Did you go through installation instructions [1]?

[1] https://apacheignite-fs.readme.io/docs/installing-on-apache-hadoop

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IGFS-Usage-Question-tp11584p11591.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.