You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Pranay Tonpay <pt...@gmail.com> on 2017/05/16 06:09:57 UTC
HDFS IGFS Integration
hi ,
I am new to Ignite and have been exploring the usage of ignite as a caching
layer with HDFS filesystem underneath -
There was a mention that there was no need of a namenode using IGFS as it
can determine the file data locality using the hashing function.
Does it mean that Namenode can be avoided when IGFS get deployed on top of
HDFS ? If that's true, can someone please give me an example where such a
configuration is done ?
Or is it that when one reads from IGFS ( for mapreduce or only data ),
there is nothing like a namenode .... but for IGFS to talk to HDFS,
namenode is still needed.
Example.... assume that data is in HDFS initially.... Now when we read the
data, it comes in IGFS and gets cached ( but in this case namenode was
used, the regular behaviour )... But now, when we access the same data, it
can be accessed from IGFS directly and there is no need of a namenode.
thanks for your help -
thx
pranay
Re: HDFS IGFS Integration
Posted by Pranay Tonpay <pt...@gmail.com>.
Thx Ivan, that helps
On May 16, 2017 10:24 AM, "Ivan Veselovsky" <iv...@gridgain.com>
wrote:
Hi, Pranay,
> Does it mean that Namenode can be avoided when IGFS get deployed on top of
> HDFS ?
No. IGFS itself does not have namenode, it is a distributed cache storing
file blocks. But when deployed on top of HDFS, it fetches the underlying
data using ordinary namenode mechanism.
> Or is it that when one reads from IGFS (for mapreduce or only data ),
> there is nothing like a namenode .... but for IGFS to talk to HDFS,
> namenode is still needed.
Yes.
> But now, when we access the same data, it can be accessed from IGFS
> directly and there is no need of a namenode.
Yes, if the requested data were fully cached, it will be given to the client
w/o namenode access.
--
View this message in context: http://apache-ignite-users.
70518.x6.nabble.com/HDFS-IGFS-Integration-tp12874p12891.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: HDFS IGFS Integration
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Hi, Pranay,
> Does it mean that Namenode can be avoided when IGFS get deployed on top of
> HDFS ?
No. IGFS itself does not have namenode, it is a distributed cache storing
file blocks. But when deployed on top of HDFS, it fetches the underlying
data using ordinary namenode mechanism.
> Or is it that when one reads from IGFS (for mapreduce or only data ),
> there is nothing like a namenode .... but for IGFS to talk to HDFS,
> namenode is still needed.
Yes.
> But now, when we access the same data, it can be accessed from IGFS
> directly and there is no need of a namenode.
Yes, if the requested data were fully cached, it will be given to the client
w/o namenode access.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDFS-IGFS-Integration-tp12874p12891.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.