You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Pranay Tonpay <pt...@gmail.com> on 2017/05/18 16:14:30 UTC
Few questions
Hi,
I have setup igfs as a caching layer on top of a HA configured hadoop
cluster.. few questions ~
1) if I put data on hdfs directly, and then read it via ignite, the data
would get cached in ignite and any subsequent fetches wud fetch it from
igfs and not go to hdfs at all... Correct?
2) if after doing step 1 , I explicitly delete data from hdfs , i saw that
data gets deleted from igfs too.. is that expected behavior? I think, in a
couple of instances I was able to see it in igfs, but am not seeing that
happening anymore... Not sure what I am missing
3) simply putting data on ignite cache does not put data in hdfs.. it wud
remain in igfs cache... To have it persist to hdfs too, we should be
writing to igfs with secondary file system and using filesystemfactory to
cachinghadoopfilesystemfactory.
Pls ignore typos as I am typing over phone....
Regards
Pranay
Re: Few questions
Posted by Ivan Veselovsky <iv...@gridgain.com>.
Hi, Paranay,
The logic of why IGFS does not have implemented CacheStore is that it aims
different goals.
Please see https://apacheignite-fs.readme.io/docs/in-memory-file-system for
general IGFS concepts: it has both "native" and Hadoop API , both these are
file-system like APIs. As a persistent layer the underlying secondary file
system is used.
So, IGFS is designed with the "in-memory file system" being the central
concept, distributed cache is just its implementation.
IGFS is implemented using 2 caches , and the mapping between files and K-V
pairs in caches is not trivial.
In contrast, CacheStore is used to load and persist caches , when (1) cache
is central concept of the task, and (2) when K-V pairs more-or-less
directly map to the underlying storage (e.g. as in case of
org.apache.ignite.internal.processors.cache.H2CacheStoreStrategy.H2CacheStore).
Of course, you can add a CacheStore to the IGFS caches, but the question
arises, would an underlying secondary file system be better and cheaper
persistent layer?
p.s. HDFS is not read-only, it is read-write file system.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Few-questions-tp13004p13070.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Few questions
Posted by Pranay Tonpay <pt...@gmail.com>.
Hi Ivan,
Sorry , I am using my phone to type as emails are blocked in the
organization..
Example ~
IgniteCache cache = ignite.getorcreatecache("my cache")
Cache.put(key, value) ...
For the above, I am assuming there is no write thru cache implementation to
write to Hdfs ... One wud have to implement the cachestore interface and
have the logic written... There r implementations for rdbms but not for
Hdfs.. I understand the reason why it may not be there due to immutability
of Hdfs, but wanted to confirm on this...
On May 19, 2017 10:16 AM, "Ivan V." <iv...@gridgain.com> wrote:
Hi, Pranay,
to be honest, I'm not sure I 100% understand the question.
Yes, IGFS internally implemented by 2 caches (meta & data), but these 2
caches do not assume any direct modifications, that is, modifications done
otherwise than through IGFS API.
On Fri, May 19, 2017 at 7:12 PM, Pranay Tonpay <pt...@gmail.com> wrote:
> Thx Ivan.. appreciate your help...
>
> Regarding point 3, I was specifically referring to the IgniteCache and not
> Igfs. Sorry if my jargons are a bit incorrect in the context, but per my
> understanding, ignite has ~
> 1) in memory data grid (ignite cache)
> 2) ignite file system
>
> 2 sits on top of 1 (rather makes use of 1) ...
> Doing anything on igfs (2) does reflect in hdfs as expected, but
> manipulating cache (key value pairs), which is (1) ,simply remain in
> memory, unless I implement the CacheStore interface for hdfs and implement
> load, write etc ..
> Is that correct interpretation?
>
> Pls opine...
>
>
> On May 19, 2017 5:20 AM, "Ivan Veselovsky" <iv...@gridgain.com>
> wrote:
>
> 1) Yes, unless some data have been evicted from Ignite cache.
> 2) Sure, this is expected: IGFS in dual modes always tries to reflect the
> underlying file system contents.
> 3) This sounds strange. In DUAL_ASYNC mode it is possible that changes made
> to IGFS appear in underlying file system with some delays, but they should
> appear eventually. In DUAL_SYNC mode the changes should appear immediately
> upon operation completion.
>
>
>
> --
> View this message in context: http://apache-ignite-users.705
> 18.x6.nabble.com/Few-questions-tp13004p13027.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
Re: Few questions
Posted by "Ivan V." <iv...@gridgain.com>.
Hi, Pranay,
to be honest, I'm not sure I 100% understand the question.
Yes, IGFS internally implemented by 2 caches (meta & data), but these 2
caches do not assume any direct modifications, that is, modifications done
otherwise than through IGFS API.
On Fri, May 19, 2017 at 7:12 PM, Pranay Tonpay <pt...@gmail.com> wrote:
> Thx Ivan.. appreciate your help...
>
> Regarding point 3, I was specifically referring to the IgniteCache and not
> Igfs. Sorry if my jargons are a bit incorrect in the context, but per my
> understanding, ignite has ~
> 1) in memory data grid (ignite cache)
> 2) ignite file system
>
> 2 sits on top of 1 (rather makes use of 1) ...
> Doing anything on igfs (2) does reflect in hdfs as expected, but
> manipulating cache (key value pairs), which is (1) ,simply remain in
> memory, unless I implement the CacheStore interface for hdfs and implement
> load, write etc ..
> Is that correct interpretation?
>
> Pls opine...
>
>
> On May 19, 2017 5:20 AM, "Ivan Veselovsky" <iv...@gridgain.com>
> wrote:
>
> 1) Yes, unless some data have been evicted from Ignite cache.
> 2) Sure, this is expected: IGFS in dual modes always tries to reflect the
> underlying file system contents.
> 3) This sounds strange. In DUAL_ASYNC mode it is possible that changes made
> to IGFS appear in underlying file system with some delays, but they should
> appear eventually. In DUAL_SYNC mode the changes should appear immediately
> upon operation completion.
>
>
>
> --
> View this message in context: http://apache-ignite-users.705
> 18.x6.nabble.com/Few-questions-tp13004p13027.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
Re: Few questions
Posted by Pranay Tonpay <pt...@gmail.com>.
Thx Ivan.. appreciate your help...
Regarding point 3, I was specifically referring to the IgniteCache and not
Igfs. Sorry if my jargons are a bit incorrect in the context, but per my
understanding, ignite has ~
1) in memory data grid (ignite cache)
2) ignite file system
2 sits on top of 1 (rather makes use of 1) ...
Doing anything on igfs (2) does reflect in hdfs as expected, but
manipulating cache (key value pairs), which is (1) ,simply remain in
memory, unless I implement the CacheStore interface for hdfs and implement
load, write etc ..
Is that correct interpretation?
Pls opine...
On May 19, 2017 5:20 AM, "Ivan Veselovsky" <iv...@gridgain.com>
wrote:
1) Yes, unless some data have been evicted from Ignite cache.
2) Sure, this is expected: IGFS in dual modes always tries to reflect the
underlying file system contents.
3) This sounds strange. In DUAL_ASYNC mode it is possible that changes made
to IGFS appear in underlying file system with some delays, but they should
appear eventually. In DUAL_SYNC mode the changes should appear immediately
upon operation completion.
--
View this message in context: http://apache-ignite-users.
70518.x6.nabble.com/Few-questions-tp13004p13027.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Few questions
Posted by Ivan Veselovsky <iv...@gridgain.com>.
1) Yes, unless some data have been evicted from Ignite cache.
2) Sure, this is expected: IGFS in dual modes always tries to reflect the
underlying file system contents.
3) This sounds strange. In DUAL_ASYNC mode it is possible that changes made
to IGFS appear in underlying file system with some delays, but they should
appear eventually. In DUAL_SYNC mode the changes should appear immediately
upon operation completion.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Few-questions-tp13004p13027.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.