You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by vijayendra bhati <ve...@yahoo.com> on 2016/04/13 13:38:25 UTC
Ignite Data Grid with write through mode to HDFS with layer of IGFS
Hi,
Can some body please provide me any pointers regarding how I can use Ignite Data Grid/ In Memory caching with write through/write behind mode and writing to HDFS ?
I know Ignite provides IGFS but its different from what I am looking for.
The other way could be I can use IGFS as my In Memory store but is it the right approach ?
Regards,Vijayendra Bhati
Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Vij,
Yes, cache store is normally used when you want to cache as much data from
it as possible. In your case you have two separate data processing modes -
one for hot data with possible caching, and another one for historical
data.
It seems to me that you could just put Ignite cache "near" operational data
storage and cache required data on demand, rather than implementing a real
Ignite store.
Vladimir.
On Thu, Apr 14, 2016 at 9:29 PM, vijayendra bhati <ve...@yahoo.com>
wrote:
> Hi Vladimir,
> Not really, we do not want to store historical data in cache or may be
> cache it for few hours and then evict it.But if recent data is missing in
> cache then yes we want to cache it.So it would require some custom caching
> logic to decide which data to cache.
> So seems like storing historical data in persistent storage seems
> reasonable.The only thing is I have to use ignite cache as side cache
> rather than a write through cache to HDFS, because i dont think it would be
> nice idea to store individual key value pair in cache.May be can think
> about HBase bt not directly to HDFS.
>
> Regards,
> Vij
>
> Sent from Yahoo Mail on Android
> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>
> On Thu, 14 Apr, 2016 at 7:26 pm, Vladimir Ozerov
> <vo...@gridgain.com> wrote:
> Hi Vij,
>
> Storing hot recent data in cache, and historical data in persistent store
> sounds like a perfectly reasonable idea.
>
> If you decide to store historical data in HDFS, then you should be able to
> construct HDFS path from the key because store interface accepts keys to
> store/load data. If this is possible, then I do not see any obvious
> problems with this approach.
>
> On the other hand, do you want this historical data to be cached on access?
>
> Vladimir.
>
> On Thu, Apr 14, 2016 at 3:17 PM, vijayendra bhati <ve...@yahoo.com>
> wrote:
>
>> Thanks Vladimir !!
>>
>> The drawback with using HDFS as persistent store behind Ignite cache is
>> how we will take care of appending single key value pair to HDFS
>> file.Ideally we should use some NoSQL store or RDBMS as persistent back up
>> behind Ignite cache and then run some scheduled batch to transfer the data
>> to HDFS as it happens in normal Lambda Architecture.
>>
>> Now question comes why we want to use Ignite Cache ? Answer is it gives
>> SQL interface that means we can query on any attribute on the fly.Other we
>> could have used any other NoSQL. But NoSQL data model is entirely based
>> upon query pattern so to bring the flexibility at the time of query we
>> think Ignite cache would be better.
>>
>> For our use case we want to put the latest 2 week data in Ignite cache to
>> meet the latency requirements and then for any back date get the data from
>> backend persistent storage, for which we are thinking about HDFS.Thats why
>> we were thinking if we can make Ignite cache write through cache with HDFS
>> as backed up persistent storage it would serve the purpose.
>>
>> Please let me know whats your view on this.
>>
>> Many thanks,
>> Vij
>>
>>
>> On Wednesday, April 13, 2016 8:58 PM, Vladimir Ozerov <
>> vozerov@gridgain.com> wrote:
>>
>>
>> Vij,
>>
>> No, it doesn't. IGFS serves the very different purpose - it is
>> Hadoop-compatible file system. It means that, for example, you can load
>> data to IGFS and then query it using Hive. But native Ignite SQL is not
>> applicable here.
>>
>> Vladimir.
>>
>> On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhati <veejayendraa@yahoo.com
>> > wrote:
>>
>> Thanks Vladimir,
>>
>> I have not gone through complete documentation but if you could let me
>> know does IGFS provide SQL support like Ignite cache does ?
>>
>> Regards,
>> Vij
>>
>>
>> On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <
>> vozerov@gridgain.com> wrote:
>>
>>
>> Hi Vijayendra,
>>
>> IGFS is designed to be a distributed file system which could cache data
>> from Hadoop file systems. It cannot be used as cache store by design.
>> Ignite doesn't have store implementation for HDFS, so you should
>> implement your own if needed. Particularly, you should implement
>> org.apache.ignite.cache.store.CacheStore interface.
>>
>> Vladimir.
>>
>> On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <veejayendraa@yahoo.com
>> > wrote:
>>
>> Hi,
>>
>> Can some body please provide me any pointers regarding how I can use
>> Ignite Data Grid/ In Memory caching with write through/write behind mode
>> and writing to HDFS ?
>>
>> I know Ignite provides IGFS but its different from what I am looking for.
>>
>> The other way could be I can use IGFS as my In Memory store but is it the
>> right approach ?
>>
>> Regards,
>> Vijayendra Bhati
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: Ignite Data Grid with write through mode to HDFS with layer of
IGFS
Posted by vijayendra bhati <ve...@yahoo.com>.
Hi Vladimir,Not really, we do not want to store historical data in cache or may be cache it for few hours and then evict it.But if recent data is missing in cache then yes we want to cache it.So it would require some custom caching logic to decide which data to cache.So seems like storing historical data in persistent storage seems reasonable.The only thing is I have to use ignite cache as side cache rather than a write through cache to HDFS, because i dont think it would be nice idea to store individual key value pair in cache.May be can think about HBase bt not directly to HDFS.
Regards,Vij
Sent from Yahoo Mail on Android
On Thu, 14 Apr, 2016 at 7:26 pm, Vladimir Ozerov<vo...@gridgain.com> wrote: Hi Vij,
Storing hot recent data in cache, and historical data in persistent store sounds like a perfectly reasonable idea.
If you decide to store historical data in HDFS, then you should be able to construct HDFS path from the key because store interface accepts keys to store/load data. If this is possible, then I do not see any obvious problems with this approach.
On the other hand, do you want this historical data to be cached on access?
Vladimir.
On Thu, Apr 14, 2016 at 3:17 PM, vijayendra bhati <ve...@yahoo.com> wrote:
Thanks Vladimir !!
The drawback with using HDFS as persistent store behind Ignite cache is how we will take care of appending single key value pair to HDFS file.Ideally we should use some NoSQL store or RDBMS as persistent back up behind Ignite cache and then run some scheduled batch to transfer the data to HDFS as it happens in normal Lambda Architecture.
Now question comes why we want to use Ignite Cache ? Answer is it gives SQL interface that means we can query on any attribute on the fly.Other we could have used any other NoSQL. But NoSQL data model is entirely based upon query pattern so to bring the flexibility at the time of query we think Ignite cache would be better.
For our use case we want to put the latest 2 week data in Ignite cache to meet the latency requirements and then for any back date get the data from backend persistent storage, for which we are thinking about HDFS.Thats why we were thinking if we can make Ignite cache write through cache with HDFS as backed up persistent storage it would serve the purpose.
Please let me know whats your view on this.
Many thanks,Vij
On Wednesday, April 13, 2016 8:58 PM, Vladimir Ozerov <vo...@gridgain.com> wrote:
Vij,
No, it doesn't. IGFS serves the very different purpose - it is Hadoop-compatible file system. It means that, for example, you can load data to IGFS and then query it using Hive. But native Ignite SQL is not applicable here.
Vladimir.
On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhati <ve...@yahoo.com> wrote:
Thanks Vladimir,
I have not gone through complete documentation but if you could let me know does IGFS provide SQL support like Ignite cache does ?
Regards,Vij
On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <vo...@gridgain.com> wrote:
Hi Vijayendra,
IGFS is designed to be a distributed file system which could cache data from Hadoop file systems. It cannot be used as cache store by design.Ignite doesn't have store implementation for HDFS, so you should implement your own if needed. Particularly, you should implement org.apache.ignite.cache.store.CacheStore interface.
Vladimir.
On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <ve...@yahoo.com> wrote:
Hi,
Can some body please provide me any pointers regarding how I can use Ignite Data Grid/ In Memory caching with write through/write behind mode and writing to HDFS ?
I know Ignite provides IGFS but its different from what I am looking for.
The other way could be I can use IGFS as my In Memory store but is it the right approach ?
Regards,Vijayendra Bhati
Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Vij,
Storing hot recent data in cache, and historical data in persistent store
sounds like a perfectly reasonable idea.
If you decide to store historical data in HDFS, then you should be able to
construct HDFS path from the key because store interface accepts keys to
store/load data. If this is possible, then I do not see any obvious
problems with this approach.
On the other hand, do you want this historical data to be cached on access?
Vladimir.
On Thu, Apr 14, 2016 at 3:17 PM, vijayendra bhati <ve...@yahoo.com>
wrote:
> Thanks Vladimir !!
>
> The drawback with using HDFS as persistent store behind Ignite cache is
> how we will take care of appending single key value pair to HDFS
> file.Ideally we should use some NoSQL store or RDBMS as persistent back up
> behind Ignite cache and then run some scheduled batch to transfer the data
> to HDFS as it happens in normal Lambda Architecture.
>
> Now question comes why we want to use Ignite Cache ? Answer is it gives
> SQL interface that means we can query on any attribute on the fly.Other we
> could have used any other NoSQL. But NoSQL data model is entirely based
> upon query pattern so to bring the flexibility at the time of query we
> think Ignite cache would be better.
>
> For our use case we want to put the latest 2 week data in Ignite cache to
> meet the latency requirements and then for any back date get the data from
> backend persistent storage, for which we are thinking about HDFS.Thats why
> we were thinking if we can make Ignite cache write through cache with HDFS
> as backed up persistent storage it would serve the purpose.
>
> Please let me know whats your view on this.
>
> Many thanks,
> Vij
>
>
> On Wednesday, April 13, 2016 8:58 PM, Vladimir Ozerov <
> vozerov@gridgain.com> wrote:
>
>
> Vij,
>
> No, it doesn't. IGFS serves the very different purpose - it is
> Hadoop-compatible file system. It means that, for example, you can load
> data to IGFS and then query it using Hive. But native Ignite SQL is not
> applicable here.
>
> Vladimir.
>
> On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhati <ve...@yahoo.com>
> wrote:
>
> Thanks Vladimir,
>
> I have not gone through complete documentation but if you could let me
> know does IGFS provide SQL support like Ignite cache does ?
>
> Regards,
> Vij
>
>
> On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <
> vozerov@gridgain.com> wrote:
>
>
> Hi Vijayendra,
>
> IGFS is designed to be a distributed file system which could cache data
> from Hadoop file systems. It cannot be used as cache store by design.
> Ignite doesn't have store implementation for HDFS, so you should implement
> your own if needed. Particularly, you should implement
> org.apache.ignite.cache.store.CacheStore interface.
>
> Vladimir.
>
> On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <ve...@yahoo.com>
> wrote:
>
> Hi,
>
> Can some body please provide me any pointers regarding how I can use
> Ignite Data Grid/ In Memory caching with write through/write behind mode
> and writing to HDFS ?
>
> I know Ignite provides IGFS but its different from what I am looking for.
>
> The other way could be I can use IGFS as my In Memory store but is it the
> right approach ?
>
> Regards,
> Vijayendra Bhati
>
>
>
>
>
>
>
>
Re: Ignite Data Grid with write through mode to HDFS with layer of
IGFS
Posted by vijayendra bhati <ve...@yahoo.com>.
Thanks Vladimir !!
The drawback with using HDFS as persistent store behind Ignite cache is how we will take care of appending single key value pair to HDFS file.Ideally we should use some NoSQL store or RDBMS as persistent back up behind Ignite cache and then run some scheduled batch to transfer the data to HDFS as it happens in normal Lambda Architecture.
Now question comes why we want to use Ignite Cache ? Answer is it gives SQL interface that means we can query on any attribute on the fly.Other we could have used any other NoSQL. But NoSQL data model is entirely based upon query pattern so to bring the flexibility at the time of query we think Ignite cache would be better.
For our use case we want to put the latest 2 week data in Ignite cache to meet the latency requirements and then for any back date get the data from backend persistent storage, for which we are thinking about HDFS.Thats why we were thinking if we can make Ignite cache write through cache with HDFS as backed up persistent storage it would serve the purpose.
Please let me know whats your view on this.
Many thanks,Vij
On Wednesday, April 13, 2016 8:58 PM, Vladimir Ozerov <vo...@gridgain.com> wrote:
Vij,
No, it doesn't. IGFS serves the very different purpose - it is Hadoop-compatible file system. It means that, for example, you can load data to IGFS and then query it using Hive. But native Ignite SQL is not applicable here.
Vladimir.
On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhati <ve...@yahoo.com> wrote:
Thanks Vladimir,
I have not gone through complete documentation but if you could let me know does IGFS provide SQL support like Ignite cache does ?
Regards,Vij
On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <vo...@gridgain.com> wrote:
Hi Vijayendra,
IGFS is designed to be a distributed file system which could cache data from Hadoop file systems. It cannot be used as cache store by design.Ignite doesn't have store implementation for HDFS, so you should implement your own if needed. Particularly, you should implement org.apache.ignite.cache.store.CacheStore interface.
Vladimir.
On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <ve...@yahoo.com> wrote:
Hi,
Can some body please provide me any pointers regarding how I can use Ignite Data Grid/ In Memory caching with write through/write behind mode and writing to HDFS ?
I know Ignite provides IGFS but its different from what I am looking for.
The other way could be I can use IGFS as my In Memory store but is it the right approach ?
Regards,Vijayendra Bhati
Re: Ignite Data Grid with write through mode to HDFS with layer of
IGFS
Posted by tomk <rr...@gmail.com>.
Could you tell more about it ? How to do it ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Data-Grid-with-write-through-mode-to-HDFS-with-layer-of-IGFS-tp4122p4171.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Vij,
No, it doesn't. IGFS serves the very different purpose - it is
Hadoop-compatible file system. It means that, for example, you can load
data to IGFS and then query it using Hive. But native Ignite SQL is not
applicable here.
Vladimir.
On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhati <ve...@yahoo.com>
wrote:
> Thanks Vladimir,
>
> I have not gone through complete documentation but if you could let me
> know does IGFS provide SQL support like Ignite cache does ?
>
> Regards,
> Vij
>
>
> On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <
> vozerov@gridgain.com> wrote:
>
>
> Hi Vijayendra,
>
> IGFS is designed to be a distributed file system which could cache data
> from Hadoop file systems. It cannot be used as cache store by design.
> Ignite doesn't have store implementation for HDFS, so you should implement
> your own if needed. Particularly, you should implement
> org.apache.ignite.cache.store.CacheStore interface.
>
> Vladimir.
>
> On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <ve...@yahoo.com>
> wrote:
>
> Hi,
>
> Can some body please provide me any pointers regarding how I can use
> Ignite Data Grid/ In Memory caching with write through/write behind mode
> and writing to HDFS ?
>
> I know Ignite provides IGFS but its different from what I am looking for.
>
> The other way could be I can use IGFS as my In Memory store but is it the
> right approach ?
>
> Regards,
> Vijayendra Bhati
>
>
>
>
>
Re: Ignite Data Grid with write through mode to HDFS with layer of
IGFS
Posted by vijayendra bhati <ve...@yahoo.com>.
Thanks Vladimir,
I have not gone through complete documentation but if you could let me know does IGFS provide SQL support like Ignite cache does ?
Regards,Vij
On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <vo...@gridgain.com> wrote:
Hi Vijayendra,
IGFS is designed to be a distributed file system which could cache data from Hadoop file systems. It cannot be used as cache store by design.Ignite doesn't have store implementation for HDFS, so you should implement your own if needed. Particularly, you should implement org.apache.ignite.cache.store.CacheStore interface.
Vladimir.
On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <ve...@yahoo.com> wrote:
Hi,
Can some body please provide me any pointers regarding how I can use Ignite Data Grid/ In Memory caching with write through/write behind mode and writing to HDFS ?
I know Ignite provides IGFS but its different from what I am looking for.
The other way could be I can use IGFS as my In Memory store but is it the right approach ?
Regards,Vijayendra Bhati
Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Vijayendra,
IGFS is designed to be a distributed file system which could cache data
from Hadoop file systems. It cannot be used as cache store by design.
Ignite doesn't have store implementation for HDFS, so you should implement
your own if needed. Particularly, you should implement
org.apache.ignite.cache.store.CacheStore interface.
Vladimir.
On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <ve...@yahoo.com>
wrote:
> Hi,
>
> Can some body please provide me any pointers regarding how I can use
> Ignite Data Grid/ In Memory caching with write through/write behind mode
> and writing to HDFS ?
>
> I know Ignite provides IGFS but its different from what I am looking for.
>
> The other way could be I can use IGFS as my In Memory store but is it the
> right approach ?
>
> Regards,
> Vijayendra Bhati
>