You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Mark Farnan <ma...@petrolink.com> on 2017/08/31 13:15:29 UTC
Ignite as Cache and Query layer for Cassandra.
Howdy,
I'm currently evaluating Ignite for a project, and am looking for some
guidance and project sample for using Ignite over Cassandra as a cache and
query layer.
Note: The Cassandra database and schema already exists and is mature.
In a Recent Infoworld post, (
https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html
) it stated
"No remodeling of Cassandra data
Apache Ignite reads from Apache Cassandra and other NoSQL databases, so
moving Cassandra data into Ignite requires no data modification. The data
schema can also be migrated directly into Ignite as is. "
However, I can't find in any documentation, or online articles, guidance
on just how to do this and there are no links in the article.
- The Ignite docs /cassandra section appear to only cover using
Cassandra as a persistence layer, not for going over and caching /
querying an existing schema. (unless I'm missing something obvious !)
- The Youtube video cuts out to blank at the end, so no links are shown.
The links on the slideshare document appear to be broken.
- Can't find any git hub for examples on how to do this (especially not
for the couple you tube vids)
Can anyone please point me to some working sample code and/or
documentation on how to do this ?
I'm especially interested in not having to migrate any schema's, nor
having to pre-load the data into the cache. (lazy load), which the
articles imply.
Also some of the data would be too large to load it all into Ram anyway.
Any assistance greatly appreciated.
Regards
Mark.
Re: Ignite as Cache and Query layer for Cassandra.
Posted by Denis Magda <dm...@gridgain.com>.
Came across this technical gettings started:
https://t.co/KboV0LtPmv?ssr=true
Check it up and share if it helps you more than the existing docs.
Denis
On Thursday, August 31, 2017, Denis Magda <dm...@apache.org> wrote:
> Hello Mark,
>
> The article is not technical, covers only the top of the iceberg and this
> is probably why it confused you a lot.
>
> In short, the conclusion you got
>
> I'm especially interested in not having to migrate any schema's, nor
> having to pre-load the data into the cache. (lazy load), which the
> articles imply.
>
>
> is not 100% correct.
>
> Cassandra is a 3rd party store for Ignite (as any RDBMS or NoSQL database)
> and you should create corresponding Ignite schema and fully pre-load data
> in RAM if you plan using SQL. That’s not required only if Ignite Native
> Persistence is used as a disk store. See the differences here:
> https://apacheignite.readme.io/v2.1/docs/distributed-
> persistent-store#section-ignite-native-vs-3rd-party-persistence
>
> Basically, if to follow the earlier provided Cassandra integration doc you
> will be able to stick Ignite and Cassandra together.
>
> P.S.
> I’ve updated the broken links. Thanks for reporting!
>
> —
> Denis
>
> On Aug 31, 2017, at 12:29 PM, Mark Farnan <mark.farnan@petrolink.com
> <javascript:_e(%7B%7D,'cvml','mark.farnan@petrolink.com');>> wrote:
>
> Hi Ilya,
>
> That documentation I saw, however it seems to describe only using
> Cassandra as a Persistant store for Ignite Caches, i.e.
>
> "Ignite Cassandra module implements persistent store
> <https://apacheignite.readme.io/docs/persistent-store> for Ignite caches
> by utilizing Cassandra <http://cassandra.apache.org/>as a persistent
> storage for expired cache records.
> It functions pretty much the same way like CacheJdbcBlobStore
> <https://apacheignite.readme.io/docs/persistent-store#cachejdbcblobstore>
> and CacheJdbcPojoStore
> <https://apacheignite.readme.io/docs/persistent-store#cachejdbcpojostore>and
> provides such benefits:"
>
> It dosn't seem to describe anywhere, how to set it up over an existing
> large cassandra table structure to act as a cache and SQL query mechanism
> for it. Nor how to deal with the fact you can't have it all in memory and
> how Writes and Reads would be handled etc.
> Also several of the links on the page are broken.
>
> Ideally want to be able to run ad-hoc SQL queries over the data, including
> joins etc. \
>
> I agree there are config examples, AWS etc, but they all seem to me, to
> be for persistance of general cache, BLOB or POJO, not for what the
> InfoWorld article described.
>
> Unless I'm missing something obvious !.
>
>
> Regards
>
> Mark.
>
>
>
>
>
> From: Ilya Kasnacheev <ilya.kasnacheev@gmail.com
> <javascript:_e(%7B%7D,'cvml','ilya.kasnacheev@gmail.com');>>
> To: user@ignite.apache.org
> <javascript:_e(%7B%7D,'cvml','user@ignite.apache.org');>,
> Date: 08/31/2017 06:56 PM
> Subject: Re: Ignite as Cache and Query layer for Cassandra.
> ------------------------------
>
>
>
> Hello Mark,
>
> I have found some documentation on how to do what the article describes:
>
> "Apache Ignite can be inserted between Apache Cassandra and an existing
> application layer with no changes to the Cassandra data and only minimal
> changes to the application"
>
> *https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra*
> <https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra>
>
> On the left you can see there are config examples, tests, AWS
> configuration, etc.
>
> Hope it's good enough.
>
>
> --
> Ilya Kasnacheev
>
> 2017-08-31 16:15 GMT+03:00 Mark Farnan <*mark.farnan@petrolink.com*
> <javascript:_e(%7B%7D,'cvml','mark.farnan@petrolink.com');>>:
> Howdy,
>
> I'm currently evaluating Ignite for a project, and am looking for some
> guidance and project sample for using Ignite over Cassandra as a cache and
> query layer.
> Note: The Cassandra database and schema already exists and is mature.
>
> In a Recent Infoworld post, (
> *https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html*
> <https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html>)
> it stated
>
> "*No remodeling of Cassandra data*
> Apache Ignite reads from Apache Cassandra and other NoSQL databases, so
> moving Cassandra data into Ignite requires no data modification. The data
> schema can also be migrated directly into Ignite as is. "
>
> However, I can't find in *any* documentation, or online articles,
> guidance on just how to do this and there are no links in the article.
> - The Ignite docs /cassandra section appear to only cover using
> Cassandra as a persistence layer, not for going over and caching /
> querying an existing schema. (unless I'm missing something obvious !)
> - The Youtube video cuts out to blank at the end, so no links are shown.
> The links on the slideshare document appear to be broken.
> - Can't find any git hub for examples on how to do this (especially not
> for the couple you tube vids)
>
> Can anyone please point me to some working sample code and/or
> documentation on how to do this ?
>
> I'm especially interested in not having to migrate any schema's, nor
> having to pre-load the data into the cache. (lazy load), which the
> articles imply.
> Also some of the data would be too large to load it all into Ram anyway.
>
>
> Any assistance greatly appreciated.
>
>
> Regards
>
> Mark.
>
>
>
Re: Ignite as Cache and Query layer for Cassandra.
Posted by Denis Magda <dm...@apache.org>.
Hello Mark,
The article is not technical, covers only the top of the iceberg and this is probably why it confused you a lot.
In short, the conclusion you got
> I'm especially interested in not having to migrate any schema's, nor having to pre-load the data into the cache. (lazy load), which the articles imply.
is not 100% correct.
Cassandra is a 3rd party store for Ignite (as any RDBMS or NoSQL database) and you should create corresponding Ignite schema and fully pre-load data in RAM if you plan using SQL. That’s not required only if Ignite Native Persistence is used as a disk store. See the differences here:
https://apacheignite.readme.io/v2.1/docs/distributed-persistent-store#section-ignite-native-vs-3rd-party-persistence
Basically, if to follow the earlier provided Cassandra integration doc you will be able to stick Ignite and Cassandra together.
P.S.
I’ve updated the broken links. Thanks for reporting!
—
Denis
> On Aug 31, 2017, at 12:29 PM, Mark Farnan <ma...@petrolink.com> wrote:
>
> Hi Ilya,
>
> That documentation I saw, however it seems to describe only using Cassandra as a Persistant store for Ignite Caches, i.e.
>
> "Ignite Cassandra module implements persistent store <https://apacheignite.readme.io/docs/persistent-store> for Ignite caches by utilizing Cassandra <http://cassandra.apache.org/>as a persistent storage for expired cache records.
> It functions pretty much the same way like CacheJdbcBlobStore <https://apacheignite.readme.io/docs/persistent-store#cachejdbcblobstore> and CacheJdbcPojoStore <https://apacheignite.readme.io/docs/persistent-store#cachejdbcpojostore>and provides such benefits:"
>
> It dosn't seem to describe anywhere, how to set it up over an existing large cassandra table structure to act as a cache and SQL query mechanism for it. Nor how to deal with the fact you can't have it all in memory and how Writes and Reads would be handled etc.
> Also several of the links on the page are broken.
>
> Ideally want to be able to run ad-hoc SQL queries over the data, including joins etc. \
>
> I agree there are config examples, AWS etc, but they all seem to me, to be for persistance of general cache, BLOB or POJO, not for what the InfoWorld article described.
>
> Unless I'm missing something obvious !.
>
>
> Regards
>
> Mark.
>
>
>
>
>
> From: Ilya Kasnacheev <il...@gmail.com>
> To: user@ignite.apache.org,
> Date: 08/31/2017 06:56 PM
> Subject: Re: Ignite as Cache and Query layer for Cassandra.
>
>
>
> Hello Mark,
>
> I have found some documentation on how to do what the article describes:
>
> "Apache Ignite can be inserted between Apache Cassandra and an existing application layer with no changes to the Cassandra data and only minimal changes to the application"
>
> https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra <https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra>
>
> On the left you can see there are config examples, tests, AWS configuration, etc.
>
> Hope it's good enough.
>
>
> --
> Ilya Kasnacheev
>
> 2017-08-31 16:15 GMT+03:00 Mark Farnan <mark.farnan@petrolink.com <ma...@petrolink.com>>:
> Howdy,
>
> I'm currently evaluating Ignite for a project, and am looking for some guidance and project sample for using Ignite over Cassandra as a cache and query layer.
> Note: The Cassandra database and schema already exists and is mature.
>
> In a Recent Infoworld post, (https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html <https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html>) it stated
>
> "No remodeling of Cassandra data
> Apache Ignite reads from Apache Cassandra and other NoSQL databases, so moving Cassandra data into Ignite requires no data modification. The data schema can also be migrated directly into Ignite as is. "
>
> However, I can't find in any documentation, or online articles, guidance on just how to do this and there are no links in the article.
> - The Ignite docs /cassandra section appear to only cover using Cassandra as a persistence layer, not for going over and caching / querying an existing schema. (unless I'm missing something obvious !)
> - The Youtube video cuts out to blank at the end, so no links are shown. The links on the slideshare document appear to be broken.
> - Can't find any git hub for examples on how to do this (especially not for the couple you tube vids)
>
> Can anyone please point me to some working sample code and/or documentation on how to do this ?
>
> I'm especially interested in not having to migrate any schema's, nor having to pre-load the data into the cache. (lazy load), which the articles imply.
> Also some of the data would be too large to load it all into Ram anyway.
>
>
> Any assistance greatly appreciated.
>
>
> Regards
>
> Mark.
>
Re: Ignite as Cache and Query layer for Cassandra.
Posted by Mark Farnan <ma...@petrolink.com>.
Hi Ilya,
That documentation I saw, however it seems to describe only using
Cassandra as a Persistant store for Ignite Caches, i.e.
"Ignite Cassandra module implements persistent store for Ignite caches by
utilizing Cassandraas a persistent storage for expired cache records.
It functions pretty much the same way like CacheJdbcBlobStore and
CacheJdbcPojoStoreand provides such benefits:"
It dosn't seem to describe anywhere, how to set it up over an existing
large cassandra table structure to act as a cache and SQL query mechanism
for it. Nor how to deal with the fact you can't have it all in memory
and how Writes and Reads would be handled etc.
Also several of the links on the page are broken.
Ideally want to be able to run ad-hoc SQL queries over the data, including
joins etc. \
I agree there are config examples, AWS etc, but they all seem to me, to
be for persistance of general cache, BLOB or POJO, not for what the
InfoWorld article described.
Unless I'm missing something obvious !.
Regards
Mark.
From: Ilya Kasnacheev <il...@gmail.com>
To: user@ignite.apache.org,
Date: 08/31/2017 06:56 PM
Subject: Re: Ignite as Cache and Query layer for Cassandra.
Hello Mark,
I have found some documentation on how to do what the article describes:
"Apache Ignite can be inserted between Apache Cassandra and an existing
application layer with no changes to the Cassandra data and only minimal
changes to the application"
https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra
On the left you can see there are config examples, tests, AWS
configuration, etc.
Hope it's good enough.
--
Ilya Kasnacheev
2017-08-31 16:15 GMT+03:00 Mark Farnan <ma...@petrolink.com>:
Howdy,
I'm currently evaluating Ignite for a project, and am looking for some
guidance and project sample for using Ignite over Cassandra as a cache and
query layer.
Note: The Cassandra database and schema already exists and is mature.
In a Recent Infoworld post, (
https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html
) it stated
"No remodeling of Cassandra data
Apache Ignite reads from Apache Cassandra and other NoSQL databases, so
moving Cassandra data into Ignite requires no data modification. The data
schema can also be migrated directly into Ignite as is. "
However, I can't find in any documentation, or online articles, guidance
on just how to do this and there are no links in the article.
- The Ignite docs /cassandra section appear to only cover using
Cassandra as a persistence layer, not for going over and caching /
querying an existing schema. (unless I'm missing something obvious !)
- The Youtube video cuts out to blank at the end, so no links are shown.
The links on the slideshare document appear to be broken.
- Can't find any git hub for examples on how to do this (especially not
for the couple you tube vids)
Can anyone please point me to some working sample code and/or
documentation on how to do this ?
I'm especially interested in not having to migrate any schema's, nor
having to pre-load the data into the cache. (lazy load), which the
articles imply.
Also some of the data would be too large to load it all into Ram anyway.
Any assistance greatly appreciated.
Regards
Mark.
RE: Ignite as Cache and Query layer for Cassandra.
Posted by "Roger Fischer (CW)" <rf...@Brocade.com>.
Hi Mark,
I recently did a Proof-of-Concept with Ignite and Cassandra. There are a number of challenges you need to look out for:
1) In order to do SQL queries, you need to have all the data in Ignite. It is not practical to include other persistence layers in the query process.
The exception is Ignite Native Persistence.
2) You need to deploy the POJO for the objects (and keys if they are not basic types) in Ignite. This may become unnecessary in a near-future release.
3) Preloading data is not particularly efficient. Each Ignite node sends the preload query. Each Ignite node ignores the data it is not primary or backup for. It may be possible to optimize this for the case where the partitioning is the same in Ignite and Cassandra.
We wanted to dynamically load data from Cassandra, and thus were sensitive to this point.
4) Things get a bit trickier when the primary keys (partitioning) are not the same in Ignite and Cassandra. Only one key POJO can be deployed in Ignite, and it needs to handle the Ignite and Cassandra use cases.
Data Source is configured at the top level:
<!-- Cassandra Data Source -->
<bean id="cassandraDataSource" class="org.apache.ignite.cache.store.cassandra.datasource.DataSource">
<property name="user" value="cassandra"></property>
<property name="password" value="cassandra"></property>
<property name="contactPoints" value="10.24.51.190,10.24.51.187,10.24.51.150,10.24.51.240"/>
<property name="port" value="9042"/>
</bean>
Persistence settings for each cache are also at the top level. Here is an example that has a composite primary key and some field name mappings. Note the CDATA section.
<!-- Persistence settings for PortStats cache -->
<bean id="PortStatsCachePersistenceSettings" class="org.apache.ignite.cache.store.cassandra.persistence.KeyValuePersistenceSettings">
<constructor-arg type="java.lang.String">
<value><![CDATA[
<persistence keyspace="test_11" table="port_stats">
<keyPersistence class="com.abc.poc.icpoc.model.PortStatsKey" strategy="POJO">
<partitionKey>
<field name="day"/>
<field name="group"/> <!-- hash of switch-id -->
</partitionKey>
<clusterKey>
<field name="dateTime" column="date_time"/>
<field name="portId" column="port_id"/>
</clusterKey>
</keyPersistence>
<valuePersistence class="com.abc.poc.icpoc.model.PortStats" strategy="POJO">
<field name="portId" column="port_id"/>
<field name="dateTime" column="date_time"/>
<field name="portType" column="port_type"/>
<field name="switchId" column="switch_id"/>
<field name="rxUtil" column="rx_util"/>
<field name="txUtil" column="tx_util"/>
</valuePersistence>
</persistence>
]]></value>
</constructor-arg>
</bean>
Lastly, in the cache config, you need to reference the persistence settings. We did not use write-through or write-behind.
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="PortStatsCache"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="1"/>
<!-- for Cassandra backing store -->
<property name="readThrough" value="true"/>
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
<property name="dataSourceBean" value="cassandraDataSource"/>
<property name="persistenceSettingsBean" value="PortStatsCachePersistenceSettings"/>
</bean>
</property>
Hope this helps.
Roger
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: Thursday, August 31, 2017 8:56 AM
To: user@ignite.apache.org
Subject: Re: Ignite as Cache and Query layer for Cassandra.
Hello Mark,
I have found some documentation on how to do what the article describes:
"Apache Ignite can be inserted between Apache Cassandra and an existing application layer with no changes to the Cassandra data and only minimal changes to the application"
https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra<https://urldefense.proofpoint.com/v2/url?u=https-3A__apacheignite-2Dmix.readme.io_docs_ignite-2Dwith-2Dapache-2Dcassandra&d=DwMFaQ&c=IL_XqQWOjubgfqINi2jTzg&r=1esZO0r0bYS90lcsaLA6N4AFxuNo6lzauhETGwdJQoQ&m=V1yhk9HVShi4pdJWzPqkw3L6T-IdasiuF8bYdpMPp7w&s=JEswLCWrb1oQBPKMEjakE1kWv0rr5KARYVMUphxYciI&e=>
On the left you can see there are config examples, tests, AWS configuration, etc.
Hope it's good enough.
--
Ilya Kasnacheev
2017-08-31 16:15 GMT+03:00 Mark Farnan <ma...@petrolink.com>>:
Howdy,
I'm currently evaluating Ignite for a project, and am looking for some guidance and project sample for using Ignite over Cassandra as a cache and query layer.
Note: The Cassandra database and schema already exists and is mature.
In a Recent Infoworld post, (https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.infoworld.com_article_3191895_application-2Ddevelopment_light-2Da-2Dfire-2Dunder-2Dcassandra-2Dwith-2Dapache-2Dignite.html&d=DwMFaQ&c=IL_XqQWOjubgfqINi2jTzg&r=1esZO0r0bYS90lcsaLA6N4AFxuNo6lzauhETGwdJQoQ&m=V1yhk9HVShi4pdJWzPqkw3L6T-IdasiuF8bYdpMPp7w&s=9B2fqNQEKRiN4vS_MCJyh1K_6SCV1xVaPiLVEoqUqUQ&e=>) it stated
"No remodeling of Cassandra data
Apache Ignite reads from Apache Cassandra and other NoSQL databases, so moving Cassandra data into Ignite requires no data modification. The data schema can also be migrated directly into Ignite as is. "
However, I can't find in any documentation, or online articles, guidance on just how to do this and there are no links in the article.
- The Ignite docs /cassandra section appear to only cover using Cassandra as a persistence layer, not for going over and caching / querying an existing schema. (unless I'm missing something obvious !)
- The Youtube video cuts out to blank at the end, so no links are shown. The links on the slideshare document appear to be broken.
- Can't find any git hub for examples on how to do this (especially not for the couple you tube vids)
Can anyone please point me to some working sample code and/or documentation on how to do this ?
I'm especially interested in not having to migrate any schema's, nor having to pre-load the data into the cache. (lazy load), which the articles imply.
Also some of the data would be too large to load it all into Ram anyway.
Any assistance greatly appreciated.
Regards
Mark.
Re: Ignite as Cache and Query layer for Cassandra.
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Mark,
I have found some documentation on how to do what the article describes:
"Apache Ignite can be inserted between Apache Cassandra and an existing
application layer with no changes to the Cassandra data and only minimal
changes to the application"
https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra
On the left you can see there are config examples, tests, AWS
configuration, etc.
Hope it's good enough.
--
Ilya Kasnacheev
2017-08-31 16:15 GMT+03:00 Mark Farnan <ma...@petrolink.com>:
> Howdy,
>
> I'm currently evaluating Ignite for a project, and am looking for some
> guidance and project sample for using Ignite over Cassandra as a cache and
> query layer.
> Note: The Cassandra database and schema already exists and is mature.
>
> In a Recent Infoworld post, (https://www.infoworld.com/
> article/3191895/application-development/light-a-fire-
> under-cassandra-with-apache-ignite.html) it stated
>
> "*No remodeling of Cassandra data*
> Apache Ignite reads from Apache Cassandra and other NoSQL databases, so
> moving Cassandra data into Ignite requires no data modification. The data
> schema can also be migrated directly into Ignite as is. "
>
> However, I can't find in *any* documentation, or online articles,
> guidance on just how to do this and there are no links in the article.
> - The Ignite docs /cassandra section appear to only cover using
> Cassandra as a persistence layer, not for going over and caching /
> querying an existing schema. (unless I'm missing something obvious !)
> - The Youtube video cuts out to blank at the end, so no links are shown.
> The links on the slideshare document appear to be broken.
> - Can't find any git hub for examples on how to do this (especially not
> for the couple you tube vids)
>
> Can anyone please point me to some working sample code and/or
> documentation on how to do this ?
>
> I'm especially interested in not having to migrate any schema's, nor
> having to pre-load the data into the cache. (lazy load), which the
> articles imply.
> Also some of the data would be too large to load it all into Ram anyway.
>
>
> Any assistance greatly appreciated.
>
>
> Regards
>
> Mark.
>