RE: Ignite as Cache and Query layer for Cassandra.

["Roger Fischer (CW)" <rf...@Brocade.com>]

Hi Mark,

I recently did a Proof-of-Concept with Ignite and Cassandra. There are a number of challenges you need to look out for:

1) In order to do SQL queries, you need to have all the data in Ignite. It is not practical to include other persistence layers in the query process.
The exception is Ignite Native Persistence.

2) You need to deploy the POJO for the objects (and keys if they are not basic types) in Ignite. This may become unnecessary in a near-future release.

3) Preloading data is not particularly efficient. Each Ignite node sends the preload query. Each Ignite node ignores the data it is not primary or backup for. It may be possible to optimize this for the case where the partitioning is the same in Ignite and Cassandra.
We wanted to dynamically load data from Cassandra, and thus were sensitive to this point.

4) Things get a bit trickier when the primary keys (partitioning) are not the same in Ignite and Cassandra. Only one key POJO can be deployed in Ignite, and it needs to handle the Ignite and Cassandra use cases.

Data Source is configured at the top level:

    <!-- Cassandra Data Source -->
    <bean id="cassandraDataSource" class="org.apache.ignite.cache.store.cassandra.datasource.DataSource">
        <property name="user" value="cassandra"></property>
        <property name="password" value="cassandra"></property>
        <property name="contactPoints" value="10.24.51.190,10.24.51.187,10.24.51.150,10.24.51.240"/>
        <property name="port" value="9042"/>
    </bean>

Persistence settings for each cache are also at the top level. Here is an example that has a composite primary key and some field name mappings. Note the CDATA section.

    <!-- Persistence settings for PortStats cache -->
    <bean id="PortStatsCachePersistenceSettings" class="org.apache.ignite.cache.store.cassandra.persistence.KeyValuePersistenceSettings">
        <constructor-arg type="java.lang.String">
            <value><![CDATA[
                <persistence keyspace="test_11" table="port_stats">
                    <keyPersistence class="com.abc.poc.icpoc.model.PortStatsKey" strategy="POJO">
                        <partitionKey>
                            <field name="day"/>
                            <field name="group"/>  <!-- hash of switch-id -->
                        </partitionKey>
                        <clusterKey>
                            <field name="dateTime" column="date_time"/>
                            <field name="portId" column="port_id"/>
                        </clusterKey>
                    </keyPersistence>
                    <valuePersistence class="com.abc.poc.icpoc.model.PortStats" strategy="POJO">
                        <field name="portId" column="port_id"/>
                        <field name="dateTime" column="date_time"/>
                        <field name="portType" column="port_type"/>
                        <field name="switchId" column="switch_id"/>
                        <field name="rxUtil" column="rx_util"/>
                        <field name="txUtil" column="tx_util"/>
                    </valuePersistence>
                </persistence>
            ]]></value>
        </constructor-arg>
    </bean>

Lastly, in the cache config, you need to reference the persistence settings. We did not use write-through or write-behind.

                <bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="PortStatsCache"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="backups" value="1"/>

                    <!-- for Cassandra backing store -->
                    <property name="readThrough" value="true"/>
                    <property name="cacheStoreFactory">
                        <bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
                            <property name="dataSourceBean" value="cassandraDataSource"/>
                            <property name="persistenceSettingsBean" value="PortStatsCachePersistenceSettings"/>
                        </bean>
                    </property>

Hope this helps.

Roger

From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: Thursday, August 31, 2017 8:56 AM
To: user@ignite.apache.org
Subject: Re: Ignite as Cache and Query layer for Cassandra.

Hello Mark,

I have found some documentation on how to do what the article describes:

"Apache Ignite can be inserted between Apache Cassandra and an existing application layer with no changes to the Cassandra data and only minimal changes to the application"

https://apacheignite-mix.readme.io/docs/ignite-with-apache-cassandra<https://urldefense.proofpoint.com/v2/url?u=https-3A__apacheignite-2Dmix.readme.io_docs_ignite-2Dwith-2Dapache-2Dcassandra&d=DwMFaQ&c=IL_XqQWOjubgfqINi2jTzg&r=1esZO0r0bYS90lcsaLA6N4AFxuNo6lzauhETGwdJQoQ&m=V1yhk9HVShi4pdJWzPqkw3L6T-IdasiuF8bYdpMPp7w&s=JEswLCWrb1oQBPKMEjakE1kWv0rr5KARYVMUphxYciI&e=>
On the left you can see there are config examples, tests, AWS configuration, etc.
Hope it's good enough.


--
Ilya Kasnacheev

2017-08-31 16:15 GMT+03:00 Mark Farnan <ma...@petrolink.com>>:
Howdy,

I'm currently evaluating Ignite for a project, and am looking for some guidance and project sample for using Ignite over Cassandra as a cache and query layer.
Note: The Cassandra database and schema already exists and is mature.

In a Recent Infoworld post,   (https://www.infoworld.com/article/3191895/application-development/light-a-fire-under-cassandra-with-apache-ignite.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.infoworld.com_article_3191895_application-2Ddevelopment_light-2Da-2Dfire-2Dunder-2Dcassandra-2Dwith-2Dapache-2Dignite.html&d=DwMFaQ&c=IL_XqQWOjubgfqINi2jTzg&r=1esZO0r0bYS90lcsaLA6N4AFxuNo6lzauhETGwdJQoQ&m=V1yhk9HVShi4pdJWzPqkw3L6T-IdasiuF8bYdpMPp7w&s=9B2fqNQEKRiN4vS_MCJyh1K_6SCV1xVaPiLVEoqUqUQ&e=>) it stated

"No remodeling of Cassandra data
Apache Ignite reads from Apache Cassandra and other NoSQL databases, so moving Cassandra data into Ignite requires no data modification. The data schema can also be migrated directly into Ignite as is. "

However, I can't find in any documentation,  or online articles,  guidance on just how to do this and there are no links in the article.
  - The Ignite docs /cassandra section appear to only cover using Cassandra as a persistence layer,  not for going over and caching / querying an existing schema.   (unless I'm missing something obvious !)
  - The Youtube video cuts out to blank at the end, so no links are shown.    The  links on the slideshare document appear to be broken.
  -  Can't find any git hub for examples on how to do this (especially not for the couple you tube vids)

Can anyone please point me to some working sample code and/or documentation on how to do this ?

I'm especially interested in not having to migrate any schema's,  nor having to pre-load the data into the cache.  (lazy load), which the articles imply.
Also some of the data would be too large to load it all into Ram anyway.


Any assistance greatly appreciated.


Regards

Mark.