You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Guofeng Zhang <gu...@gmail.com> on 2025/05/15 06:35:29 UTC

How Custom Cache Store or External Storage supported in Ignite 3

Hi,

In ignite 2, we use the CacheStore interface to cache our data in the
Ignite cache. But the Ignite 3 storage engine only supports in-memory store
and rocksdb, the docs has no description about the custom storage that
Ignite 2 support.

Does Ignite 3 support custom storage like Ignite 2?

 Thanks for your help.

guofeng

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Stephen Darlington <sd...@apache.org>.
Then you already have your answer. The cache store is that feature. It's
present in Ignite 2 but not in Ignite 3. You could build it yourself in
Ignite 3 or there are commercial solutions.

On Thu, 15 May 2025 at 10:29, Charlin S <ch...@hotelhub.com> wrote:

> Hi,
> Thank you for your reply,
>
> I am looking for transferring data from the DB table to Ignite cache grid
> without any interpreter. direct connectivity to DB to grid.
>
> Thanks,
> Charlin
>
>
>
> On Thu, 15 May 2025 at 14:50, Stephen Darlington <sd...@apache.org>
> wrote:
>
>> With the cache store, Ignite remains memory-only and delegates
>> responsibility for persisting data to disk to another system, usually a
>> database. When you start up Ignite, you need to copy your data from your
>> source database into Ignite. The cache store helps with that, but,
>> obviously, copying data is something you can fairly easily do by other
>> means.
>>
>> If you want to cache data in your client application (which is how I
>> interpret "keep data in memory on the system where programa/application
>> runs"), that's actually a "near cache" in Ignite.
>>
>>
>> On Thu, 15 May 2025 at 10:12, Charlin S <ch...@hotelhub.com> wrote:
>>
>>> Hi,
>>> My understanding of cachestore is to keep data in memory on the system
>>> where programa/application runs. If yes can we use a grid for the
>>> Cachestore concept ?
>>>
>>> Thanks,
>>> Charlin
>>>
>>> On Thu, 15 May 2025 at 14:22, Stephen Darlington <sd...@apache.org>
>>> wrote:
>>>
>>>> Just to add: there are commercial solutions that implement Cache
>>>> Store-like functionality for Ignite 3. But there are ways to build your own
>>>> solution using the data streamer and/or the Table APIs.
>>>>
>>>> On Thu, 15 May 2025 at 08:02, <yp...@t-online.de> wrote:
>>>>
>>>>> On 2025-05-15 14:35, Guofeng Zhang wrote:
>>>>> > Hi,
>>>>> >
>>>>> > In ignite 2, we use the CacheStore interface to cache our data in the
>>>>> > Ignite cache. But the Ignite 3 storage engine only supports
>>>>> in-memory
>>>>> > store
>>>>> > and rocksdb, the docs has no description about the custom storage
>>>>> that
>>>>> > Ignite 2 support.
>>>>> >
>>>>>
>>>>> This is a significant architectural change from Ignite 2, where you
>>>>> could implement custom CacheStore interfaces to integrate with
>>>>> external
>>>>> data sources. The Ignite 3 documentation indeed lacks descriptions
>>>>> about
>>>>> custom storage because this capability has been redesigned.
>>>>>
>>>>> If you need to integrate with external data sources in Ignite 3, you
>>>>> would typically use the Table API or consider using the change data
>>>>> capture (CDC) features instead of the CacheStore approach that was
>>>>> available in Ignite 2.
>>>>>
>>>>> --
>>>>> https://ypeng.pages.dev/
>>>>>
>>>>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Charlin S <ch...@hotelhub.com>.
Hi,
Thank you for your reply,

I am looking for transferring data from the DB table to Ignite cache grid
without any interpreter. direct connectivity to DB to grid.

Thanks,
Charlin



On Thu, 15 May 2025 at 14:50, Stephen Darlington <sd...@apache.org>
wrote:

> With the cache store, Ignite remains memory-only and delegates
> responsibility for persisting data to disk to another system, usually a
> database. When you start up Ignite, you need to copy your data from your
> source database into Ignite. The cache store helps with that, but,
> obviously, copying data is something you can fairly easily do by other
> means.
>
> If you want to cache data in your client application (which is how I
> interpret "keep data in memory on the system where programa/application
> runs"), that's actually a "near cache" in Ignite.
>
>
> On Thu, 15 May 2025 at 10:12, Charlin S <ch...@hotelhub.com> wrote:
>
>> Hi,
>> My understanding of cachestore is to keep data in memory on the system
>> where programa/application runs. If yes can we use a grid for the
>> Cachestore concept ?
>>
>> Thanks,
>> Charlin
>>
>> On Thu, 15 May 2025 at 14:22, Stephen Darlington <sd...@apache.org>
>> wrote:
>>
>>> Just to add: there are commercial solutions that implement Cache
>>> Store-like functionality for Ignite 3. But there are ways to build your own
>>> solution using the data streamer and/or the Table APIs.
>>>
>>> On Thu, 15 May 2025 at 08:02, <yp...@t-online.de> wrote:
>>>
>>>> On 2025-05-15 14:35, Guofeng Zhang wrote:
>>>> > Hi,
>>>> >
>>>> > In ignite 2, we use the CacheStore interface to cache our data in the
>>>> > Ignite cache. But the Ignite 3 storage engine only supports in-memory
>>>> > store
>>>> > and rocksdb, the docs has no description about the custom storage that
>>>> > Ignite 2 support.
>>>> >
>>>>
>>>> This is a significant architectural change from Ignite 2, where you
>>>> could implement custom CacheStore interfaces to integrate with external
>>>> data sources. The Ignite 3 documentation indeed lacks descriptions
>>>> about
>>>> custom storage because this capability has been redesigned.
>>>>
>>>> If you need to integrate with external data sources in Ignite 3, you
>>>> would typically use the Table API or consider using the change data
>>>> capture (CDC) features instead of the CacheStore approach that was
>>>> available in Ignite 2.
>>>>
>>>> --
>>>> https://ypeng.pages.dev/
>>>>
>>>

Re: Use pattern to ensure queue messages are never lost

Posted by Pavel Tupitsyn <pt...@apache.org>.
I suggest to implement your own queue logic with a transactional
cache, here is an example to demonstrate the idea:
https://gist.github.com/ptupitsyn/18a38db4dd1702558ebe273261f54233




On Sat, Jun 7, 2025 at 12:09 AM Alex Roytman <ro...@gmail.com> wrote:
>
> Could anyone share your patterns for ensuring queue messages are not lost on consumer node crash or a bug with unhandled error?
>
> Since Queue does not appear to offer transactional behavior for users of the queue (and it is pretty unclear what it would be), something else is needed. Option I am using currently is creating a message copy in shared file system to which all client nodes have access, updating its expiration timestamp in a separate thread while the message being processed (if needed) and checking periodically for any expired  copies pushing them back on the queue and deleting message copies  after successful processing or recovery. Same could be done with database more efficiently but this particular application fandles large flow of files we did not want to introduce a DB.
> Now we are going to AWS S3 and I am not sure I want to use S3 to mimic this behavior (charge for every S3 operation). I guess the options I have are to
> a) rewrite away from Ignite queues to Amazon SQS which is based on polling but has a very nice feature where taken messages become invisible for a configured period of time which could be extended and show up on the queue once time expires if not deleted by the consumer. Actually exactly what I was doing  with my file based recovery.
> b) Provision a small DynamoDB to replicate what I was doing with files,
> c) try to use Ignite (persistent) map to deposit copies of in-flight messages to be handled in the same way
> d) something I did not think about you may suggest :-)
>
> If I switch away from Ignite queues to AWS there will be only extensive Semaphores and Locks (and a little maps) left it amy not worth to maintain Ignite cluster I may have to consider using some Dynamo/polling based AWS distributed locking ...
>
> Would appreciate your comments - maybe I am missing something Ignite could do. I do not have that much experience  with Ignite - just switched from Hazelcast.
>

Use pattern to ensure queue messages are never lost

Posted by Alex Roytman <ro...@gmail.com>.
Could anyone share your patterns for ensuring queue messages are not lost
on consumer node crash or a bug with unhandled error?

Since Queue does not appear to offer transactional behavior for users of
the queue (and it is pretty unclear what it would be), something else is
needed. Option I am using currently is creating a message copy in shared
file system to which all client nodes have access, updating its expiration
timestamp in a separate thread while the message being processed (if
needed) and checking periodically for any expired  copies pushing them back
on the queue and deleting message copies  after successful processing or
recovery. Same could be done with database more efficiently but this
particular application fandles large flow of files we did not want to
introduce a DB.
Now we are going to AWS S3 and I am not sure I want to use S3 to mimic this
behavior (charge for every S3 operation). I guess the options I have are to
a) rewrite away from Ignite queues to Amazon SQS which is based on polling
but has a very nice feature where taken messages become invisible for a
configured period of time which could be extended and show up on the queue
once time expires if not deleted by the consumer. Actually exactly what I
was doing  with my file based recovery.
b) Provision a small DynamoDB to replicate what I was doing with files,
c) try to use Ignite (persistent) map to deposit copies of in-flight
messages to be handled in the same way
d) something I did not think about you may suggest :-)

If I switch away from Ignite queues to AWS there will be only extensive
Semaphores and Locks (and a little maps) left it amy not worth to maintain
Ignite cluster I may have to consider using some Dynamo/polling based AWS
distributed locking ...

Would appreciate your comments - maybe I am missing something Ignite could
do. I do not have that much experience  with Ignite - just switched from
Hazelcast.

Re: Can Queue be transactional

Posted by Alex Roytman <ro...@gmail.com>.
Thank you Pavel for the quick response so from the queue user perspective
transactions are not applicable.
Interestingly my quick test did not raise an exception simply added 2 items

final IgniteQueue<Value> queue = ignite.queue("myQueue", 8, new
CollectionConfiguration()
    .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)
    .setCacheMode(CacheMode.REPLICATED)
    .setBackups(4)
    .setCollocated(true)
);

          IgniteTransactions transactions = node.transactions();
          try (Transaction tx = transactions.txStart()) {
            for (int i = 0; i < 2; i++) {
              System.out.println("Queue Adding one entry ");
              queue.put(new Value(id.incrementAndGet(), "Queue node"));
            }
            tx.rollback();
//            tx.commit();
          }



On Thu, Jun 5, 2025 at 12:57 AM Pavel Tupitsyn <pt...@apache.org> wrote:

> Hello,
>
> With a transactional queue (CollectionConfiguration.atomicityMode =
> TRANSACTIONAL), the queue implementation starts a transaction
> internally [1].
>
> If a transaction already exists in the current thread, queue operation
> will fail with "Failed to start new transaction (current thread
> already has a transaction)" [2]
>
>
> [1]
> https://github.com/apache/ignite/blob/ee955df9fb80737292aac5f7ad3c82f8f0d8ea8e/modules/core/src/main/java/org/apache/ignite/internal/processors/datastructures/GridTransactionalCacheQueueImpl.java#L56
> [2]
> https://github.com/apache/ignite/blob/297bc9cc470c569d4c0e6ed12c734e3b92bc3d58/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTransactionsImpl.java#L194
>
> On Thu, Jun 5, 2025 at 5:24 AM Alex Roytman <ro...@gmail.com> wrote:
> >
> > Hello,
> >
> > Can Ignite 2.0 queue be transactional and if so what does it mean?
> > If a thread starts a transaction will it block the queue for other
> threads/clients?
> > If a thread rolls back its transaction will it return the taken item(s)
> the queue? In what place on the queue?
> >
> > Thank you,
> > Alex
> >
>

Re: Can Queue be transactional

Posted by Pavel Tupitsyn <pt...@apache.org>.
Hello,

With a transactional queue (CollectionConfiguration.atomicityMode =
TRANSACTIONAL), the queue implementation starts a transaction
internally [1].

If a transaction already exists in the current thread, queue operation
will fail with "Failed to start new transaction (current thread
already has a transaction)" [2]


[1] https://github.com/apache/ignite/blob/ee955df9fb80737292aac5f7ad3c82f8f0d8ea8e/modules/core/src/main/java/org/apache/ignite/internal/processors/datastructures/GridTransactionalCacheQueueImpl.java#L56
[2] https://github.com/apache/ignite/blob/297bc9cc470c569d4c0e6ed12c734e3b92bc3d58/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTransactionsImpl.java#L194

On Thu, Jun 5, 2025 at 5:24 AM Alex Roytman <ro...@gmail.com> wrote:
>
> Hello,
>
> Can Ignite 2.0 queue be transactional and if so what does it mean?
> If a thread starts a transaction will it block the queue for other threads/clients?
> If a thread rolls back its transaction will it return the taken item(s) the queue? In what place on the queue?
>
> Thank you,
> Alex
>

Can Queue be transactional

Posted by Alex Roytman <ro...@gmail.com>.
Hello,

Can Ignite 2.0 queue be transactional and if so what does it mean?
If a thread starts a transaction will it block the queue for
other threads/clients?
If a thread rolls back its transaction will it return the taken item(s) the
queue? In what place on the queue?

Thank you,
Alex

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Stephen Darlington <sd...@apache.org>.
With the cache store, Ignite remains memory-only and delegates
responsibility for persisting data to disk to another system, usually a
database. When you start up Ignite, you need to copy your data from your
source database into Ignite. The cache store helps with that, but,
obviously, copying data is something you can fairly easily do by other
means.

If you want to cache data in your client application (which is how I
interpret "keep data in memory on the system where programa/application
runs"), that's actually a "near cache" in Ignite.


On Thu, 15 May 2025 at 10:12, Charlin S <ch...@hotelhub.com> wrote:

> Hi,
> My understanding of cachestore is to keep data in memory on the system
> where programa/application runs. If yes can we use a grid for the
> Cachestore concept ?
>
> Thanks,
> Charlin
>
> On Thu, 15 May 2025 at 14:22, Stephen Darlington <sd...@apache.org>
> wrote:
>
>> Just to add: there are commercial solutions that implement Cache
>> Store-like functionality for Ignite 3. But there are ways to build your own
>> solution using the data streamer and/or the Table APIs.
>>
>> On Thu, 15 May 2025 at 08:02, <yp...@t-online.de> wrote:
>>
>>> On 2025-05-15 14:35, Guofeng Zhang wrote:
>>> > Hi,
>>> >
>>> > In ignite 2, we use the CacheStore interface to cache our data in the
>>> > Ignite cache. But the Ignite 3 storage engine only supports in-memory
>>> > store
>>> > and rocksdb, the docs has no description about the custom storage that
>>> > Ignite 2 support.
>>> >
>>>
>>> This is a significant architectural change from Ignite 2, where you
>>> could implement custom CacheStore interfaces to integrate with external
>>> data sources. The Ignite 3 documentation indeed lacks descriptions about
>>> custom storage because this capability has been redesigned.
>>>
>>> If you need to integrate with external data sources in Ignite 3, you
>>> would typically use the Table API or consider using the change data
>>> capture (CDC) features instead of the CacheStore approach that was
>>> available in Ignite 2.
>>>
>>> --
>>> https://ypeng.pages.dev/
>>>
>>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Charlin S <ch...@hotelhub.com>.
Hi,
My understanding of cachestore is to keep data in memory on the system
where programa/application runs. If yes can we use a grid for the
Cachestore concept ?

Thanks,
Charlin

On Thu, 15 May 2025 at 14:22, Stephen Darlington <sd...@apache.org>
wrote:

> Just to add: there are commercial solutions that implement Cache
> Store-like functionality for Ignite 3. But there are ways to build your own
> solution using the data streamer and/or the Table APIs.
>
> On Thu, 15 May 2025 at 08:02, <yp...@t-online.de> wrote:
>
>> On 2025-05-15 14:35, Guofeng Zhang wrote:
>> > Hi,
>> >
>> > In ignite 2, we use the CacheStore interface to cache our data in the
>> > Ignite cache. But the Ignite 3 storage engine only supports in-memory
>> > store
>> > and rocksdb, the docs has no description about the custom storage that
>> > Ignite 2 support.
>> >
>>
>> This is a significant architectural change from Ignite 2, where you
>> could implement custom CacheStore interfaces to integrate with external
>> data sources. The Ignite 3 documentation indeed lacks descriptions about
>> custom storage because this capability has been redesigned.
>>
>> If you need to integrate with external data sources in Ignite 3, you
>> would typically use the Table API or consider using the change data
>> capture (CDC) features instead of the CacheStore approach that was
>> available in Ignite 2.
>>
>> --
>> https://ypeng.pages.dev/
>>
>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by yp...@t-online.de.
On 2025-05-15 17:09, Nandhini S wrote:
> My doubt is how to test authentication?
> 

there is already the doc on their official website:
https://ignite.apache.org/docs/latest/security/authentication

regards.

-- 
https://ypeng.pages.dev/

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Nandhini S <na...@gmail.com>.
My doubt is how to test authentication?

On Thu, May 15, 2025, 2:20 PM Stephen Darlington <sd...@apache.org>
wrote:

> Just to add: there are commercial solutions that implement Cache
> Store-like functionality for Ignite 3. But there are ways to build your own
> solution using the data streamer and/or the Table APIs.
>
> On Thu, 15 May 2025 at 08:02, <yp...@t-online.de> wrote:
>
>> On 2025-05-15 14:35, Guofeng Zhang wrote:
>> > Hi,
>> >
>> > In ignite 2, we use the CacheStore interface to cache our data in the
>> > Ignite cache. But the Ignite 3 storage engine only supports in-memory
>> > store
>> > and rocksdb, the docs has no description about the custom storage that
>> > Ignite 2 support.
>> >
>>
>> This is a significant architectural change from Ignite 2, where you
>> could implement custom CacheStore interfaces to integrate with external
>> data sources. The Ignite 3 documentation indeed lacks descriptions about
>> custom storage because this capability has been redesigned.
>>
>> If you need to integrate with external data sources in Ignite 3, you
>> would typically use the Table API or consider using the change data
>> capture (CDC) features instead of the CacheStore approach that was
>> available in Ignite 2.
>>
>> --
>> https://ypeng.pages.dev/
>>
>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Stephen Darlington <sd...@apache.org>.
Just to add: there are commercial solutions that implement Cache Store-like
functionality for Ignite 3. But there are ways to build your own solution
using the data streamer and/or the Table APIs.

On Thu, 15 May 2025 at 08:02, <yp...@t-online.de> wrote:

> On 2025-05-15 14:35, Guofeng Zhang wrote:
> > Hi,
> >
> > In ignite 2, we use the CacheStore interface to cache our data in the
> > Ignite cache. But the Ignite 3 storage engine only supports in-memory
> > store
> > and rocksdb, the docs has no description about the custom storage that
> > Ignite 2 support.
> >
>
> This is a significant architectural change from Ignite 2, where you
> could implement custom CacheStore interfaces to integrate with external
> data sources. The Ignite 3 documentation indeed lacks descriptions about
> custom storage because this capability has been redesigned.
>
> If you need to integrate with external data sources in Ignite 3, you
> would typically use the Table API or consider using the change data
> capture (CDC) features instead of the CacheStore approach that was
> available in Ignite 2.
>
> --
> https://ypeng.pages.dev/
>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by yp...@t-online.de.
On 2025-05-15 15:40, Nandhini S wrote:
> Hi Team,
> 
> I am working on Apache Ignite. I am working on authentication part.
> 
> As per document, I see default user name and password is Ignite.
> 
> I want to create one ignite server node and ignite client node.. I want 
> to
> test with this username and password. And I want to give incorrect 
> username
> and password to test invalid authentication.
> 

so what's your exact question?

-- 
https://ypeng.pages.dev/

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Nandhini S <na...@gmail.com>.
Hi Team,

I am working on Apache Ignite. I am working on authentication part.

As per document, I see default user name and password is Ignite.

I want to create one ignite server node and ignite client node.. I want to
test with this username and password. And I want to give incorrect username
and password to test invalid authentication.

On Thu, May 15, 2025, 1:06 PM <yp...@t-online.de> wrote:

> On 2025-05-15 15:19, Charlin S wrote:
> > Hi,
> > Do we have sample code/example for CacheStore implementation? in Java
> > or
> > Dotnet.
> >
>
> Hi
>
> Here's a sample CacheStore implementation in Java for Ignite 2:
>
> import org.apache.ignite.cache.store.CacheStore;
> import org.apache.ignite.cache.store.CacheStoreAdapter;
> import org.apache.ignite.lang.IgniteBiInClosure;
> import org.jetbrains.annotations.Nullable;
>
> import javax.cache.Cache;
> import javax.cache.integration.CacheLoaderException;
> import javax.cache.integration.CacheWriterException;
> import java.sql.Connection;
> import java.sql.DriverManager;
> import java.sql.PreparedStatement;
> import java.sql.ResultSet;
> import java.sql.SQLException;
> import java.util.Map;
>
> public class PersonCacheStore extends CacheStoreAdapter<Integer, Person>
> {
>
>      private static final String DB_URL =
> "jdbc:mysql://localhost:3306/mydb";
>      private static final String USER = "username";
>      private static final String PASS = "password";
>
>      @Override
>      public Person load(Integer key) throws CacheLoaderException {
>          try (Connection conn = getConnection();
>               PreparedStatement stmt = conn.prepareStatement("SELECT id,
> name, age FROM Person WHERE id = ?")) {
>
>              stmt.setInt(1, key);
>
>              try (ResultSet rs = stmt.executeQuery()) {
>                  if (rs.next()) {
>                      return new Person(rs.getInt("id"),
> rs.getString("name"), rs.getInt("age"));
>                  }
>              }
>
>              return null;
>          }
>          catch (SQLException e) {
>              throw new CacheLoaderException("Failed to load Person with
> ID: " + key, e);
>          }
>      }
>
>      @Override
>      public void write(Cache.Entry<? extends Integer, ? extends Person>
> entry) throws CacheWriterException {
>          Person person = entry.getValue();
>          Integer key = entry.getKey();
>
>          try (Connection conn = getConnection();
>               PreparedStatement stmt = conn.prepareStatement(
>                   "INSERT INTO Person (id, name, age) VALUES (?, ?, ?) "
> +
>                   "ON DUPLICATE KEY UPDATE name = ?, age = ?")) {
>
>              stmt.setInt(1, key);
>              stmt.setString(2, person.getName());
>              stmt.setInt(3, person.getAge());
>              stmt.setString(4, person.getName());
>              stmt.setInt(5, person.getAge());
>
>              stmt.executeUpdate();
>          }
>          catch (SQLException e) {
>              throw new CacheWriterException("Failed to write Person: " +
> person, e);
>          }
>      }
>
>      @Override
>      public void delete(Object key) throws CacheWriterException {
>          try (Connection conn = getConnection();
>               PreparedStatement stmt = conn.prepareStatement("DELETE FROM
> Person WHERE id = ?")) {
>
>              stmt.setInt(1, (Integer) key);
>              stmt.executeUpdate();
>          }
>          catch (SQLException e) {
>              throw new CacheWriterException("Failed to delete Person with
> ID: " + key, e);
>          }
>      }
>
>      @Override
>      public void loadCache(IgniteBiInClosure<Integer, Person> clo,
> @Nullable Object... args) {
>          try (Connection conn = getConnection();
>               PreparedStatement stmt = conn.prepareStatement("SELECT id,
> name, age FROM Person");
>               ResultSet rs = stmt.executeQuery()) {
>
>              while (rs.next()) {
>                  Integer id = rs.getInt("id");
>                  String name = rs.getString("name");
>                  int age = rs.getInt("age");
>
>                  Person person = new Person(id, name, age);
>                  clo.apply(id, person);
>              }
>          }
>          catch (SQLException e) {
>              throw new CacheLoaderException("Failed to load all persons",
> e);
>          }
>      }
>
>      private Connection getConnection() throws SQLException {
>          return DriverManager.getConnection(DB_URL, USER, PASS);
>      }
> }
>
> // Person class
> class Person {
>      private int id;
>      private String name;
>      private int age;
>
>      public Person(int id, String name, int age) {
>          this.id = id;
>          this.name = name;
>          this.age = age;
>      }
>
>      // Getters and setters
>      public int getId() { return id; }
>      public void setId(int id) { this.id = id; }
>      public String getName() { return name; }
>      public void setName(String name) { this.name = name; }
>      public int getAge() { return age; }
>      public void setAge(int age) { this.age = age; }
>
>      @Override
>      public String toString() {
>          return "Person{id=" + id + ", name='" + name + "', age=" + age +
> '}';
>      }
> }
>
> --
> https://ypeng.pages.dev/
>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by yp...@t-online.de.
On 2025-05-15 15:19, Charlin S wrote:
> Hi,
> Do we have sample code/example for CacheStore implementation? in Java 
> or
> Dotnet.
> 

Hi

Here's a sample CacheStore implementation in Java for Ignite 2:

import org.apache.ignite.cache.store.CacheStore;
import org.apache.ignite.cache.store.CacheStoreAdapter;
import org.apache.ignite.lang.IgniteBiInClosure;
import org.jetbrains.annotations.Nullable;

import javax.cache.Cache;
import javax.cache.integration.CacheLoaderException;
import javax.cache.integration.CacheWriterException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Map;

public class PersonCacheStore extends CacheStoreAdapter<Integer, Person> 
{

     private static final String DB_URL = 
"jdbc:mysql://localhost:3306/mydb";
     private static final String USER = "username";
     private static final String PASS = "password";

     @Override
     public Person load(Integer key) throws CacheLoaderException {
         try (Connection conn = getConnection();
              PreparedStatement stmt = conn.prepareStatement("SELECT id, 
name, age FROM Person WHERE id = ?")) {

             stmt.setInt(1, key);

             try (ResultSet rs = stmt.executeQuery()) {
                 if (rs.next()) {
                     return new Person(rs.getInt("id"), 
rs.getString("name"), rs.getInt("age"));
                 }
             }

             return null;
         }
         catch (SQLException e) {
             throw new CacheLoaderException("Failed to load Person with 
ID: " + key, e);
         }
     }

     @Override
     public void write(Cache.Entry<? extends Integer, ? extends Person> 
entry) throws CacheWriterException {
         Person person = entry.getValue();
         Integer key = entry.getKey();

         try (Connection conn = getConnection();
              PreparedStatement stmt = conn.prepareStatement(
                  "INSERT INTO Person (id, name, age) VALUES (?, ?, ?) " 
+
                  "ON DUPLICATE KEY UPDATE name = ?, age = ?")) {

             stmt.setInt(1, key);
             stmt.setString(2, person.getName());
             stmt.setInt(3, person.getAge());
             stmt.setString(4, person.getName());
             stmt.setInt(5, person.getAge());

             stmt.executeUpdate();
         }
         catch (SQLException e) {
             throw new CacheWriterException("Failed to write Person: " + 
person, e);
         }
     }

     @Override
     public void delete(Object key) throws CacheWriterException {
         try (Connection conn = getConnection();
              PreparedStatement stmt = conn.prepareStatement("DELETE FROM 
Person WHERE id = ?")) {

             stmt.setInt(1, (Integer) key);
             stmt.executeUpdate();
         }
         catch (SQLException e) {
             throw new CacheWriterException("Failed to delete Person with 
ID: " + key, e);
         }
     }

     @Override
     public void loadCache(IgniteBiInClosure<Integer, Person> clo, 
@Nullable Object... args) {
         try (Connection conn = getConnection();
              PreparedStatement stmt = conn.prepareStatement("SELECT id, 
name, age FROM Person");
              ResultSet rs = stmt.executeQuery()) {

             while (rs.next()) {
                 Integer id = rs.getInt("id");
                 String name = rs.getString("name");
                 int age = rs.getInt("age");

                 Person person = new Person(id, name, age);
                 clo.apply(id, person);
             }
         }
         catch (SQLException e) {
             throw new CacheLoaderException("Failed to load all persons", 
e);
         }
     }

     private Connection getConnection() throws SQLException {
         return DriverManager.getConnection(DB_URL, USER, PASS);
     }
}

// Person class
class Person {
     private int id;
     private String name;
     private int age;

     public Person(int id, String name, int age) {
         this.id = id;
         this.name = name;
         this.age = age;
     }

     // Getters and setters
     public int getId() { return id; }
     public void setId(int id) { this.id = id; }
     public String getName() { return name; }
     public void setName(String name) { this.name = name; }
     public int getAge() { return age; }
     public void setAge(int age) { this.age = age; }

     @Override
     public String toString() {
         return "Person{id=" + id + ", name='" + name + "', age=" + age + 
'}';
     }
}

-- 
https://ypeng.pages.dev/

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by Charlin S <ch...@hotelhub.com>.
Hi,
Do we have sample code/example for CacheStore implementation? in Java or
Dotnet.

Thanks,
Charlin

On Thu, 15 May 2025 at 12:32, <yp...@t-online.de> wrote:

> On 2025-05-15 14:35, Guofeng Zhang wrote:
> > Hi,
> >
> > In ignite 2, we use the CacheStore interface to cache our data in the
> > Ignite cache. But the Ignite 3 storage engine only supports in-memory
> > store
> > and rocksdb, the docs has no description about the custom storage that
> > Ignite 2 support.
> >
>
> This is a significant architectural change from Ignite 2, where you
> could implement custom CacheStore interfaces to integrate with external
> data sources. The Ignite 3 documentation indeed lacks descriptions about
> custom storage because this capability has been redesigned.
>
> If you need to integrate with external data sources in Ignite 3, you
> would typically use the Table API or consider using the change data
> capture (CDC) features instead of the CacheStore approach that was
> available in Ignite 2.
>
> --
> https://ypeng.pages.dev/
>

Re: How Custom Cache Store or External Storage supported in Ignite 3

Posted by yp...@t-online.de.
On 2025-05-15 14:35, Guofeng Zhang wrote:
> Hi,
> 
> In ignite 2, we use the CacheStore interface to cache our data in the
> Ignite cache. But the Ignite 3 storage engine only supports in-memory 
> store
> and rocksdb, the docs has no description about the custom storage that
> Ignite 2 support.
> 

This is a significant architectural change from Ignite 2, where you 
could implement custom CacheStore interfaces to integrate with external 
data sources. The Ignite 3 documentation indeed lacks descriptions about 
custom storage because this capability has been redesigned.

If you need to integrate with external data sources in Ignite 3, you 
would typically use the Table API or consider using the change data 
capture (CDC) features instead of the CacheStore approach that was 
available in Ignite 2.

-- 
https://ypeng.pages.dev/