You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by nithin91 <ni...@franklintempleton.com> on 2020/02/12 18:11:12 UTC
Scheduling Cache Refresh using Ignite
Hi
We are doing a a POC on exploring the Ignite in memory capabilities and
building a rest api on
top of it using node express.
Currently as a part of POC, installed Ignite in UNIX and trying to load the
data from Oracle DB
to Ignite Cache using Cache JDBC Pojo Store.
Can someone help me whether the following scenarios can be handled using
Ignite as i couldn't find this in the official documentation.
1. If we want to add/drop/modify a column to the cache, can we update the
bean file directly
when the node is running or do we need to stop the node and then again
restart.
It would be really helpful if you can share sample code or
documentation link.
2. How to refresh the ignite cache automatically or schedule the cache
refresh.
It would be really helpful if you can share sample code or
documentation link.
3. Is incremental refresh allowed? It would be really helpful if you can
share sample code or
documentation link.
4. Is there any other way to load the caches fast other Cache JDBC POJO
Store.
It would be really helpful if you can share sample code or
documentation link.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by nithin91 <ni...@franklintempleton.com>.
Following is the java code that loads the cache.
package Load;
import java.sql.Types;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory;
import org.apache.ignite.cache.store.jdbc.JdbcType;
import org.apache.ignite.cache.store.jdbc.JdbcTypeField;
import org.apache.ignite.cache.store.jdbc.dialect.OracleDialect;
import org.apache.ignite.configuration.CacheConfiguration;
import ignite.example.IgniteUnixImplementation.OrderDetails;
import ignite.example.IgniteUnixImplementation.OrderKey;
public class OrdersLoad {
private static final class CacheJdbcPojoStoreExampleFactory extends
CacheJdbcPojoStoreFactory<OrderKey, OrderDetails> {
/**
*
*/
private static final long serialVersionUID = 1L;
/** {@inheritDoc} */
@Override public CacheJdbcPojoStore<OrderKey, OrderDetails> create()
{
setDataSourceBean("dataSource");
return super.create();
}
}
private static CacheConfiguration<OrderKey, OrderDetails>
cacheConfiguration() {
CacheConfiguration<OrderKey, OrderDetails> cfg = new
CacheConfiguration<>("OrdersCache");
CacheJdbcPojoStoreExampleFactory storefactory =new
CacheJdbcPojoStoreExampleFactory();
storefactory.setDialect(new OracleDialect());
storefactory.setDataSourceBean("dataSource");
JdbcType jdbcType = new JdbcType();
jdbcType.setCacheName("OrdersCache");
jdbcType.setDatabaseSchema("PDS_CACHE");
jdbcType.setDatabaseTable("ORDERS2");
jdbcType.setKeyType("ignite.example.IgniteUnixImplementation.OrderKey");
jdbcType.setKeyFields(new JdbcTypeField(Types.INTEGER, "ORDERID",
Long.class, "OrderID"),
new JdbcTypeField(Types.INTEGER, "CITYID", Long.class, "CityID")
);
jdbcType.setValueType("ignite.example.IgniteUnixImplementation.OrderDetails");
jdbcType.setValueFields(
new JdbcTypeField(Types.VARCHAR, "PRODUCTNAME", String.class,
"Productname"),
new JdbcTypeField(Types.VARCHAR, "CUSTOMERNAME", String.class,
"CustomerName"),
new JdbcTypeField(Types.VARCHAR, "STORENAME", String.class,
"StoreName")
);
storefactory.setTypes(jdbcType);
cfg.setCacheStoreFactory(storefactory);
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cfg.setReadThrough(true);
cfg.setWriteThrough(true);
cfg.setSqlSchema("PIE");
return cfg;
}
public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("Ignite-Client.xml")) {
System.out.println(">>> Loading cache OrderDetails");
IgniteCache<OrderKey, OrderDetails> cache =
ignite.getOrCreateCache(cacheConfiguration());
cache.clear();
ignite.cache("OrdersCache").loadCache(null);
System.out.println(">>> Loaded cache: OrdersCache
Size="+cache.size());
}
}
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by Andrei Aleksandrov <ae...@gmail.com>.
Hi Nithin,
You face current message because your client lost the connection to the
server. It tries to get the acknowledge message on some operation (I
guess it should be some cache operation).
You can see that IgniteClientDisconnectedException was thrown. In this
case, you can get the reconnect future and wait for the client reconnection:
https://apacheignite.readme.io/docs/clients-vs-servers#reconnecting-a-client
Please add try/catch blocks around your cache operation and add next logic:
|catch (IgniteClientDisconnectedException e) {
e.reconnectFuture().get(); // Wait for reconnect. // Can proceed and use
the same IgniteCompute instance. }|
I can't say why your client was disconnected. Highly likely it's because
of some network issues. You can try to take a look at server logs and
find there *NODE_LEFT *or *NODE_FAILED *messages.
BR,
Andrei
2/14/2020 8:08 AM, nithin91 пишет:
> Hi
>
> I am unable to attach any file as a result of which i pasted the code and
> bean file in my previous messages.
>
> Following is error i get.
>
> Feb 13, 2020 11:34:40 AM org.apache.ignite.logger.java.JavaLogger error
> SEVERE: Failed to send message: null
> java.io.IOException: Failed to get acknowledge for message:
> TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
>
>
>
> [sndNodeId=null, id=b9bb52d3071-613fd9b8-0c00-4dde-ba8f-8f5341734a3c,
> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
> isClient=true]]
> at
> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
> Feb 13, 2020 11:34:47 AM org.apache.ignite.logger.java.JavaLogger error
> SEVERE: Failed to reconnect to cluster (consider increasing 'networkTimeout'
> configuration property) [networkTimeout=5000]
> [11:34:52] Ignite node stopped OK [uptime=00:00:24.772]
> Exception in thread "main" javax.cache.CacheException: class
> org.apache.ignite.IgniteClientDisconnectedException: Failed to execute
> dynamic cache change request, client node disconnected.
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3023)
> at
> org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2992)
> at Load.OrdersLoad.main(OrdersLoad.java:82)
> Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Failed
> to execute dynamic cache change request, client node disconnected.
> at
> org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:952)
> at
> org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:948)
> ... 4 more
> Caused by: class
> org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Failed
> to execute dynamic cache change request, client node disconnected.
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:1180)
> at
> org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:3949)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:821)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by nithin91 <ni...@franklintempleton.com>.
Hi
I am unable to attach any file as a result of which i pasted the code and
bean file in my previous messages.
Following is error i get.
Feb 13, 2020 11:34:40 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=b9bb52d3071-613fd9b8-0c00-4dde-ba8f-8f5341734a3c,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Feb 13, 2020 11:34:47 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to reconnect to cluster (consider increasing 'networkTimeout'
configuration property) [networkTimeout=5000]
[11:34:52] Ignite node stopped OK [uptime=00:00:24.772]
Exception in thread "main" javax.cache.CacheException: class
org.apache.ignite.IgniteClientDisconnectedException: Failed to execute
dynamic cache change request, client node disconnected.
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache0(IgniteKernal.java:3023)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2992)
at Load.OrdersLoad.main(OrdersLoad.java:82)
Caused by: class org.apache.ignite.IgniteClientDisconnectedException: Failed
to execute dynamic cache change request, client node disconnected.
at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:952)
at
org.apache.ignite.internal.util.IgniteUtils$15.apply(IgniteUtils.java:948)
... 4 more
Caused by: class
org.apache.ignite.internal.IgniteClientDisconnectedCheckedException: Failed
to execute dynamic cache change request, client node disconnected.
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onDisconnected(GridCacheProcessor.java:1180)
at
org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:3949)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:821)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:604)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2667)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by Andrei Aleksandrov <ae...@gmail.com>.
Hi,
Can you please attach the full logs with the mentioned exception? BTW I
don't see any attaches in the previous message (probably user list can't
do it).
BR,
Andrei
2/13/2020 3:44 PM, nithin91 пишет:
> Attached the bean file used
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by nithin91 <ni...@franklintempleton.com>.
Attached the bean file used
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by nithin91 <ni...@franklintempleton.com>.
Thanks aealexsandrov. This information is very useful.
Also i have one more query.
Currently as a part of POC, installed Ignite in UNIX and trying to load the
data from Oracle DB to Ignite Cache using Cache JDBC Pojo Store.
As a part of this process, bean file is custom configured to start
ignite
node in unix. Attached the bean file.
This bean file consists of both cache configuration details and
ignite
configuration details.
Once the node is running, we are trying to do the following
1. Connecting to the ignite node running on unix through eclipse by
creating a replica of
the attached bean file from local system and adding an additional property
in Bean file with
Client Mode = true and
then loading the cache that are defined in the bean file deployed
in
unix using the
following method from local system using JAVA
ignite.cache("CacheName").loadCache(null);
* We are able to do this successfully.*
2. Connecting to the ignite node running on unix by creating a
replica of
the attached bean file
in local system and adding an additional property in Bean
file with Client
Mode = true
and then trying to create a cache and configure the cache
and then finally
loading
the cache using the attached JAVA Code.
* When we are trying this approach, we are getting an error
like dynamic
cache change
is not allowed.Not getting this error when Ignite server
node and client node is running on local machine.Getting this error when
server node is running in unix and trying to connect to this node from local
system.*
It would be really helpful if you can help me in resolving
this issue.
If this not the right approach, then
Configuring all the caches in the bean file is the only
available
option?If this is case,
What should be the approach for building some additional
caches in ignite
and load these Caches using Cache JDBC POJO Store when the node is running.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Scheduling Cache Refresh using Ignite
Posted by Andrei Aleksandrov <ae...@gmail.com>.
Hi,
Please read my comments:
1)Ignite generally doesn't support changing of the cache configuration
without re-creation of the the cache. But for SQL caches that were
created via QueryEntity or CREATE TABLE you can add and remove the
columns using ALTER TABLE commands:
https://apacheignite-sql.readme.io/docs/alter-table
https://apacheignite.readme.io/docs/cache-queries#query-configuration-using-queryentity
https://apacheignite-sql.readme.io/docs/create-table
2)First of all, you can use the following options:
https://apacheignite.readme.io/docs/3rd-party-store#section-read-through-and-write-through
Read through can load the requested keys from DB
Write through will load all the updates to DB.
In case if you require some cache invalidation or refresh then you can
create some cron job for it.
3)I guess that loadCache is the only to do it. It will filter the values
that have already existed in the cache.
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#loadCache-org.apache.ignite.lang.IgniteBiPredicate-java.lang.Object...-
4)You can use a different subset of integrations that can do distributed
streaming to Ignite like Spark or Kafka:
https://apacheignite-mix.readme.io/docs/getting-started
BR,
Andrei
2/12/2020 9:11 PM, nithin91 пишет:
> Hi
>
> We are doing a a POC on exploring the Ignite in memory capabilities and
> building a rest api on
> top of it using node express.
>
>
> Currently as a part of POC, installed Ignite in UNIX and trying to load the
> data from Oracle DB
> to Ignite Cache using Cache JDBC Pojo Store.
>
> Can someone help me whether the following scenarios can be handled using
> Ignite as i couldn't find this in the official documentation.
>
> 1. If we want to add/drop/modify a column to the cache, can we update the
> bean file directly
> when the node is running or do we need to stop the node and then again
> restart.
> It would be really helpful if you can share sample code or
> documentation link.
>
> 2. How to refresh the ignite cache automatically or schedule the cache
> refresh.
> It would be really helpful if you can share sample code or
> documentation link.
>
> 3. Is incremental refresh allowed? It would be really helpful if you can
> share sample code or
> documentation link.
>
>
> 4. Is there any other way to load the caches fast other Cache JDBC POJO
> Store.
> It would be really helpful if you can share sample code or
> documentation link.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/