You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by nithin91 <ni...@franklintempleton.com> on 2020/01/27 08:36:26 UTC
Data Load to Ignite cache is very slow from Oracle Table
Hi
I am trying to load data from Oracle Table to Ignite Cache using Cache Store
Load Cache method.
Following is the logic implemented in Load Cache method to load the data
from Oracle Table using Ignite Cache.
1. JDBC connector is used to connect to Oracle Table and the data is
available in Result Set Cursor.
2. While loop is used to loop through the Result Set Object and insert the
data into cache.
Is there any other way to insert the data from Oracle Table to Ignite Cache.
If possible please share sample code.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by nithin91 <ni...@franklintempleton.com>.
Its taking almost 1hour to load the 0.1 million data using result set Cursor
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by nithin91 <ni...@franklintempleton.com>.
Hi Ilya,
Thank you so much. Issue is resolved after following the steps mentioned in
the link you have shared.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I can see that you only define data source locally. It needs to be defined
on all server nodes participating in cache load.
Please take a look at https://apacheignite-mix.readme.io/docs/examples
Regards,
--
Ilya Kasnacheev
пн, 27 янв. 2020 г. в 17:45, nithin91 <
nithinbharadwaj.govindaraju@franklintempleton.com>:
> Hi Belyakov,
>
> Thank you so much. This is very helpful.
>
> I am facing the following error when i am using this approach
>
> Failed to start component: class org.apache.ignite.IgniteException: Failed
> to initialize cache store (data source is not provided).
>
> Below is the code used for implementation.I have configured the data source
> property correctly but not sure why this error pops up.Can you please help
> me on this.
>
> package ignite.example.ignite_read;
>
> import java.sql.SQLException;
> import java.util.ArrayList;
> import java.util.HashSet;
> import java.util.LinkedHashMap;
> import java.util.Set;
>
> //import javax.activation.DataSource;
> import javax.cache.configuration.Factory;
> import javax.cache.integration.CacheLoaderException;
> import javax.sql.DataSource;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.IgniteException;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.cache.QueryEntity;
> import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory;
> import org.apache.ignite.cache.store.jdbc.JdbcType;
> import org.apache.ignite.cache.store.jdbc.JdbcTypeField;
> import org.apache.ignite.cache.store.jdbc.dialect.OracleDialect;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
>
> import oracle.jdbc.pool.OracleDataSource;
>
> public class IgniteCacheload {
>
> @SuppressWarnings("unchecked")
> public static void main(String[] args) throws IgniteException,
> SQLException
> {
>
>
>
>
> IgniteConfiguration config = new IgniteConfiguration();
> /*
> // config code
> *
>
> try (Ignite ignite = Ignition.start(config)){
>
> CacheConfiguration<ProductKey,Products> prdCacheCfg = new
> CacheConfiguration<>();
>
> prdCacheCfg.setName("ProdCache");
> prdCacheCfg.setCacheMode(CacheMode.PARTITIONED);
> prdCacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
>
> //personCacheCfg.setReadThrough(true);
> //personCacheCfg.setWriteThrough(true);
>
> CacheJdbcPojoStoreFactory<ProductKey,Products> factory =
> new
> CacheJdbcPojoStoreFactory<>();
>
>
> factory.setDialect(new OracleDialect());
> //factory.setDataSource(dsdetails());
> factory.setDataSourceFactory(dsdetails());
>
> JdbcType productType = new JdbcType();
> productType.setCacheName("ProdCache");
> productType.setKeyType(ProductKey.class);
> productType.setValueType(Products.class);
> // Specify the schema if applicable
>
> productType.setDatabaseTable("table");
>
> productType.setKeyFields(new
> JdbcTypeField(java.sql.Types.VARCHAR, "fid",
> ProductKey.class, "FID"));
> productType.setValueFields(new
> JdbcTypeField(java.sql.Types.VARCHAR,
> "scode", Products.class, "scode"));
>
>
> factory.setTypes(productType);
>
> prdCacheCfg.setCacheStoreFactory(factory);
>
>
> config.setCacheConfiguration(prdCacheCfg);
>
>
>
> IgniteCache<ProductKey, Products> cache =
> ignite.getOrCreateCache(prdCacheCfg);
>
> cache.clear();
>
> // Load cache on all data nodes with default SQL statement.
> System.out.println(">>> Load ALL data to cache from DB...");
> cache.loadCache(null);
>
>
> System.out.println(">>> Loaded cache entries: " +
> cache.size());
>
> }
> catch (Exception e) {
> throw new CacheLoaderException("Failed to load the cache"+
> e.getMessage());
>
> }
>
>
> }
>
> public static Factory<DataSource> dsdetails() throws SQLException{
> //public static DataSource dsdetails() throws SQLException{
> OracleDataSource oraDataSrc = new OracleDataSource();
> oraDataSrc.setURL("url");
> oraDataSrc.setUser("username");
> oraDataSrc.setPassword("pswd");
> //return oraDataSrc;
> return (Factory<DataSource>)oraDataSrc;
>
> }
>
> }
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by nithin91 <ni...@franklintempleton.com>.
Hi Belyakov,
Thank you so much. This is very helpful.
I am facing the following error when i am using this approach
Failed to start component: class org.apache.ignite.IgniteException: Failed
to initialize cache store (data source is not provided).
Below is the code used for implementation.I have configured the data source
property correctly but not sure why this error pops up.Can you please help
me on this.
package ignite.example.ignite_read;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.Set;
//import javax.activation.DataSource;
import javax.cache.configuration.Factory;
import javax.cache.integration.CacheLoaderException;
import javax.sql.DataSource;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteException;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.QueryEntity;
import org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory;
import org.apache.ignite.cache.store.jdbc.JdbcType;
import org.apache.ignite.cache.store.jdbc.JdbcTypeField;
import org.apache.ignite.cache.store.jdbc.dialect.OracleDialect;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import oracle.jdbc.pool.OracleDataSource;
public class IgniteCacheload {
@SuppressWarnings("unchecked")
public static void main(String[] args) throws IgniteException, SQLException
{
IgniteConfiguration config = new IgniteConfiguration();
/*
// config code
*
try (Ignite ignite = Ignition.start(config)){
CacheConfiguration<ProductKey,Products> prdCacheCfg = new
CacheConfiguration<>();
prdCacheCfg.setName("ProdCache");
prdCacheCfg.setCacheMode(CacheMode.PARTITIONED);
prdCacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
//personCacheCfg.setReadThrough(true);
//personCacheCfg.setWriteThrough(true);
CacheJdbcPojoStoreFactory<ProductKey,Products> factory = new
CacheJdbcPojoStoreFactory<>();
factory.setDialect(new OracleDialect());
//factory.setDataSource(dsdetails());
factory.setDataSourceFactory(dsdetails());
JdbcType productType = new JdbcType();
productType.setCacheName("ProdCache");
productType.setKeyType(ProductKey.class);
productType.setValueType(Products.class);
// Specify the schema if applicable
productType.setDatabaseTable("table");
productType.setKeyFields(new JdbcTypeField(java.sql.Types.VARCHAR, "fid",
ProductKey.class, "FID"));
productType.setValueFields(new JdbcTypeField(java.sql.Types.VARCHAR,
"scode", Products.class, "scode"));
factory.setTypes(productType);
prdCacheCfg.setCacheStoreFactory(factory);
config.setCacheConfiguration(prdCacheCfg);
IgniteCache<ProductKey, Products> cache =
ignite.getOrCreateCache(prdCacheCfg);
cache.clear();
// Load cache on all data nodes with default SQL statement.
System.out.println(">>> Load ALL data to cache from DB...");
cache.loadCache(null);
System.out.println(">>> Loaded cache entries: " +
cache.size());
}
catch (Exception e) {
throw new CacheLoaderException("Failed to load the cache"+
e.getMessage());
}
}
public static Factory<DataSource> dsdetails() throws SQLException{
//public static DataSource dsdetails() throws SQLException{
OracleDataSource oraDataSrc = new OracleDataSource();
oraDataSrc.setURL("url");
oraDataSrc.setUser("username");
oraDataSrc.setPassword("pswd");
//return oraDataSrc;
return (Factory<DataSource>)oraDataSrc;
}
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by Igor Belyakov <ig...@gmail.com>.
Hi,
Example of CacheJdbcPojoStore configuration via code is available here
(check "Java Configuration" tab):
https://www.gridgain.com/docs/latest/developers-guide/persistence/external-storage#cachejdbcpojostore
Regards,
Igor Belyakov
On Mon, Jan 27, 2020 at 12:44 PM nithin91 <
nithinbharadwaj.govindaraju@franklintempleton.com> wrote:
> Hi Mikael,
>
> Thanks for your quick response.
>
> I have gone through the documentation reg usage of IgniteCache.loadcache
> method.
>
> Documentation Link:
> https://apacheignite.readme.io/docs/3rd-party-store#section-loadcache-
>
> in the Documentation It was mentioned to enable the JDBC POJO store
> manually
> in the Ignite XML configuration file (or via code).
>
> Can you please provide any reference link on how to enable the JDBC POJO
> store via Code.
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by nithin91 <ni...@franklintempleton.com>.
Hi Mikael,
Thanks for your quick response.
I have gone through the documentation reg usage of IgniteCache.loadcache
method.
Documentation Link:
https://apacheignite.readme.io/docs/3rd-party-store#section-loadcache-
in the Documentation It was mentioned to enable the JDBC POJO store manually
in the Ignite XML configuration file (or via code).
Can you please provide any reference link on how to enable the JDBC POJO
store via Code.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Data Load to Ignite cache is very slow from Oracle Table
Posted by Mikael <mi...@telia.com>.
Hi!
If you use put() to insert the data it's not the fastest way, using
putAll(), IgniteCache.loadCache() or a streamer is usually much faster,
but it depends a little on how you use your data, a streamer is fast but
you can't expect all data to be available until you close or flush the
streamer, there are many examples in the documentation.
Mikael
Den 2020-01-27 kl. 09:36, skrev nithin91:
> Hi
>
> I am trying to load data from Oracle Table to Ignite Cache using Cache Store
> Load Cache method.
>
> Following is the logic implemented in Load Cache method to load the data
> from Oracle Table using Ignite Cache.
>
> 1. JDBC connector is used to connect to Oracle Table and the data is
> available in Result Set Cursor.
> 2. While loop is used to loop through the Result Set Object and insert the
> data into cache.
>
> Is there any other way to insert the data from Oracle Table to Ignite Cache.
> If possible please share sample code.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/