You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Zhengqingzheng <zh...@huawei.com> on 2016/04/13 05:34:26 UTC
cache cannot load all the data into cache
Dear all,
I am trying to load two tables data into caches to speed up my queries.
table1 contains 564 records, with one primary key as index.
definition of table content from java as follows:
@QuerySqlField
private String orgId;
@QuerySqlField(index=true)
private String objId;
@QuerySqlField
private int numRows;
table2 contains 9626 records, with no primary key defined but a group index is defined.
definition of table2 from java as follows:
@QuerySqlField
private String orgId;
@QuerySqlField(orderedGroups={@QuerySqlField.Group(
name="objId_fieldName_idx", order=0, descending = true)})
private String objId;
@QuerySqlField(orderedGroups={@QuerySqlField.Group(
name="objId_fieldName_idx", order=1, descending = true)})
private String fieldName;
@QuerySqlField
private int fieldNum;
@QuerySqlField
private int statVal;
I defined two caches to load all the data from two tables:
the first cache load data from table1, and works fine.
but the second cache which load data from table2 cannot load all the data, only few of them.
I think this is due to the configuration of cache2 was probability wrong[ because cache content shows that objid was the unique key to retrieve the data record]:
final String CACHE_NAME1= IgniteMetaDatabaseFieldStat.class.getSimpleName()+"_Cache";
CacheConfiguration<String, IgniteMetaDatabaseFieldStat> cfg =
new CacheConfiguration<String, IgniteMetaDatabaseFieldStat>(CACHE_NAME1);
CacheJdbcPojoStoreExampleFactory<String, IgniteMetaDatabaseFieldStat> storeFactory =
new CacheJdbcPojoStoreExampleFactory<String,IgniteMetaDatabaseFieldStat>();
storeFactory.setDialect(new OracleDialect());
JdbcType jdbcType = new JdbcType();
jdbcType.setCacheName(CACHE_NAME1);
jdbcType.setDatabaseSchema("besdb");
jdbcType.setDatabaseTable("data_base_field_stat");
<b>
jdbcType.setKeyType("java.lang.String");
jdbcType.setKeyFields(new JdbcTypeField(Types.VARCHAR, "OBJID", String.class, "objId")
/* ,new JdbcTypeField(Types.VARCHAR, "FIELDNAME", String.class, "fieldName")*/);
</b>
jdbcType.setValueType("org.apache.ignite.examples.model.IgniteMetaDatabaseFieldStat");
jdbcType.setValueFields(
new JdbcTypeField(Types.VARCHAR,"ORGID", String.class, "orgId"),
new JdbcTypeField(Types.VARCHAR,"OBJID", String.class, "objId"),
new JdbcTypeField(Types.VARCHAR,"FIELDNAME", String.class, "fieldName"),
new JdbcTypeField(Types.INTEGER,"FIELDNUM", Integer.class, "fieldNum"),
new JdbcTypeField(Types.INTEGER,"STAT_VAL", Integer.class, "statVal")
);
storeFactory.setTypes(jdbcType);
cfg.setCacheStoreFactory(storeFactory);
// Set atomicity as transaction, since we are showing transactions in the example.
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cfg.setIndexedTypes(String.class, IgniteMetaDatabaseFieldStat.class);
cfg.setReadThrough(true);
cfg.setWriteThrough(true);
cfg.setCacheMode(CacheMode.PARTITIONED);
//cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
//cfg.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
//cfg.setOffHeapMaxMemory(64 * 1024L * 1024L);
//cfg.setStartSize(100 * 1024 * 1024);
cfg.setBackups(0);
please note the jdbcType.setKeyTypes and setKeyFields part. I want to use the <b>groupIndex</b> as the cache key setting, which was defined in the annotation part of @QuerySqlField(orderedGroups={@QuerySqlField.Group(
name="objId_fieldName_idx", order=0, descending = true)})
private String objId;
@QuerySqlField(orderedGroups={@QuerySqlField.Group(
name="objId_fieldName_idx", order=1, descending = true)})
private String fieldName;
but I don't know how to do that, in my example I just use objid as the key. In this case, if there are duplicate values come into the cache, the rest values was ignored.
How to setup the correct key for jdbcTypes in my cache configuration?
Best regards,
Kevin
Re: cache cannot load all the data into cache
Posted by Alexey Kuznetsov <ak...@gridgain.com>.
Hi, Kevin.
Could you please make small example + db scripts to reproduce and debug
this issue?
On Wed, Apr 13, 2016 at 10:34 AM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Dear all,
> I am trying to load two tables data into caches to speed up my queries.
> table1 contains 564 records, with one primary key as index.
> definition of table content from java as follows:
> @QuerySqlField
> private String orgId;
>
> @QuerySqlField(index=true)
> private String objId;
>
> @QuerySqlField
> private int numRows;
>
> table2 contains 9626 records, with no primary key defined but a group
> index is defined.
> definition of table2 from java as follows:
> @QuerySqlField
> private String orgId;
>
> @QuerySqlField(orderedGroups={@QuerySqlField.Group(
> name="objId_fieldName_idx", order=0, descending = true)})
> private String objId;
>
> @QuerySqlField(orderedGroups={@QuerySqlField.Group(
> name="objId_fieldName_idx", order=1, descending = true)})
> private String fieldName;
>
> @QuerySqlField
> private int fieldNum;
>
> @QuerySqlField
> private int statVal;
>
> I defined two caches to load all the data from two tables:
> the first cache load data from table1, and works fine.
> but the second cache which load data from table2 cannot load all the data,
> only few of them.
> I think this is due to the configuration of cache2 was probability wrong[
> because cache content shows that objid was the unique key to retrieve the
> data record]:
>
> final String CACHE_NAME1=
> IgniteMetaDatabaseFieldStat.class.getSimpleName()+"_Cache";
> CacheConfiguration<String, IgniteMetaDatabaseFieldStat> cfg =
> new CacheConfiguration<String,
> IgniteMetaDatabaseFieldStat>(CACHE_NAME1);
>
> CacheJdbcPojoStoreExampleFactory<String,
> IgniteMetaDatabaseFieldStat> storeFactory =
> new
> CacheJdbcPojoStoreExampleFactory<String,IgniteMetaDatabaseFieldStat>();
>
> storeFactory.setDialect(new OracleDialect());
>
> JdbcType jdbcType = new JdbcType();
>
> jdbcType.setCacheName(CACHE_NAME1);
> jdbcType.setDatabaseSchema("besdb");
> jdbcType.setDatabaseTable("data_base_field_stat");
>
> <b>
> jdbcType.setKeyType("java.lang.String");
> jdbcType.setKeyFields(new JdbcTypeField(Types.VARCHAR, "OBJID",
> String.class, "objId")
> /* ,new JdbcTypeField(Types.VARCHAR, "FIELDNAME", String.class,
> "fieldName")*/);
> </b>
>
> jdbcType.setValueType("org.apache.ignite.examples.model.IgniteMetaDatabaseFieldStat");
> jdbcType.setValueFields(
> new JdbcTypeField(Types.VARCHAR,"ORGID", String.class,
> "orgId"),
> new JdbcTypeField(Types.VARCHAR,"OBJID", String.class,
> "objId"),
> new JdbcTypeField(Types.VARCHAR,"FIELDNAME", String.class,
> "fieldName"),
> new JdbcTypeField(Types.INTEGER,"FIELDNUM", Integer.class,
> "fieldNum"),
> new JdbcTypeField(Types.INTEGER,"STAT_VAL", Integer.class,
> "statVal")
> );
>
>
> storeFactory.setTypes(jdbcType);
>
> cfg.setCacheStoreFactory(storeFactory);
>
> // Set atomicity as transaction, since we are showing transactions
> in the example.
> cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cfg.setIndexedTypes(String.class,
> IgniteMetaDatabaseFieldStat.class);
>
> cfg.setReadThrough(true);
> cfg.setWriteThrough(true);
>
> cfg.setCacheMode(CacheMode.PARTITIONED);
> //cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> //cfg.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
> //cfg.setOffHeapMaxMemory(64 * 1024L * 1024L);
>
> //cfg.setStartSize(100 * 1024 * 1024);
> cfg.setBackups(0);
>
> please note the jdbcType.setKeyTypes and setKeyFields part. I want to use
> the <b>groupIndex</b> as the cache key setting, which was defined in the
> annotation part of
> @QuerySqlField(orderedGroups={@QuerySqlField.Group(
> name="objId_fieldName_idx", order=0, descending = true)})
> private String objId;
>
> @QuerySqlField(orderedGroups={@QuerySqlField.Group(
> name="objId_fieldName_idx", order=1, descending = true)})
> private String fieldName;
>
> but I don't know how to do that, in my example I just use objid as the
> key. In this case, if there are duplicate values come into the cache, the
> rest values was ignored.
>
> How to setup the correct key for jdbcTypes in my cache configuration?
>
>
> Best regards,
> Kevin
>
>
--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com