You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Lieuwe <li...@novatiq.com> on 2020/11/12 17:04:09 UTC
IgniteC++ throughput
I wonder if anyone can shed some light on the Apache Ignite performance I am
seeing.
I am running a single node & have a very simple CacheConfiguration
consisting of 4 fields.
The program is very much like the put-get-example code shipped with Ignite &
I am doing a few tests to see how fast (how many transactions per second) I
can read & write data to the cache.
1: Just incrementing the key and doing ignite::cache::Cache::Put(key,
dataObject) I can push 100K entries in the cache at about 12K TPS
2: Doing the same for ignite::cache::Cache::Get(key) yields 150K TPS
3: I then use a ignite::cache::query::SqlFieldsQuery &
ignite::cache::query::QueryFieldsCursor to do "SELECT A, B, C, D FROM
MyCache WHERE _key = ?"
Only doing cursor.isValid() && cursor.hasNext() yields 26K TPS
4: The last test I do is as above, but instead of the where clause being
'_key = ?' .. I change this to 'A=?'. In other words I use one of the fields
as a select criteria. I only get a shocking 20 TPS.
Having an index on field A makes no difference. The size of the cache does -
when I reduce that to a handful of entries that last rate will go up to
about 2K TPS.
My questions:
- There seems to be a big difference between Put & Get .. is that normal?
- There is also big difference between scenario 2 & 3 whilst they are
essentially doing the same thing .. why is SQL having so much overhead? And
example 3 doesn't even parse the columns out of the cursor whereas example 2
gives me all 4 columns for the key.
- And most importantly - why the shocking performance in scenario 4?
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteC++ throughput
Posted by Lieuwe <li...@novatiq.com>.
Ilya:
0: jdbc:ignite:thin://127.0.0.1/> EXPLAIN SELECT A, B, C, D FROM TEST.TEST
WHERE A = 1;
+--------------------------------+
| PLAN |
+--------------------------------+
| SELECT
__Z0.A AS __C0_0,
__Z0.B AS __C0_1,
__Z0.C AS __C0_2,
__Z0.D AS __C0_3
FROM TEST.TEST __Z0
/* TEST.TEST.__SCAN_ */
WHERE __Z |
| SELECT
__C0_0 AS A,
__C0_1 AS B,
__C0_2 AS C,
__C0_3 AS D
FROM PUBLIC.__T0
/* PUBLIC."merge_scan" */ |
+--------------------------------+
2 rows selected (0.006 seconds)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteC++ throughput
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please provide all rows&columns of "EXPLAIN SELECT ... WHERE A=?"
query?
Regards,
--
Ilya Kasnacheev
чт, 12 нояб. 2020 г. в 20:04, Lieuwe <li...@novatiq.com>:
> I wonder if anyone can shed some light on the Apache Ignite performance I
> am
> seeing.
>
> I am running a single node & have a very simple CacheConfiguration
> consisting of 4 fields.
>
> The program is very much like the put-get-example code shipped with Ignite
> &
> I am doing a few tests to see how fast (how many transactions per second) I
> can read & write data to the cache.
>
> 1: Just incrementing the key and doing ignite::cache::Cache::Put(key,
> dataObject) I can push 100K entries in the cache at about 12K TPS
>
> 2: Doing the same for ignite::cache::Cache::Get(key) yields 150K TPS
>
> 3: I then use a ignite::cache::query::SqlFieldsQuery &
> ignite::cache::query::QueryFieldsCursor to do "SELECT A, B, C, D FROM
> MyCache WHERE _key = ?"
> Only doing cursor.isValid() && cursor.hasNext() yields 26K TPS
>
> 4: The last test I do is as above, but instead of the where clause being
> '_key = ?' .. I change this to 'A=?'. In other words I use one of the
> fields
> as a select criteria. I only get a shocking 20 TPS.
>
> Having an index on field A makes no difference. The size of the cache does
> -
> when I reduce that to a handful of entries that last rate will go up to
> about 2K TPS.
>
>
> My questions:
> - There seems to be a big difference between Put & Get .. is that normal?
> - There is also big difference between scenario 2 & 3 whilst they are
> essentially doing the same thing .. why is SQL having so much overhead? And
> example 3 doesn't even parse the columns out of the cursor whereas example
> 2
> gives me all 4 columns for the key.
> - And most importantly - why the shocking performance in scenario 4?
>
> Thanks
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: IgniteC++ throughput
Posted by Lieuwe <li...@novatiq.com>.
Why are you using index type FULLTEXT for a Long data type? That should be
SORTED I think.
As I said - I used the example shipped with ignite as a starting point. I
changed it to SORTED and it looks better indeed. Scenario 4 now runs at
about 7K TPS.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteC++ throughput
Posted by Stephen Darlington <st...@gridgain.com>.
Why are you using index type FULLTEXT for a Long data type? That should be SORTED I think.
> On 13 Nov 2020, at 09:57, Lieuwe <li...@novatiq.com> wrote:
>
> Gangaiah - this will reproduce what I am seeing:
>
> #pragma GCC diagnostic ignored "-Wdeprecated-declarations"
> #include <ignite/ignite_configuration.h>
> #include <ignite/ignition.h>
> #include <chrono>
>
> #define CACHE_SIZE 100000
> #define CHRONO_MS std::chrono::duration_cast<std::chrono::milliseconds>
>
> struct DataObject
> {
> int64_t A, B, C;
> ignite::Guid D;
> };
>
> namespace ignite::binary
> {
> template<>
> struct BinaryType<DataObject> : BinaryTypeDefaultAll<DataObject>
> {
> static void GetTypeName(std::string &typeName) { typeName = "TEST";
> }
>
> static void Write(BinaryWriter& writer, const DataObject& obj)
> {
> writer.WriteInt64("A", obj.A);
> writer.WriteInt64("B", obj.B);
> writer.WriteInt64("C", obj.C);
> writer.WriteGuid("D", obj.D);
> }
>
> static void Read(BinaryReader& reader, DataObject& obj)
> {
> obj.A = reader.ReadInt64("A");
> obj.B = reader.ReadInt64("B");
> obj.C = reader.ReadInt64("C");
> obj.D = reader.ReadGuid("D");
> }
> };
> }
>
> int main(int arc, char* argv[])
> {
> ignite::IgniteConfiguration cfg;
> cfg.springCfgPath = "/tmp/ignite.xml";
> ignite::Ignition::Start(cfg);
>
> ignite::cache::Cache<int64_t, DataObject> DataCache =
> ignite::Ignition::Get().GetCache<int64_t, DataObject>("TEST");
>
> DataObject dataOb;
>
> auto time0 = std::chrono::high_resolution_clock::now();
>
> for(int64_t x=1;x<=CACHE_SIZE;x++)
> {
> dataOb.A = x;
> DataCache.Put(x, dataOb);
> }
>
> auto time1 = std::chrono::high_resolution_clock::now();
>
> std::cout << "Put: " << CACHE_SIZE*1000/CHRONO_MS(time1 - time0).count()
> <<" per sec" << std::endl;
>
> for(int64_t x=1;x<=CACHE_SIZE;x++)
> {
> DataObject newDataOb = DataCache.Get(x);
> assert(newDataOb.A == x);
> }
>
> auto time2 = std::chrono::high_resolution_clock::now();
>
> std::cout << "Get: " << CACHE_SIZE*1000/CHRONO_MS(time2 - time1).count()
> <<" per sec" << std::endl;
>
> ignite::cache::query::SqlFieldsQuery query1("SELECT A, B, C, D FROM TEST
> WHERE _key = ?");
>
> for(int64_t x=1;x<=CACHE_SIZE;x++)
> {
> query1.AddArgument<int64_t>(x);
> ignite::cache::query::QueryFieldsCursor cursor =
> DataCache.Query(query1);
> assert(cursor.IsValid() && cursor.HasNext());
> query1.ClearArguments();
> }
>
> auto time3 = std::chrono::high_resolution_clock::now();
>
> std::cout << "Select1: " << CACHE_SIZE*1000/CHRONO_MS(time3 -
> time2).count() << " per sec" << std::endl;
>
> ignite::cache::query::SqlFieldsQuery query2("SELECT A, B, C, D FROM TEST
> WHERE A = ?");
>
> for(int64_t x=1;x<=1000;x++)
> {
> query2.AddArgument<int64_t>(x);
> ignite::cache::query::QueryFieldsCursor cursor =
> DataCache.Query(query2);
> assert(cursor.IsValid() && cursor.HasNext());
> query2.ClearArguments();
> }
>
> auto time4 = std::chrono::high_resolution_clock::now();
>
> std::cout << "Select2: " << 1000*1000/CHRONO_MS(time4 - time3).count()
> <<" per sec" << std::endl;
>
> ignite::Ignition::StopAll(false);
>
> return 0;
> }
>
>
>
> The xml I use looks like this:
>
> <property name="cacheConfiguration">
> <list>
> <bean
> class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="TEST"/>
> <property name="cacheMode" value="PARTITIONED"/>
> <property name="atomicityMode" value="TRANSACTIONAL"/>
> <property name="writeSynchronizationMode"
> value="FULL_SYNC"/>
>
> <property name="queryEntities">
> <list>
> <bean
> class="org.apache.ignite.cache.QueryEntity">
> <property name="keyType"
> value="java.lang.Long"/>
> <property name="valueType" value="TEST"/>
>
> <property name="fields">
> <map>
> <entry key="A"
> value="java.lang.Long"/>
> <entry key="B"
> value="java.lang.Long"/>
> <entry key="C"
> value="java.lang.Long"/>
> <entry key="D"
> value="java.util.UUID"/>
> </map>
> </property>
>
> <property name="indexes">
> <list>
> <bean
> class="org.apache.ignite.cache.QueryIndex">
> <property name="fields">
> <map>
> <entry key="A"
> value="true"/>
> </map>
> </property>
> <property name="indexType"
> value="FULLTEXT"/>
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
> </bean>
> </list>
> </property>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteC++ throughput
Posted by Lieuwe <li...@novatiq.com>.
Gangaiah - this will reproduce what I am seeing:
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
#include <ignite/ignite_configuration.h>
#include <ignite/ignition.h>
#include <chrono>
#define CACHE_SIZE 100000
#define CHRONO_MS std::chrono::duration_cast<std::chrono::milliseconds>
struct DataObject
{
int64_t A, B, C;
ignite::Guid D;
};
namespace ignite::binary
{
template<>
struct BinaryType<DataObject> : BinaryTypeDefaultAll<DataObject>
{
static void GetTypeName(std::string &typeName) { typeName = "TEST";
}
static void Write(BinaryWriter& writer, const DataObject& obj)
{
writer.WriteInt64("A", obj.A);
writer.WriteInt64("B", obj.B);
writer.WriteInt64("C", obj.C);
writer.WriteGuid("D", obj.D);
}
static void Read(BinaryReader& reader, DataObject& obj)
{
obj.A = reader.ReadInt64("A");
obj.B = reader.ReadInt64("B");
obj.C = reader.ReadInt64("C");
obj.D = reader.ReadGuid("D");
}
};
}
int main(int arc, char* argv[])
{
ignite::IgniteConfiguration cfg;
cfg.springCfgPath = "/tmp/ignite.xml";
ignite::Ignition::Start(cfg);
ignite::cache::Cache<int64_t, DataObject> DataCache =
ignite::Ignition::Get().GetCache<int64_t, DataObject>("TEST");
DataObject dataOb;
auto time0 = std::chrono::high_resolution_clock::now();
for(int64_t x=1;x<=CACHE_SIZE;x++)
{
dataOb.A = x;
DataCache.Put(x, dataOb);
}
auto time1 = std::chrono::high_resolution_clock::now();
std::cout << "Put: " << CACHE_SIZE*1000/CHRONO_MS(time1 - time0).count()
<<" per sec" << std::endl;
for(int64_t x=1;x<=CACHE_SIZE;x++)
{
DataObject newDataOb = DataCache.Get(x);
assert(newDataOb.A == x);
}
auto time2 = std::chrono::high_resolution_clock::now();
std::cout << "Get: " << CACHE_SIZE*1000/CHRONO_MS(time2 - time1).count()
<<" per sec" << std::endl;
ignite::cache::query::SqlFieldsQuery query1("SELECT A, B, C, D FROM TEST
WHERE _key = ?");
for(int64_t x=1;x<=CACHE_SIZE;x++)
{
query1.AddArgument<int64_t>(x);
ignite::cache::query::QueryFieldsCursor cursor =
DataCache.Query(query1);
assert(cursor.IsValid() && cursor.HasNext());
query1.ClearArguments();
}
auto time3 = std::chrono::high_resolution_clock::now();
std::cout << "Select1: " << CACHE_SIZE*1000/CHRONO_MS(time3 -
time2).count() << " per sec" << std::endl;
ignite::cache::query::SqlFieldsQuery query2("SELECT A, B, C, D FROM TEST
WHERE A = ?");
for(int64_t x=1;x<=1000;x++)
{
query2.AddArgument<int64_t>(x);
ignite::cache::query::QueryFieldsCursor cursor =
DataCache.Query(query2);
assert(cursor.IsValid() && cursor.HasNext());
query2.ClearArguments();
}
auto time4 = std::chrono::high_resolution_clock::now();
std::cout << "Select2: " << 1000*1000/CHRONO_MS(time4 - time3).count()
<<" per sec" << std::endl;
ignite::Ignition::StopAll(false);
return 0;
}
The xml I use looks like this:
<property name="cacheConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="TEST"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="writeSynchronizationMode"
value="FULL_SYNC"/>
<property name="queryEntities">
<list>
<bean
class="org.apache.ignite.cache.QueryEntity">
<property name="keyType"
value="java.lang.Long"/>
<property name="valueType" value="TEST"/>
<property name="fields">
<map>
<entry key="A"
value="java.lang.Long"/>
<entry key="B"
value="java.lang.Long"/>
<entry key="C"
value="java.lang.Long"/>
<entry key="D"
value="java.util.UUID"/>
</map>
</property>
<property name="indexes">
<list>
<bean
class="org.apache.ignite.cache.QueryIndex">
<property name="fields">
<map>
<entry key="A"
value="true"/>
</map>
</property>
<property name="indexType"
value="FULLTEXT"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteC++ throughput
Posted by Gangaiah Gundeboina <ga...@gmail.com>.
Hi Lieuwe,
To understand this in detail, please give more details like how you are
calculating TPS and code snippet for each one where you are doing
put/get/cursor.hasNext().
Regards,
Gangaiah
-----
Thanks and Regards,
Gangaiah
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/