You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Denis Magda <dm...@apache.org> on 2020/06/10 23:28:19 UTC
Re: Ignite concepts
Hi Eugene,
Let me help you with that as much as I can. Please help me understand the
following:
- Is the cluster supposed to be deployed outside of Kubernetes (on VMs
or bare-metal)? Is this a private cloud (OpenShift, PCF) or a public
environment (Azure, AWS, etc.)?
- If you use primary keys to access the records, do you really need SQL
for this type of operations? Or do you perform additional filtering of
records with SQL operands?
In the meantime, please check this page to understand the pros and cons of
thick and thin clients:
https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#thick-vs-thin-clients
-
Denis
On Wed, Jun 10, 2020 at 1:07 PM Eugene McGowan <eu...@gmail.com>
wrote:
> Hi Igniters,
> Our company has recently started using Ignite and our team are now to it.
> We would like to use Ignite as a Cache layer for an upcoming project
>
> Our use case is that we have an Ignite cluster that we want to interact
> with , our client service is Springboot based and running in a Kubernetes
> Cluster but not co located with our Ignite cluster , we anticipate our
> service will receive *3000+ request per second* , the service will scale
> based on the volume of requests, and we will need to query Ignite for each
> of these requests.
> The data we need to query is fairly straight forward , a single row from
> either 1 or 2 caches based on primary key.
> We are at the point where we are beginning to design the architecture.
>
> I have done some initial reading on Ignite and was wondering if someone
> could
> Verify my understanding is correct.
>
> I would also be interested to know what architectures other folks are
> using for similar use cases
>
> *=======*
> *Thin Client*
> ========
> JDBC connection are not thread safe
> https://apacheignite-sql.readme.io/docs/jdbc-driver
> To use this approach we need to ensure non cuncurrent use of
> Connections, Statements, and ResultSet perhaps using a locking mechanism.
>
> Key value operations or SQL Queries via the API over the thin client are
> thread safe
> Connection Pooling is not provided via the API , but it is possible to
> implement your own connection pool.
> If you implement your own connection pool you could keep connections
> alive to reduce
> the cost of establishing a connection to Ignite and reusing
> the connections for various requests.
>
>
> ==========
> *Thick Client *
> ==========
>
> Does not have a concept of Connection pooling
>
> When using either JDBC or the IGNITE API on the Thick Client
> It is thread safe and can handle multiple concurrent threads
>
> Is part of the Ignite cluster
> Ignite Servers can iniatate TCP connection needs to be reachable via a
> TCP connection
> If the Client is behind a firewall,NAT or Load Balancer it makes using
> ,thick client challenging
>
>
>
> =========
> *Question on GRPC*
> =============
>
> The Thin Client TCP socket connection uses a Binary protocol
> https://apacheignite.readme.io/docs/binary-client-protocol
>
> I may be mixing up network levels here but , is it possible to run this
> over a GRPC connection
>
Re: Ignite concepts
Posted by Denis Magda <dm...@apache.org>.
Eugene,
> Also it looks like prior to 2.8.0 the Java thin Client was thread safe but
> single threaded , but since 2.8.0 the Java thin client is Multithreaded, (I
> have a book that mentioned it was not multithread, which confused me).
> Is this correct and why you suggest I only need one connection per service?
@Igor Sapego <is...@apache.org>, @Pavel Tupitsyn <pt...@apache.org>,
could you please shed some light on this question? Did the Java thin client
become multi-threaded in Ignite 2.8.0?
As for a single connection per an application service, here is my logic.
It's a proxy server, a thin client is connected to, that does most of a
heavy-lifting job
<https://www.gridgain.com/docs/latest/developers-guide/thin-clients/getting-started-with-thin-clients#partition-awareness>.
The client merely issues queries while the server takes care of their
forwarding across the cluster, result sets collection, and other tasks.
That proxy server gets offloaded for key-value requests only if the
partition-awareness is enabled - in this case, the client connects to
primary nodes directly when the application issues key-value commands.
Overall, to avoid possible bottlenecks try to follow this pattern:
- The partition-awareness is disabled. It means a proxy server can
become a bottleneck easily. As remediation, have all your thin clients
connected to different proxy servers initially to spread the load.
- The partition-awareness is enabled. The key-value calls will be
directed to primary nodes right from a thin client connection. However, a
proxy server still can become a bottleneck if the client executes many SQL
queries.
As a rule of thumb is it better more preformanent to use gets and puts as
> opposed to SQL , only use the SQL API if you need to do something that
> requires SQL?
Yes, key-value calls are more lightweight and would perform faster in
most scenarios.
-
Denis
On Fri, Jun 12, 2020 at 5:11 AM Eugene McGowan <eu...@gmail.com>
wrote:
> Hi Dennis ,
>
> Again thanks for you feedback and links these are very helpful.
>
> Also it looks like prior to 2.8.0 the Java thin Client was thread safe but
> single threaded , but since 2.8.0 the Java thin client is Multithreaded, (I
> have a book that mentioned it was not multithread, which confused me).
> Is this correct and why you suggest I only need one connection per service?
>
> As a rule of thumb is it better more preformanent to use gets and puts as
> opposed to SQL , only use the SQL API if you need to do something that
> requires SQL?
>
> Regarding the 3 different caches problem , I did a bit of research on this
> and think I am good with this essentially by not using the try and
> autoclose with the igniteClient I can control when the igniteClient
> object is closed and therefore use it for all 3 Cache checks.
>
> Regards Eugene
>
> On Thu, Jun 11, 2020 at 10:57 PM Denis Magda <dm...@apache.org> wrote:
>
>> Eugene,
>>
>> In such a case, I would go for the thin client for several reasons:
>>
>> - With the thick client (aka. standard client), you will come across
>> several limitations if the servers are deployed on VMs, but the thick
>> clients are running in a K8S environment. Those limitations are to be
>> addressed soon [1].
>> - With the partition-awareness feature [2], the thin client will be
>> able to send requests to those server nodes that store the primary copy of
>> a requested record. The functionality should be released in Ignite 2.9 and
>> already exists in GridGain.
>>
>> I don't think that you need to be bothered about a connection pool. Just
>> have a single thin client connection per service instead and enable the
>> partition-awareness. If you don't use the latter feature, then ensure that
>> the thin clients on those service instances are connected to random proxy
>> servers (you need to pass different IPs of the server nodes in the
>> connection string parameter [3]).
>>
>> One thing I am not clear on is in the scenario where we need to check 3
>>> different caches, for a single call to our service, is this going to be an
>>> expensive operation?
>>> Will we have to initiate a connection to Ignite for every call to the
>>> cache?
>>
>>
>> To help with this, I need to understand your business use case better.
>> There is a chance you can do all the checks with one network hop.
>>
>> [1]
>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html
>> [2]
>> https://www.gridgain.com/docs/latest/developers-guide/thin-clients/getting-started-with-thin-clients#partition-awareness
>> [3]
>> https://apacheignite.readme.io/docs/java-thin-client-initialization-and-configuration#initialization
>>
>> -
>> Denis
>>
>>
>> On Thu, Jun 11, 2020 at 12:12 AM Eugene McGowan <eu...@gmail.com>
>> wrote:
>>
>>> Thanks for getting back to me on this Dennis,
>>> The company's Ignite Cluster is already up and running on VMs on our
>>> company network. The service my team is standing up will run in our Private
>>> cloud K8S environment.
>>>
>>> Regarding our usecase we will have say 2 or 3 caches, our service will
>>> be passed 3 parameters , if it gets a match to parameter 1 in cache 1 no
>>> further action is required , if not it needs to check cache 2 for parameter
>>> 2 ,again if it gets a match it uses data from cache 2 and no further
>>> action is required , otherwise it needs to check cache 3 . So cache 1 has
>>> most relevant data.
>>> We would have somewhere between 2 and 10 instances of our service
>>> running servicing a max of 3000+ request per second
>>>
>>> Our preference if possible would be to use a thin client. One thing I am
>>> not clear on is in the scenario where we need to check 3 different caches,
>>> for a single call to our service, is this going to be an expensive
>>> operation?
>>> Will we have to initiate a connection to Ignite for every call to the
>>> cache? From reading connection pooling does not come out of the box with
>>> key value operations over thin client, but its possible to implement your
>>> own connection pooling, is it therefore possible to have long lived
>>> connections to avoid the expense of constantly connecting and disconnecting
>>> from Ignite caches.
>>>
>>> Regarding the possibility of using the SQL API the background that is:
>>> (1) We have a rarely used usecase where we want to get a
>>> combination of data from all 3 caches , so was thinking the SQL API would
>>> be one way of doing that.
>>> (2) Looking at the docs it seems that connection pooling is
>>> possible via JDBC over thin client , so was wondering would that be a more
>>> performant approach, if it's not possible to have a connection pool for key
>>> value operations over the thin client.
>>>
>>> Regards Eugene
>>>
>>> On Thu, Jun 11, 2020 at 12:28 AM Denis Magda <dm...@apache.org> wrote:
>>>
>>>> Hi Eugene,
>>>>
>>>> Let me help you with that as much as I can. Please help me understand
>>>> the following:
>>>>
>>>> - Is the cluster supposed to be deployed outside of Kubernetes (on
>>>> VMs or bare-metal)? Is this a private cloud (OpenShift, PCF) or a public
>>>> environment (Azure, AWS, etc.)?
>>>> - If you use primary keys to access the records, do you really need
>>>> SQL for this type of operations? Or do you perform additional filtering of
>>>> records with SQL operands?
>>>>
>>>> In the meantime, please check this page to understand the pros and cons
>>>> of thick and thin clients:
>>>> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#thick-vs-thin-clients
>>>>
>>>> -
>>>> Denis
>>>>
>>>>
>>>> On Wed, Jun 10, 2020 at 1:07 PM Eugene McGowan <
>>>> eugenemacgowan@gmail.com> wrote:
>>>>
>>>>> Hi Igniters,
>>>>> Our company has recently started using Ignite and our team are now to
>>>>> it.
>>>>> We would like to use Ignite as a Cache layer for an upcoming project
>>>>>
>>>>> Our use case is that we have an Ignite cluster that we want to
>>>>> interact with , our client service is Springboot based and running in a
>>>>> Kubernetes Cluster but not co located with our Ignite cluster , we
>>>>> anticipate our service will receive *3000+ request per second* , the
>>>>> service will scale based on the volume of requests, and we will need to
>>>>> query Ignite for each of these requests.
>>>>> The data we need to query is fairly straight forward , a single row
>>>>> from either 1 or 2 caches based on primary key.
>>>>> We are at the point where we are beginning to design the architecture.
>>>>>
>>>>> I have done some initial reading on Ignite and was wondering if
>>>>> someone could
>>>>> Verify my understanding is correct.
>>>>>
>>>>> I would also be interested to know what architectures other folks
>>>>> are using for similar use cases
>>>>>
>>>>> *=======*
>>>>> *Thin Client*
>>>>> ========
>>>>> JDBC connection are not thread safe
>>>>> https://apacheignite-sql.readme.io/docs/jdbc-driver
>>>>> To use this approach we need to ensure non cuncurrent use of
>>>>> Connections, Statements, and ResultSet perhaps using a locking mechanism.
>>>>>
>>>>> Key value operations or SQL Queries via the API over the thin client
>>>>> are thread safe
>>>>> Connection Pooling is not provided via the API , but it is possible
>>>>> to implement your own connection pool.
>>>>> If you implement your own connection pool you could keep connections
>>>>> alive to reduce
>>>>> the cost of establishing a connection to Ignite and reusing
>>>>> the connections for various requests.
>>>>>
>>>>>
>>>>> ==========
>>>>> *Thick Client *
>>>>> ==========
>>>>>
>>>>> Does not have a concept of Connection pooling
>>>>>
>>>>> When using either JDBC or the IGNITE API on the Thick Client
>>>>> It is thread safe and can handle multiple concurrent threads
>>>>>
>>>>> Is part of the Ignite cluster
>>>>> Ignite Servers can iniatate TCP connection needs to be reachable
>>>>> via a TCP connection
>>>>> If the Client is behind a firewall,NAT or Load Balancer it makes
>>>>> using ,thick client challenging
>>>>>
>>>>>
>>>>>
>>>>> =========
>>>>> *Question on GRPC*
>>>>> =============
>>>>>
>>>>> The Thin Client TCP socket connection uses a Binary protocol
>>>>> https://apacheignite.readme.io/docs/binary-client-protocol
>>>>>
>>>>> I may be mixing up network levels here but , is it possible to run
>>>>> this over a GRPC connection
>>>>>
>>>>
Re: Ignite concepts
Posted by Eugene McGowan <eu...@gmail.com>.
Hi Dennis ,
Again thanks for you feedback and links these are very helpful.
Also it looks like prior to 2.8.0 the Java thin Client was thread safe but
single threaded , but since 2.8.0 the Java thin client is Multithreaded, (I
have a book that mentioned it was not multithread, which confused me).
Is this correct and why you suggest I only need one connection per service?
As a rule of thumb is it better more preformanent to use gets and puts as
opposed to SQL , only use the SQL API if you need to do something that
requires SQL?
Regarding the 3 different caches problem , I did a bit of research on this
and think I am good with this essentially by not using the try and
autoclose with the igniteClient I can control when the igniteClient object
is closed and therefore use it for all 3 Cache checks.
Regards Eugene
On Thu, Jun 11, 2020 at 10:57 PM Denis Magda <dm...@apache.org> wrote:
> Eugene,
>
> In such a case, I would go for the thin client for several reasons:
>
> - With the thick client (aka. standard client), you will come across
> several limitations if the servers are deployed on VMs, but the thick
> clients are running in a K8S environment. Those limitations are to be
> addressed soon [1].
> - With the partition-awareness feature [2], the thin client will be
> able to send requests to those server nodes that store the primary copy of
> a requested record. The functionality should be released in Ignite 2.9 and
> already exists in GridGain.
>
> I don't think that you need to be bothered about a connection pool. Just
> have a single thin client connection per service instead and enable the
> partition-awareness. If you don't use the latter feature, then ensure that
> the thin clients on those service instances are connected to random proxy
> servers (you need to pass different IPs of the server nodes in the
> connection string parameter [3]).
>
> One thing I am not clear on is in the scenario where we need to check 3
>> different caches, for a single call to our service, is this going to be an
>> expensive operation?
>> Will we have to initiate a connection to Ignite for every call to the
>> cache?
>
>
> To help with this, I need to understand your business use case better.
> There is a chance you can do all the checks with one network hop.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html
> [2]
> https://www.gridgain.com/docs/latest/developers-guide/thin-clients/getting-started-with-thin-clients#partition-awareness
> [3]
> https://apacheignite.readme.io/docs/java-thin-client-initialization-and-configuration#initialization
>
> -
> Denis
>
>
> On Thu, Jun 11, 2020 at 12:12 AM Eugene McGowan <eu...@gmail.com>
> wrote:
>
>> Thanks for getting back to me on this Dennis,
>> The company's Ignite Cluster is already up and running on VMs on our
>> company network. The service my team is standing up will run in our Private
>> cloud K8S environment.
>>
>> Regarding our usecase we will have say 2 or 3 caches, our service will be
>> passed 3 parameters , if it gets a match to parameter 1 in cache 1 no
>> further action is required , if not it needs to check cache 2 for parameter
>> 2 ,again if it gets a match it uses data from cache 2 and no further
>> action is required , otherwise it needs to check cache 3 . So cache 1 has
>> most relevant data.
>> We would have somewhere between 2 and 10 instances of our service running
>> servicing a max of 3000+ request per second
>>
>> Our preference if possible would be to use a thin client. One thing I am
>> not clear on is in the scenario where we need to check 3 different caches,
>> for a single call to our service, is this going to be an expensive
>> operation?
>> Will we have to initiate a connection to Ignite for every call to the
>> cache? From reading connection pooling does not come out of the box with
>> key value operations over thin client, but its possible to implement your
>> own connection pooling, is it therefore possible to have long lived
>> connections to avoid the expense of constantly connecting and disconnecting
>> from Ignite caches.
>>
>> Regarding the possibility of using the SQL API the background that is:
>> (1) We have a rarely used usecase where we want to get a
>> combination of data from all 3 caches , so was thinking the SQL API would
>> be one way of doing that.
>> (2) Looking at the docs it seems that connection pooling is
>> possible via JDBC over thin client , so was wondering would that be a more
>> performant approach, if it's not possible to have a connection pool for key
>> value operations over the thin client.
>>
>> Regards Eugene
>>
>> On Thu, Jun 11, 2020 at 12:28 AM Denis Magda <dm...@apache.org> wrote:
>>
>>> Hi Eugene,
>>>
>>> Let me help you with that as much as I can. Please help me understand
>>> the following:
>>>
>>> - Is the cluster supposed to be deployed outside of Kubernetes (on
>>> VMs or bare-metal)? Is this a private cloud (OpenShift, PCF) or a public
>>> environment (Azure, AWS, etc.)?
>>> - If you use primary keys to access the records, do you really need
>>> SQL for this type of operations? Or do you perform additional filtering of
>>> records with SQL operands?
>>>
>>> In the meantime, please check this page to understand the pros and cons
>>> of thick and thin clients:
>>> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#thick-vs-thin-clients
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Wed, Jun 10, 2020 at 1:07 PM Eugene McGowan <eu...@gmail.com>
>>> wrote:
>>>
>>>> Hi Igniters,
>>>> Our company has recently started using Ignite and our team are now to
>>>> it.
>>>> We would like to use Ignite as a Cache layer for an upcoming project
>>>>
>>>> Our use case is that we have an Ignite cluster that we want to interact
>>>> with , our client service is Springboot based and running in a Kubernetes
>>>> Cluster but not co located with our Ignite cluster , we anticipate our
>>>> service will receive *3000+ request per second* , the service will
>>>> scale based on the volume of requests, and we will need to query Ignite for
>>>> each of these requests.
>>>> The data we need to query is fairly straight forward , a single row
>>>> from either 1 or 2 caches based on primary key.
>>>> We are at the point where we are beginning to design the architecture.
>>>>
>>>> I have done some initial reading on Ignite and was wondering if someone
>>>> could
>>>> Verify my understanding is correct.
>>>>
>>>> I would also be interested to know what architectures other folks are
>>>> using for similar use cases
>>>>
>>>> *=======*
>>>> *Thin Client*
>>>> ========
>>>> JDBC connection are not thread safe
>>>> https://apacheignite-sql.readme.io/docs/jdbc-driver
>>>> To use this approach we need to ensure non cuncurrent use of
>>>> Connections, Statements, and ResultSet perhaps using a locking mechanism.
>>>>
>>>> Key value operations or SQL Queries via the API over the thin client
>>>> are thread safe
>>>> Connection Pooling is not provided via the API , but it is possible
>>>> to implement your own connection pool.
>>>> If you implement your own connection pool you could keep connections
>>>> alive to reduce
>>>> the cost of establishing a connection to Ignite and reusing
>>>> the connections for various requests.
>>>>
>>>>
>>>> ==========
>>>> *Thick Client *
>>>> ==========
>>>>
>>>> Does not have a concept of Connection pooling
>>>>
>>>> When using either JDBC or the IGNITE API on the Thick Client
>>>> It is thread safe and can handle multiple concurrent threads
>>>>
>>>> Is part of the Ignite cluster
>>>> Ignite Servers can iniatate TCP connection needs to be reachable via
>>>> a TCP connection
>>>> If the Client is behind a firewall,NAT or Load Balancer it makes
>>>> using ,thick client challenging
>>>>
>>>>
>>>>
>>>> =========
>>>> *Question on GRPC*
>>>> =============
>>>>
>>>> The Thin Client TCP socket connection uses a Binary protocol
>>>> https://apacheignite.readme.io/docs/binary-client-protocol
>>>>
>>>> I may be mixing up network levels here but , is it possible to run this
>>>> over a GRPC connection
>>>>
>>>
Re: Ignite concepts
Posted by Denis Magda <dm...@apache.org>.
Eugene,
In such a case, I would go for the thin client for several reasons:
- With the thick client (aka. standard client), you will come across
several limitations if the servers are deployed on VMs, but the thick
clients are running in a K8S environment. Those limitations are to be
addressed soon [1].
- With the partition-awareness feature [2], the thin client will be able
to send requests to those server nodes that store the primary copy of a
requested record. The functionality should be released in Ignite 2.9 and
already exists in GridGain.
I don't think that you need to be bothered about a connection pool. Just
have a single thin client connection per service instead and enable the
partition-awareness. If you don't use the latter feature, then ensure that
the thin clients on those service instances are connected to random proxy
servers (you need to pass different IPs of the server nodes in the
connection string parameter [3]).
One thing I am not clear on is in the scenario where we need to check 3
> different caches, for a single call to our service, is this going to be an
> expensive operation?
> Will we have to initiate a connection to Ignite for every call to the
> cache?
To help with this, I need to understand your business use case better.
There is a chance you can do all the checks with one network hop.
[1]
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html
[2]
https://www.gridgain.com/docs/latest/developers-guide/thin-clients/getting-started-with-thin-clients#partition-awareness
[3]
https://apacheignite.readme.io/docs/java-thin-client-initialization-and-configuration#initialization
-
Denis
On Thu, Jun 11, 2020 at 12:12 AM Eugene McGowan <eu...@gmail.com>
wrote:
> Thanks for getting back to me on this Dennis,
> The company's Ignite Cluster is already up and running on VMs on our
> company network. The service my team is standing up will run in our Private
> cloud K8S environment.
>
> Regarding our usecase we will have say 2 or 3 caches, our service will be
> passed 3 parameters , if it gets a match to parameter 1 in cache 1 no
> further action is required , if not it needs to check cache 2 for parameter
> 2 ,again if it gets a match it uses data from cache 2 and no further
> action is required , otherwise it needs to check cache 3 . So cache 1 has
> most relevant data.
> We would have somewhere between 2 and 10 instances of our service running
> servicing a max of 3000+ request per second
>
> Our preference if possible would be to use a thin client. One thing I am
> not clear on is in the scenario where we need to check 3 different caches,
> for a single call to our service, is this going to be an expensive
> operation?
> Will we have to initiate a connection to Ignite for every call to the
> cache? From reading connection pooling does not come out of the box with
> key value operations over thin client, but its possible to implement your
> own connection pooling, is it therefore possible to have long lived
> connections to avoid the expense of constantly connecting and disconnecting
> from Ignite caches.
>
> Regarding the possibility of using the SQL API the background that is:
> (1) We have a rarely used usecase where we want to get a combination
> of data from all 3 caches , so was thinking the SQL API would be one way of
> doing that.
> (2) Looking at the docs it seems that connection pooling is possible
> via JDBC over thin client , so was wondering would that be a more
> performant approach, if it's not possible to have a connection pool for key
> value operations over the thin client.
>
> Regards Eugene
>
> On Thu, Jun 11, 2020 at 12:28 AM Denis Magda <dm...@apache.org> wrote:
>
>> Hi Eugene,
>>
>> Let me help you with that as much as I can. Please help me understand the
>> following:
>>
>> - Is the cluster supposed to be deployed outside of Kubernetes (on
>> VMs or bare-metal)? Is this a private cloud (OpenShift, PCF) or a public
>> environment (Azure, AWS, etc.)?
>> - If you use primary keys to access the records, do you really need
>> SQL for this type of operations? Or do you perform additional filtering of
>> records with SQL operands?
>>
>> In the meantime, please check this page to understand the pros and cons
>> of thick and thin clients:
>> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#thick-vs-thin-clients
>>
>> -
>> Denis
>>
>>
>> On Wed, Jun 10, 2020 at 1:07 PM Eugene McGowan <eu...@gmail.com>
>> wrote:
>>
>>> Hi Igniters,
>>> Our company has recently started using Ignite and our team are now to it.
>>> We would like to use Ignite as a Cache layer for an upcoming project
>>>
>>> Our use case is that we have an Ignite cluster that we want to interact
>>> with , our client service is Springboot based and running in a Kubernetes
>>> Cluster but not co located with our Ignite cluster , we anticipate our
>>> service will receive *3000+ request per second* , the service will
>>> scale based on the volume of requests, and we will need to query Ignite for
>>> each of these requests.
>>> The data we need to query is fairly straight forward , a single row from
>>> either 1 or 2 caches based on primary key.
>>> We are at the point where we are beginning to design the architecture.
>>>
>>> I have done some initial reading on Ignite and was wondering if someone
>>> could
>>> Verify my understanding is correct.
>>>
>>> I would also be interested to know what architectures other folks are
>>> using for similar use cases
>>>
>>> *=======*
>>> *Thin Client*
>>> ========
>>> JDBC connection are not thread safe
>>> https://apacheignite-sql.readme.io/docs/jdbc-driver
>>> To use this approach we need to ensure non cuncurrent use of
>>> Connections, Statements, and ResultSet perhaps using a locking mechanism.
>>>
>>> Key value operations or SQL Queries via the API over the thin client
>>> are thread safe
>>> Connection Pooling is not provided via the API , but it is possible to
>>> implement your own connection pool.
>>> If you implement your own connection pool you could keep connections
>>> alive to reduce
>>> the cost of establishing a connection to Ignite and reusing
>>> the connections for various requests.
>>>
>>>
>>> ==========
>>> *Thick Client *
>>> ==========
>>>
>>> Does not have a concept of Connection pooling
>>>
>>> When using either JDBC or the IGNITE API on the Thick Client
>>> It is thread safe and can handle multiple concurrent threads
>>>
>>> Is part of the Ignite cluster
>>> Ignite Servers can iniatate TCP connection needs to be reachable via
>>> a TCP connection
>>> If the Client is behind a firewall,NAT or Load Balancer it makes using
>>> ,thick client challenging
>>>
>>>
>>>
>>> =========
>>> *Question on GRPC*
>>> =============
>>>
>>> The Thin Client TCP socket connection uses a Binary protocol
>>> https://apacheignite.readme.io/docs/binary-client-protocol
>>>
>>> I may be mixing up network levels here but , is it possible to run this
>>> over a GRPC connection
>>>
>>
Re: Ignite concepts
Posted by Eugene McGowan <eu...@gmail.com>.
Thanks for getting back to me on this Dennis,
The company's Ignite Cluster is already up and running on VMs on our
company network. The service my team is standing up will run in our Private
cloud K8S environment.
Regarding our usecase we will have say 2 or 3 caches, our service will be
passed 3 parameters , if it gets a match to parameter 1 in cache 1 no
further action is required , if not it needs to check cache 2 for parameter
2 ,again if it gets a match it uses data from cache 2 and no further
action is required , otherwise it needs to check cache 3 . So cache 1 has
most relevant data.
We would have somewhere between 2 and 10 instances of our service running
servicing a max of 3000+ request per second
Our preference if possible would be to use a thin client. One thing I am
not clear on is in the scenario where we need to check 3 different caches,
for a single call to our service, is this going to be an expensive
operation?
Will we have to initiate a connection to Ignite for every call to the
cache? From reading connection pooling does not come out of the box with
key value operations over thin client, but its possible to implement your
own connection pooling, is it therefore possible to have long lived
connections to avoid the expense of constantly connecting and disconnecting
from Ignite caches.
Regarding the possibility of using the SQL API the background that is:
(1) We have a rarely used usecase where we want to get a combination
of data from all 3 caches , so was thinking the SQL API would be one way of
doing that.
(2) Looking at the docs it seems that connection pooling is possible
via JDBC over thin client , so was wondering would that be a more
performant approach, if it's not possible to have a connection pool for key
value operations over the thin client.
Regards Eugene
On Thu, Jun 11, 2020 at 12:28 AM Denis Magda <dm...@apache.org> wrote:
> Hi Eugene,
>
> Let me help you with that as much as I can. Please help me understand the
> following:
>
> - Is the cluster supposed to be deployed outside of Kubernetes (on VMs
> or bare-metal)? Is this a private cloud (OpenShift, PCF) or a public
> environment (Azure, AWS, etc.)?
> - If you use primary keys to access the records, do you really need
> SQL for this type of operations? Or do you perform additional filtering of
> records with SQL operands?
>
> In the meantime, please check this page to understand the pros and cons of
> thick and thin clients:
> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#thick-vs-thin-clients
>
> -
> Denis
>
>
> On Wed, Jun 10, 2020 at 1:07 PM Eugene McGowan <eu...@gmail.com>
> wrote:
>
>> Hi Igniters,
>> Our company has recently started using Ignite and our team are now to it.
>> We would like to use Ignite as a Cache layer for an upcoming project
>>
>> Our use case is that we have an Ignite cluster that we want to interact
>> with , our client service is Springboot based and running in a Kubernetes
>> Cluster but not co located with our Ignite cluster , we anticipate our
>> service will receive *3000+ request per second* , the service will scale
>> based on the volume of requests, and we will need to query Ignite for each
>> of these requests.
>> The data we need to query is fairly straight forward , a single row from
>> either 1 or 2 caches based on primary key.
>> We are at the point where we are beginning to design the architecture.
>>
>> I have done some initial reading on Ignite and was wondering if someone
>> could
>> Verify my understanding is correct.
>>
>> I would also be interested to know what architectures other folks are
>> using for similar use cases
>>
>> *=======*
>> *Thin Client*
>> ========
>> JDBC connection are not thread safe
>> https://apacheignite-sql.readme.io/docs/jdbc-driver
>> To use this approach we need to ensure non cuncurrent use of
>> Connections, Statements, and ResultSet perhaps using a locking mechanism.
>>
>> Key value operations or SQL Queries via the API over the thin client
>> are thread safe
>> Connection Pooling is not provided via the API , but it is possible to
>> implement your own connection pool.
>> If you implement your own connection pool you could keep connections
>> alive to reduce
>> the cost of establishing a connection to Ignite and reusing
>> the connections for various requests.
>>
>>
>> ==========
>> *Thick Client *
>> ==========
>>
>> Does not have a concept of Connection pooling
>>
>> When using either JDBC or the IGNITE API on the Thick Client
>> It is thread safe and can handle multiple concurrent threads
>>
>> Is part of the Ignite cluster
>> Ignite Servers can iniatate TCP connection needs to be reachable via a
>> TCP connection
>> If the Client is behind a firewall,NAT or Load Balancer it makes using
>> ,thick client challenging
>>
>>
>>
>> =========
>> *Question on GRPC*
>> =============
>>
>> The Thin Client TCP socket connection uses a Binary protocol
>> https://apacheignite.readme.io/docs/binary-client-protocol
>>
>> I may be mixing up network levels here but , is it possible to run this
>> over a GRPC connection
>>
>