You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stuart Broad <st...@moogsoft.com> on 2013/04/19 13:57:47 UTC
Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Hi,
I am using Cassandra.Client prepare_cql3_query/execute_prepared_cql3_query
to create and run some prepared statements. It is working well but I am
unclear as to how long the server side 'caches' the prepared statements.
Should a prepared statement be prepared for every new Cassandra.Client?
Based on my limited testing it seems like I can create some prepared
statements in one Cassandra.Client and use in another but I am not sure how
reliable/lasting this is i.e. If I called the prepared statement again the
next day would it still exist? What about if cassandra was re-started?
*Background:*
I am creating prepared statements for batch updates of pre-defined lengths
(e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if these could
just be set up once. We felt that using the prepared statements was easier
than escaping values within a CQL statement and probably more performant.
Thanks in advance for your help.
Regards,
Stuart
p.s. I am relatively new to cassandra.
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Hi Sorin,
Thanks very much for your response. From the sounds of it I think I can
share the prepared statements as long as I handle the case when they
disappear (out of LSU cache or due to server restart). To identify this I
think I need to catch InvalidRequestException and look at the message (I
will do some investigation on Monday and post an update if I figure out the
exact messages).
Regards,
Stuart
On Fri, Apr 19, 2013 at 11:23 PM, Sorin Manolache <so...@gmail.com> wrote:
> On 2013-04-19 13:57, Stuart Broad wrote:
>
>> Hi,
>>
>> I am using Cassandra.Client
>> prepare_cql3_query/execute_**prepared_cql3_query to create and run some
>> prepared statements. It is working well but I am unclear as to how long
>> the server side 'caches' the prepared statements. Should a prepared
>> statement be prepared for every new Cassandra.Client? Based on my
>> limited testing it seems like I can create some prepared statements in
>> one Cassandra.Client and use in another but I am not sure how
>> reliable/lasting this is i.e. If I called the prepared statement again
>> the next day would it still exist? What about if cassandra was
>> re-started?
>>
>
> I don't know the answer and I have the same question. But have a look at
> this discussion, dating from September 2012.
>
> https://issues.apache.org/**jira/browse/CASSANDRA-4449<https://issues.apache.org/jira/browse/CASSANDRA-4449>
>
> Apparently prepared statements are shared among threads (they were
> per-connection previously), they don't survive server restarts, apparently
> there's an LRU mechanism, and apparently you get a special exception if the
> prepared statement "disappeared" so you can prepare it again.
>
> Regards,
> Sorin
>
>
> _Background:_
>>
>> I am creating prepared statements for batch updates of pre-defined
>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>> these could just be set up once. We felt that using the prepared
>> statements was easier than escaping values within a CQL statement and
>> probably more performant.
>>
>> Thanks in advance for your help.
>>
>> Regards,
>>
>> Stuart
>>
>> p.s. I am relatively new to cassandra.
>>
>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Sorin Manolache <so...@gmail.com>.
On 2013-04-19 13:57, Stuart Broad wrote:
> Hi,
>
> I am using Cassandra.Client
> prepare_cql3_query/execute_prepared_cql3_query to create and run some
> prepared statements. It is working well but I am unclear as to how long
> the server side 'caches' the prepared statements. Should a prepared
> statement be prepared for every new Cassandra.Client? Based on my
> limited testing it seems like I can create some prepared statements in
> one Cassandra.Client and use in another but I am not sure how
> reliable/lasting this is i.e. If I called the prepared statement again
> the next day would it still exist? What about if cassandra was re-started?
I don't know the answer and I have the same question. But have a look at
this discussion, dating from September 2012.
https://issues.apache.org/jira/browse/CASSANDRA-4449
Apparently prepared statements are shared among threads (they were
per-connection previously), they don't survive server restarts,
apparently there's an LRU mechanism, and apparently you get a special
exception if the prepared statement "disappeared" so you can prepare it
again.
Regards,
Sorin
> _Background:_
> I am creating prepared statements for batch updates of pre-defined
> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
> these could just be set up once. We felt that using the prepared
> statements was easier than escaping values within a CQL statement and
> probably more performant.
>
> Thanks in advance for your help.
>
> Regards,
>
> Stuart
>
> p.s. I am relatively new to cassandra.
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by "Hiller, Dean" <De...@nrel.gov>.
Nice,
Thanks,
Dean
From: Sylvain Lebresne <sy...@datastax.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, April 23, 2013 11:31 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
On Tue, Apr 23, 2013 at 6:02 PM, Hiller, Dean <De...@nrel.gov>> wrote:
Out of curiosity, why did cassandra choose to re-invent the wheel instead of using something like google protobuf which spans multiple languages?
I see it as a step better than thrift since it is really only defining message format and has all sorts of goodies with it. I think you only need to frame it and that may exist already as well actually but I can't remember.
The serialization/deserialization involved in the binary protocol is not a big deal tbh, so I guess we chose to avoid the dependency. I personally don't think using protobufs would have simplified things much in practice, I don't think there is that much wheel reinventing, and so I'm reasonably happy to have something tailored to your needs. I'll admit there is some subjectivity in that opinion however, your mileage may vary.
Lastly, does the java-driver have an asynch nature to it at all?
The java driver is completely asynchronous, from the protocol to it's implementation, so yes.
It would be nice to be able to call driver.put(myData, myCallbackSuccessHandler);
In case you look at the driver API, its execute method returns a future, that happens to extends guava's ListenableFuture, and so you can add a callback/listener through that.
--
Sylvain
From: Sylvain Lebresne <sy...@datastax.com>>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>>" <us...@cassandra.apache.org>>>
Date: Tuesday, April 23, 2013 9:55 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>>" <us...@cassandra.apache.org>>>
Subject: Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
When we speak of "binary protocol", we talk about the protocol introduced in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom, binary, protocol, that has not link to thrift whatsoever.
That protocol is defined by the document here: https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD
Of course, this is just a protocol, and unless you have the time and willingness to write a proper library using that protocol, you should just use an existing driver implementing it. If you are using Java (some of your example above seems to be in Java), then you could for instance pick https://github.com/datastax/java-driver. If you're not using java, then well, since said protocol is fairly recent, there isn't an existing driver for every languages, but a bunch of drivers are in the work.
That being said, I'm not saying you *should* use a driver that uses the binary protocol, just that at least for exceptions handling, said binary protocol has a slightly cleaner handling of them than what's available through thrift. I'll not that even if you do want to use thrift, it's usually advised to use a high level client rather than raw thrift. Unless you have no choice or like suffering that is.
--
Sylvain
On Tue, Apr 23, 2013 at 5:38 PM, Stuart Broad <st...@moogsoft.com>>> wrote:
Hi Edward,
My understanding was that thrift supports a number of protocols (binary being one of them). I don't understand what switching to "binary protocol" but not using thrift means. Can you point me to any code examples?
Regards,
Stuart
On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <ed...@gmail.com>>> wrote:
Having to catch the exception and parse it is a bit ugly, however this is close to what someone might do with an SQLException to determine if the error was transient etc. If there is an error code it is possible that it could be added as an optional property of the InvalidRequestException in future versions.
Switching to the "binany protocol" is not a method in thrift, it means your not using thrift at all.
On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com>>> wrote:
Hi Edward,
Thanks for your reply - I was already using the prepare/execute cql methods that you suggested. My problem is that these methods 'mask' the PreparedQueryNotFoundException as an InvalidRequestException. At present I catch the InvalidRequestException (when cassandra has been re-started) and check the message text to figure out if I need to rebuild the prepared queries (rather than building each time I call).
Sylvain had suggested that I use the binary protocol as the exceptions are more explicit so I am trying to determine how this can be done (I don't see any obvious methods other than the cql ones for calling prepared statements).
Regards,
Stuart
On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>>> wrote:
Thrift has a prepare_cql call which returns an ID. Then it has an exececute_cql call which takes the id and a map or variable bindings.
On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>>> wrote:
Hi all,
I just realised that the binary protocol is the low-level thrift api that I was originally using (Cassandra.Client>> get / insert ...). How can a prepared statement be called through the thrift api (i.e. not the cql methods)?
Cheers,
Stuart
On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>>> wrote:
Hi Sylvain,
Thanks for your response. I am handling the 'PreparedQueryNotFoundException' more for the case of a cassandra re-start (rather then expecting to build 100000 statements).
I am not familiar with the binary protocol - which class/methods should I look at?
Regards,
Stuart
On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sy...@datastax.com>>> wrote:
In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are simply returned as InvalidRequestException. The reason for that was a mix of not wanting to change the thrift API too much and because we didn't knew how to return a lot of different exception with thrift without making it horrible to work with. So you'll probably have to parse strings here indeed.
This will be cleaner/less fragile if you use the binary protocol as exceptions are more fined grained there.
Though taking a step back (and without saying that you shouldn't handle the case where a query is not prepared on the node you contact), if you're really considering preparing more than 100000 statements, I'd suggest that it might be worth benchmarking whether using prepared statements in your case is really going to be worth the trouble. Just saying.
--
Sylvain
On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>>> wrote:
Hi Sorin,
The PreparedQueryNotFoundException is not thrown from Cassandra.Client>>execute_prepared_cql3_query method. I created some prepared statements and then re-started cassandra and received the following exception:
InvalidRequestException(why: Prepared query with ID 1124421588 not found (either the query was not prepared on this host (maybe the host has been restarted?) or you have prepared more than 100000 queries and queries 1124421588 has been evicted from the internal cache))
The best I have been able to come up with is the following:
try {
client.execute_prepared_cql3_query(psId, bindValues, ..);
} catch (InvalidRequestException invEx) {
String why = invEx.getWhy();
CLogger.logger().warning(why);
if(why.startsWith("Prepared query with ID")) {
rebuildPreparedStatement(preparedStatement);
client.execute_prepared_cql3_query(psId, bindValues, ..);
} else {
throw invEx;
}
}
Obviously this is pretty fragile and would break if the cassandra message was changed...but it least it works for now!
Cheers,
Stuart
On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>>> wrote:
On 2013-04-19 13:57, Stuart Broad wrote:
Hi,
I am using Cassandra.Client
prepare_cql3_query/execute_prepared_cql3_query to create and run some
prepared statements. It is working well but I am unclear as to how long
the server side 'caches' the prepared statements. Should a prepared
statement be prepared for every new Cassandra.Client? Based on my
limited testing it seems like I can create some prepared statements in
one Cassandra.Client and use in another but I am not sure how
reliable/lasting this is i.e. If I called the prepared statement again
the next day would it still exist? What about if cassandra was re-started?
_Background:_
I am creating prepared statements for batch updates of pre-defined
lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
these could just be set up once. We felt that using the prepared
statements was easier than escaping values within a CQL statement and
probably more performant.
Thanks in advance for your help.
I've looked in Cassandra's code (v1.2.3). The cache of prepared statements has a size of 100,000. So if you prepare more than 100 thousand statements, the least recently used ones will vanish. You'll get the exception PreparedQueryNotFoundException, code 0x2500.
Regards,
Sorin
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Tue, Apr 23, 2013 at 6:02 PM, Hiller, Dean <De...@nrel.gov> wrote:
> Out of curiosity, why did cassandra choose to re-invent the wheel instead
> of using something like google protobuf which spans multiple languages?
I see it as a step better than thrift since it is really only defining
> message format and has all sorts of goodies with it. I think you only need
> to frame it and that may exist already as well actually but I can't
> remember.
>
The serialization/deserialization involved in the binary protocol is not a
big deal tbh, so I guess we chose to avoid the dependency. I personally
don't think using protobufs would have simplified things much in practice,
I don't think there is that much wheel reinventing, and so I'm reasonably
happy to have something tailored to your needs. I'll admit there is some
subjectivity in that opinion however, your mileage may vary.
Lastly, does the java-driver have an asynch nature to it at all?
The java driver is completely asynchronous, from the protocol to it's
implementation, so yes.
It would be nice to be able to call driver.put(myData,
> myCallbackSuccessHandler);
In case you look at the driver API, its execute method returns a future,
that happens to extends guava's ListenableFuture, and so you can add a
callback/listener through that.
--
Sylvain
> From: Sylvain Lebresne <sy...@datastax.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Date: Tuesday, April 23, 2013 9:55 AM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Subject: Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
>
> When we speak of "binary protocol", we talk about the protocol introduced
> in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom,
> binary, protocol, that has not link to thrift whatsoever.
>
> That protocol is defined by the document here:
> https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD
>
> Of course, this is just a protocol, and unless you have the time and
> willingness to write a proper library using that protocol, you should just
> use an existing driver implementing it. If you are using Java (some of your
> example above seems to be in Java), then you could for instance pick
> https://github.com/datastax/java-driver. If you're not using java, then
> well, since said protocol is fairly recent, there isn't an existing driver
> for every languages, but a bunch of drivers are in the work.
>
> That being said, I'm not saying you *should* use a driver that uses the
> binary protocol, just that at least for exceptions handling, said binary
> protocol has a slightly cleaner handling of them than what's available
> through thrift. I'll not that even if you do want to use thrift, it's
> usually advised to use a high level client rather than raw thrift. Unless
> you have no choice or like suffering that is.
>
> --
> Sylvain
>
>
> On Tue, Apr 23, 2013 at 5:38 PM, Stuart Broad <stuart@moogsoft.com<mailto:
> stuart@moogsoft.com>> wrote:
> Hi Edward,
>
> My understanding was that thrift supports a number of protocols (binary
> being one of them). I don't understand what switching to "binary protocol"
> but not using thrift means. Can you point me to any code examples?
>
> Regards,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <edlinuxguru@gmail.com
> <ma...@gmail.com>> wrote:
> Having to catch the exception and parse it is a bit ugly, however this is
> close to what someone might do with an SQLException to determine if the
> error was transient etc. If there is an error code it is possible that it
> could be added as an optional property of the InvalidRequestException in
> future versions.
>
> Switching to the "binany protocol" is not a method in thrift, it means
> your not using thrift at all.
>
>
>
>
> On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <stuart@moogsoft.com
> <ma...@moogsoft.com>> wrote:
> Hi Edward,
>
> Thanks for your reply - I was already using the prepare/execute cql
> methods that you suggested. My problem is that these methods 'mask' the
> PreparedQueryNotFoundException as an InvalidRequestException. At present I
> catch the InvalidRequestException (when cassandra has been re-started) and
> check the message text to figure out if I need to rebuild the prepared
> queries (rather than building each time I call).
>
> Sylvain had suggested that I use the binary protocol as the exceptions are
> more explicit so I am trying to determine how this can be done (I don't see
> any obvious methods other than the cql ones for calling prepared
> statements).
>
> Regards,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <edlinuxguru@gmail.com
> <ma...@gmail.com>> wrote:
> Thrift has a prepare_cql call which returns an ID. Then it has an
> exececute_cql call which takes the id and a map or variable bindings.
>
>
> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <stuart@moogsoft.com
> <ma...@moogsoft.com>> wrote:
> Hi all,
>
> I just realised that the binary protocol is the low-level thrift api that
> I was originally using (Cassandra.Client>> get / insert ...). How can a
> prepared statement be called through the thrift api (i.e. not the cql
> methods)?
>
> Cheers,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <stuart@moogsoft.com
> <ma...@moogsoft.com>> wrote:
> Hi Sylvain,
>
> Thanks for your response. I am handling the
> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
> (rather then expecting to build 100000 statements).
>
> I am not familiar with the binary protocol - which class/methods should I
> look at?
>
> Regards,
>
> Stuart
>
>
>
> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sylvain@datastax.com
> <ma...@datastax.com>> wrote:
> In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are
> simply returned as InvalidRequestException. The reason for that was a mix
> of not wanting to change the thrift API too much and because we didn't knew
> how to return a lot of different exception with thrift without making it
> horrible to work with. So you'll probably have to parse strings here indeed.
>
> This will be cleaner/less fragile if you use the binary protocol as
> exceptions are more fined grained there.
>
> Though taking a step back (and without saying that you shouldn't handle
> the case where a query is not prepared on the node you contact), if you're
> really considering preparing more than 100000 statements, I'd suggest that
> it might be worth benchmarking whether using prepared statements in your
> case is really going to be worth the trouble. Just saying.
>
> --
> Sylvain
>
>
>
> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <stuart@moogsoft.com
> <ma...@moogsoft.com>> wrote:
> Hi Sorin,
>
> The PreparedQueryNotFoundException is not thrown from
> Cassandra.Client>>execute_prepared_cql3_query method. I created some
> prepared statements and then re-started cassandra and received the
> following exception:
>
> InvalidRequestException(why: Prepared query with ID 1124421588 not found
> (either the query was not prepared on this host (maybe the host has been
> restarted?) or you have prepared more than 100000 queries and queries
> 1124421588 has been evicted from the internal cache))
>
> The best I have been able to come up with is the following:
>
> try {
> client.execute_prepared_cql3_query(psId, bindValues, ..);
> } catch (InvalidRequestException invEx) {
> String why = invEx.getWhy();
> CLogger.logger().warning(why);
> if(why.startsWith("Prepared query with ID")) {
> rebuildPreparedStatement(preparedStatement);
> client.execute_prepared_cql3_query(psId, bindValues,
> ..);
> } else {
> throw invEx;
> }
> }
>
> Obviously this is pretty fragile and would break if the cassandra message
> was changed...but it least it works for now!
>
> Cheers,
>
> Stuart
>
>
> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <sorinm@gmail.com
> <ma...@gmail.com>> wrote:
> On 2013-04-19 13:57, Stuart Broad wrote:
> Hi,
>
> I am using Cassandra.Client
> prepare_cql3_query/execute_prepared_cql3_query to create and run some
> prepared statements. It is working well but I am unclear as to how long
> the server side 'caches' the prepared statements. Should a prepared
> statement be prepared for every new Cassandra.Client? Based on my
> limited testing it seems like I can create some prepared statements in
> one Cassandra.Client and use in another but I am not sure how
> reliable/lasting this is i.e. If I called the prepared statement again
> the next day would it still exist? What about if cassandra was re-started?
>
> _Background:_
>
> I am creating prepared statements for batch updates of pre-defined
> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
> these could just be set up once. We felt that using the prepared
> statements was easier than escaping values within a CQL statement and
> probably more performant.
>
> Thanks in advance for your help.
>
>
> I've looked in Cassandra's code (v1.2.3). The cache of prepared statements
> has a size of 100,000. So if you prepare more than 100 thousand statements,
> the least recently used ones will vanish. You'll get the exception
> PreparedQueryNotFoundException, code 0x2500.
>
> Regards,
> Sorin
>
>
>
>
>
>
>
>
>
>
>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by "Hiller, Dean" <De...@nrel.gov>.
Out of curiosity, why did cassandra choose to re-invent the wheel instead of using something like google protobuf which spans multiple languages? I see it as a step better than thrift since it is really only defining message format and has all sorts of goodies with it. I think you only need to frame it and that may exist already as well actually but I can't remember.
Also, it would have made other drivers easier to create as serialization/deserialization would already be there since protobuf has all the generation done there.
Lastly, does the java-driver have an asynch nature to it at all? It would be nice to be able to call driver.put(myData, myCallbackSuccessHandler); wich would return immediately so I can process my next batch of stuff…..extremely good in batching of course where I don't need an immediate success response.
Later,
Dean
From: Sylvain Lebresne <sy...@datastax.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Tuesday, April 23, 2013 9:55 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
When we speak of "binary protocol", we talk about the protocol introduced in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom, binary, protocol, that has not link to thrift whatsoever.
That protocol is defined by the document here: https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD
Of course, this is just a protocol, and unless you have the time and willingness to write a proper library using that protocol, you should just use an existing driver implementing it. If you are using Java (some of your example above seems to be in Java), then you could for instance pick https://github.com/datastax/java-driver. If you're not using java, then well, since said protocol is fairly recent, there isn't an existing driver for every languages, but a bunch of drivers are in the work.
That being said, I'm not saying you *should* use a driver that uses the binary protocol, just that at least for exceptions handling, said binary protocol has a slightly cleaner handling of them than what's available through thrift. I'll not that even if you do want to use thrift, it's usually advised to use a high level client rather than raw thrift. Unless you have no choice or like suffering that is.
--
Sylvain
On Tue, Apr 23, 2013 at 5:38 PM, Stuart Broad <st...@moogsoft.com>> wrote:
Hi Edward,
My understanding was that thrift supports a number of protocols (binary being one of them). I don't understand what switching to "binary protocol" but not using thrift means. Can you point me to any code examples?
Regards,
Stuart
On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <ed...@gmail.com>> wrote:
Having to catch the exception and parse it is a bit ugly, however this is close to what someone might do with an SQLException to determine if the error was transient etc. If there is an error code it is possible that it could be added as an optional property of the InvalidRequestException in future versions.
Switching to the "binany protocol" is not a method in thrift, it means your not using thrift at all.
On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com>> wrote:
Hi Edward,
Thanks for your reply - I was already using the prepare/execute cql methods that you suggested. My problem is that these methods 'mask' the PreparedQueryNotFoundException as an InvalidRequestException. At present I catch the InvalidRequestException (when cassandra has been re-started) and check the message text to figure out if I need to rebuild the prepared queries (rather than building each time I call).
Sylvain had suggested that I use the binary protocol as the exceptions are more explicit so I am trying to determine how this can be done (I don't see any obvious methods other than the cql ones for calling prepared statements).
Regards,
Stuart
On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>> wrote:
Thrift has a prepare_cql call which returns an ID. Then it has an exececute_cql call which takes the id and a map or variable bindings.
On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>> wrote:
Hi all,
I just realised that the binary protocol is the low-level thrift api that I was originally using (Cassandra.Client>> get / insert ...). How can a prepared statement be called through the thrift api (i.e. not the cql methods)?
Cheers,
Stuart
On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>> wrote:
Hi Sylvain,
Thanks for your response. I am handling the 'PreparedQueryNotFoundException' more for the case of a cassandra re-start (rather then expecting to build 100000 statements).
I am not familiar with the binary protocol - which class/methods should I look at?
Regards,
Stuart
On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sy...@datastax.com>> wrote:
In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are simply returned as InvalidRequestException. The reason for that was a mix of not wanting to change the thrift API too much and because we didn't knew how to return a lot of different exception with thrift without making it horrible to work with. So you'll probably have to parse strings here indeed.
This will be cleaner/less fragile if you use the binary protocol as exceptions are more fined grained there.
Though taking a step back (and without saying that you shouldn't handle the case where a query is not prepared on the node you contact), if you're really considering preparing more than 100000 statements, I'd suggest that it might be worth benchmarking whether using prepared statements in your case is really going to be worth the trouble. Just saying.
--
Sylvain
On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>> wrote:
Hi Sorin,
The PreparedQueryNotFoundException is not thrown from Cassandra.Client>>execute_prepared_cql3_query method. I created some prepared statements and then re-started cassandra and received the following exception:
InvalidRequestException(why: Prepared query with ID 1124421588 not found (either the query was not prepared on this host (maybe the host has been restarted?) or you have prepared more than 100000 queries and queries 1124421588 has been evicted from the internal cache))
The best I have been able to come up with is the following:
try {
client.execute_prepared_cql3_query(psId, bindValues, ..);
} catch (InvalidRequestException invEx) {
String why = invEx.getWhy();
CLogger.logger().warning(why);
if(why.startsWith("Prepared query with ID")) {
rebuildPreparedStatement(preparedStatement);
client.execute_prepared_cql3_query(psId, bindValues, ..);
} else {
throw invEx;
}
}
Obviously this is pretty fragile and would break if the cassandra message was changed...but it least it works for now!
Cheers,
Stuart
On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>> wrote:
On 2013-04-19 13:57, Stuart Broad wrote:
Hi,
I am using Cassandra.Client
prepare_cql3_query/execute_prepared_cql3_query to create and run some
prepared statements. It is working well but I am unclear as to how long
the server side 'caches' the prepared statements. Should a prepared
statement be prepared for every new Cassandra.Client? Based on my
limited testing it seems like I can create some prepared statements in
one Cassandra.Client and use in another but I am not sure how
reliable/lasting this is i.e. If I called the prepared statement again
the next day would it still exist? What about if cassandra was re-started?
_Background:_
I am creating prepared statements for batch updates of pre-defined
lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
these could just be set up once. We felt that using the prepared
statements was easier than escaping values within a CQL statement and
probably more performant.
Thanks in advance for your help.
I've looked in Cassandra's code (v1.2.3). The cache of prepared statements has a size of 100,000. So if you prepare more than 100 thousand statements, the least recently used ones will vanish. You'll get the exception PreparedQueryNotFoundException, code 0x2500.
Regards,
Sorin
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Aha - got it. Thanks for everyones help.
I think I will stick with the prepare/execute CQL (with the
InvalidRequestException check) for now. I will take a look at the driver
you mentioned though.
Cheers,
Stuart
On Tue, Apr 23, 2013 at 4:55 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> When we speak of "binary protocol", we talk about the protocol introduced
> in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom,
> binary, protocol, that has not link to thrift whatsoever.
>
> That protocol is defined by the document here:
> https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD
>
> Of course, this is just a protocol, and unless you have the time and
> willingness to write a proper library using that protocol, you should just
> use an existing driver implementing it. If you are using Java (some of your
> example above seems to be in Java), then you could for instance pick
> https://github.com/datastax/java-driver. If you're not using java, then
> well, since said protocol is fairly recent, there isn't an existing driver
> for every languages, but a bunch of drivers are in the work.
>
> That being said, I'm not saying you *should* use a driver that uses the
> binary protocol, just that at least for exceptions handling, said binary
> protocol has a slightly cleaner handling of them than what's available
> through thrift. I'll not that even if you do want to use thrift, it's
> usually advised to use a high level client rather than raw thrift. Unless
> you have no choice or like suffering that is.
>
> --
> Sylvain
>
>
> On Tue, Apr 23, 2013 at 5:38 PM, Stuart Broad <st...@moogsoft.com> wrote:
>
>> Hi Edward,
>>
>> My understanding was that thrift supports a number of protocols (binary
>> being one of them). I don't understand what switching to "binary protocol"
>> but not using thrift means. Can you point me to any code examples?
>>
>> Regards,
>>
>> Stuart
>>
>>
>> On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> Having to catch the exception and parse it is a bit ugly, however this
>>> is close to what someone might do with an SQLException to determine if the
>>> error was transient etc. If there is an error code it is possible that it
>>> could be added as an optional property of the InvalidRequestException in
>>> future versions.
>>>
>>> Switching to the "binany protocol" is not a method in thrift, it means
>>> your not using thrift at all.
>>>
>>>
>>>
>>>
>>> On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>
>>>> Hi Edward,
>>>>
>>>> Thanks for your reply - I was already using the prepare/execute cql
>>>> methods that you suggested. My problem is that these methods 'mask' the
>>>> PreparedQueryNotFoundException as an InvalidRequestException. At present I
>>>> catch the InvalidRequestException (when cassandra has been re-started) and
>>>> check the message text to figure out if I need to rebuild the prepared
>>>> queries (rather than building each time I call).
>>>>
>>>> Sylvain had suggested that I use the binary protocol as the exceptions
>>>> are more explicit so I am trying to determine how this can be done (I don't
>>>> see any obvious methods other than the cql ones for calling prepared
>>>> statements).
>>>>
>>>> Regards,
>>>>
>>>> Stuart
>>>>
>>>>
>>>> On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <edlinuxguru@gmail.com
>>>> > wrote:
>>>>
>>>>> Thrift has a prepare_cql call which returns an ID. Then it has an
>>>>> exececute_cql call which takes the id and a map or variable bindings.
>>>>>
>>>>>
>>>>> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I just realised that the binary protocol is the low-level thrift api
>>>>>> that I was originally using (Cassandra.Client>> get / insert ...). How can
>>>>>> a prepared statement be called through the thrift api (i.e. not the cql
>>>>>> methods)?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Stuart
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>>
>>>>>>> Hi Sylvain,
>>>>>>>
>>>>>>> Thanks for your response. I am handling the
>>>>>>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>>>>>>> (rather then expecting to build 100000 statements).
>>>>>>>
>>>>>>> I am not familiar with the binary protocol - which class/methods
>>>>>>> should I look at?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Stuart
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <
>>>>>>> sylvain@datastax.com> wrote:
>>>>>>>
>>>>>>>> In thrift, a lot of exceptions (like
>>>>>>>> PreparedQueryNotFoundException) are simply returned as
>>>>>>>> InvalidRequestException. The reason for that was a mix of not wanting to
>>>>>>>> change the thrift API too much and because we didn't knew how to return a
>>>>>>>> lot of different exception with thrift without making it horrible to work
>>>>>>>> with. So you'll probably have to parse strings here indeed.
>>>>>>>>
>>>>>>>> This will be cleaner/less fragile if you use the binary protocol as
>>>>>>>> exceptions are more fined grained there.
>>>>>>>>
>>>>>>>> Though taking a step back (and without saying that you shouldn't
>>>>>>>> handle the case where a query is not prepared on the node you contact), if
>>>>>>>> you're really considering preparing more than 100000 statements, I'd
>>>>>>>> suggest that it might be worth benchmarking whether using prepared
>>>>>>>> statements in your case is really going to be worth the trouble. Just
>>>>>>>> saying.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sylvain
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <stuart@moogsoft.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Sorin,
>>>>>>>>>
>>>>>>>>> The PreparedQueryNotFoundException is not thrown from
>>>>>>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>>>>>>> prepared statements and then re-started cassandra and received the
>>>>>>>>> following exception:
>>>>>>>>>
>>>>>>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>>>>>>> found (either the query was not prepared on this host (maybe the host has
>>>>>>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>>>>>>> 1124421588 has been evicted from the internal cache))
>>>>>>>>>
>>>>>>>>> The best I have been able to come up with is the following:
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>>>> bindValues, ..);
>>>>>>>>> } catch (InvalidRequestException invEx) {
>>>>>>>>> String why = invEx.getWhy();
>>>>>>>>> CLogger.logger().warning(why);
>>>>>>>>> if(why.startsWith("Prepared query with ID")) {
>>>>>>>>> rebuildPreparedStatement(preparedStatement);
>>>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>>>> bindValues, ..);
>>>>>>>>> } else {
>>>>>>>>> throw invEx;
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Obviously this is pretty fragile and would break if the cassandra
>>>>>>>>> message was changed...but it least it works for now!
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Stuart
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <
>>>>>>>>> sorinm@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am using Cassandra.Client
>>>>>>>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and
>>>>>>>>>>> run some
>>>>>>>>>>> prepared statements. It is working well but I am unclear as to
>>>>>>>>>>> how long
>>>>>>>>>>> the server side 'caches' the prepared statements. Should a
>>>>>>>>>>> prepared
>>>>>>>>>>> statement be prepared for every new Cassandra.Client? Based on
>>>>>>>>>>> my
>>>>>>>>>>> limited testing it seems like I can create some prepared
>>>>>>>>>>> statements in
>>>>>>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>>>>>>> reliable/lasting this is i.e. If I called the prepared
>>>>>>>>>>> statement again
>>>>>>>>>>> the next day would it still exist? What about if cassandra was
>>>>>>>>>>> re-started?
>>>>>>>>>>>
>>>>>>>>>>> _Background:_
>>>>>>>>>>>
>>>>>>>>>>> I am creating prepared statements for batch updates of
>>>>>>>>>>> pre-defined
>>>>>>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to
>>>>>>>>>>> know if
>>>>>>>>>>> these could just be set up once. We felt that using the prepared
>>>>>>>>>>> statements was easier than escaping values within a CQL
>>>>>>>>>>> statement and
>>>>>>>>>>> probably more performant.
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance for your help.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>>>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>>>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>>>>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Sorin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Sylvain Lebresne <sy...@datastax.com>.
When we speak of "binary protocol", we talk about the protocol introduced
in Cassandra 1.2 that is an alternative to thrift for CQL3. It's a custom,
binary, protocol, that has not link to thrift whatsoever.
That protocol is defined by the document here:
https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=doc/native_protocol_v1.spec;hb=HEAD
Of course, this is just a protocol, and unless you have the time and
willingness to write a proper library using that protocol, you should just
use an existing driver implementing it. If you are using Java (some of your
example above seems to be in Java), then you could for instance pick
https://github.com/datastax/java-driver. If you're not using java, then
well, since said protocol is fairly recent, there isn't an existing driver
for every languages, but a bunch of drivers are in the work.
That being said, I'm not saying you *should* use a driver that uses the
binary protocol, just that at least for exceptions handling, said binary
protocol has a slightly cleaner handling of them than what's available
through thrift. I'll not that even if you do want to use thrift, it's
usually advised to use a high level client rather than raw thrift. Unless
you have no choice or like suffering that is.
--
Sylvain
On Tue, Apr 23, 2013 at 5:38 PM, Stuart Broad <st...@moogsoft.com> wrote:
> Hi Edward,
>
> My understanding was that thrift supports a number of protocols (binary
> being one of them). I don't understand what switching to "binary protocol"
> but not using thrift means. Can you point me to any code examples?
>
> Regards,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Having to catch the exception and parse it is a bit ugly, however this is
>> close to what someone might do with an SQLException to determine if the
>> error was transient etc. If there is an error code it is possible that it
>> could be added as an optional property of the InvalidRequestException in
>> future versions.
>>
>> Switching to the "binany protocol" is not a method in thrift, it means
>> your not using thrift at all.
>>
>>
>>
>>
>> On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>
>>> Hi Edward,
>>>
>>> Thanks for your reply - I was already using the prepare/execute cql
>>> methods that you suggested. My problem is that these methods 'mask' the
>>> PreparedQueryNotFoundException as an InvalidRequestException. At present I
>>> catch the InvalidRequestException (when cassandra has been re-started) and
>>> check the message text to figure out if I need to rebuild the prepared
>>> queries (rather than building each time I call).
>>>
>>> Sylvain had suggested that I use the binary protocol as the exceptions
>>> are more explicit so I am trying to determine how this can be done (I don't
>>> see any obvious methods other than the cql ones for calling prepared
>>> statements).
>>>
>>> Regards,
>>>
>>> Stuart
>>>
>>>
>>> On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>>
>>>> Thrift has a prepare_cql call which returns an ID. Then it has an
>>>> exececute_cql call which takes the id and a map or variable bindings.
>>>>
>>>>
>>>> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I just realised that the binary protocol is the low-level thrift api
>>>>> that I was originally using (Cassandra.Client>> get / insert ...). How can
>>>>> a prepared statement be called through the thrift api (i.e. not the cql
>>>>> methods)?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>
>>>>>> Hi Sylvain,
>>>>>>
>>>>>> Thanks for your response. I am handling the
>>>>>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>>>>>> (rather then expecting to build 100000 statements).
>>>>>>
>>>>>> I am not familiar with the binary protocol - which class/methods
>>>>>> should I look at?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Stuart
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <
>>>>>> sylvain@datastax.com> wrote:
>>>>>>
>>>>>>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException)
>>>>>>> are simply returned as InvalidRequestException. The reason for that was a
>>>>>>> mix of not wanting to change the thrift API too much and because we didn't
>>>>>>> knew how to return a lot of different exception with thrift without making
>>>>>>> it horrible to work with. So you'll probably have to parse strings here
>>>>>>> indeed.
>>>>>>>
>>>>>>> This will be cleaner/less fragile if you use the binary protocol as
>>>>>>> exceptions are more fined grained there.
>>>>>>>
>>>>>>> Though taking a step back (and without saying that you shouldn't
>>>>>>> handle the case where a query is not prepared on the node you contact), if
>>>>>>> you're really considering preparing more than 100000 statements, I'd
>>>>>>> suggest that it might be worth benchmarking whether using prepared
>>>>>>> statements in your case is really going to be worth the trouble. Just
>>>>>>> saying.
>>>>>>>
>>>>>>> --
>>>>>>> Sylvain
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>>>
>>>>>>>> Hi Sorin,
>>>>>>>>
>>>>>>>> The PreparedQueryNotFoundException is not thrown from
>>>>>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>>>>>> prepared statements and then re-started cassandra and received the
>>>>>>>> following exception:
>>>>>>>>
>>>>>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>>>>>> found (either the query was not prepared on this host (maybe the host has
>>>>>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>>>>>> 1124421588 has been evicted from the internal cache))
>>>>>>>>
>>>>>>>> The best I have been able to come up with is the following:
>>>>>>>>
>>>>>>>> try {
>>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>>> bindValues, ..);
>>>>>>>> } catch (InvalidRequestException invEx) {
>>>>>>>> String why = invEx.getWhy();
>>>>>>>> CLogger.logger().warning(why);
>>>>>>>> if(why.startsWith("Prepared query with ID")) {
>>>>>>>> rebuildPreparedStatement(preparedStatement);
>>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>>> bindValues, ..);
>>>>>>>> } else {
>>>>>>>> throw invEx;
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> Obviously this is pretty fragile and would break if the cassandra
>>>>>>>> message was changed...but it least it works for now!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Stuart
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <sorinm@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am using Cassandra.Client
>>>>>>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and
>>>>>>>>>> run some
>>>>>>>>>> prepared statements. It is working well but I am unclear as to
>>>>>>>>>> how long
>>>>>>>>>> the server side 'caches' the prepared statements. Should a
>>>>>>>>>> prepared
>>>>>>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>>>>>>> limited testing it seems like I can create some prepared
>>>>>>>>>> statements in
>>>>>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>>>>>> reliable/lasting this is i.e. If I called the prepared statement
>>>>>>>>>> again
>>>>>>>>>> the next day would it still exist? What about if cassandra was
>>>>>>>>>> re-started?
>>>>>>>>>>
>>>>>>>>>> _Background:_
>>>>>>>>>>
>>>>>>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to
>>>>>>>>>> know if
>>>>>>>>>> these could just be set up once. We felt that using the prepared
>>>>>>>>>> statements was easier than escaping values within a CQL statement
>>>>>>>>>> and
>>>>>>>>>> probably more performant.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance for your help.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>>>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Sorin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Edward Capriolo <ed...@gmail.com>.
Cassandra has a non thrift protocol called the "native protocol" aka "cql
binary protocol"
http://www.datastax.com/docs/1.2/cql_cli/cql_binary_protocol
It is its own port, with it's own protocol, and it does not have thrift
methods.
In my opinion, switching from the thrift to the native protocol only to
catch the error code is overkill. I bet it would not be impossible to add
the error code to the thrift method somehow, since we have added in the
exception a field about how many replicas an operation succeeded on in the
past. However, it seems now that even though things could be added to
thrift, no one is interested in doing it, or getting the feature added to
thrift lags behind getting it added to cql.
On Tue, Apr 23, 2013 at 11:38 AM, Stuart Broad <st...@moogsoft.com> wrote:
> Hi Edward,
>
> My understanding was that thrift supports a number of protocols (binary
> being one of them). I don't understand what switching to "binary protocol"
> but not using thrift means. Can you point me to any code examples?
>
> Regards,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Having to catch the exception and parse it is a bit ugly, however this is
>> close to what someone might do with an SQLException to determine if the
>> error was transient etc. If there is an error code it is possible that it
>> could be added as an optional property of the InvalidRequestException in
>> future versions.
>>
>> Switching to the "binany protocol" is not a method in thrift, it means
>> your not using thrift at all.
>>
>>
>>
>>
>> On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>
>>> Hi Edward,
>>>
>>> Thanks for your reply - I was already using the prepare/execute cql
>>> methods that you suggested. My problem is that these methods 'mask' the
>>> PreparedQueryNotFoundException as an InvalidRequestException. At present I
>>> catch the InvalidRequestException (when cassandra has been re-started) and
>>> check the message text to figure out if I need to rebuild the prepared
>>> queries (rather than building each time I call).
>>>
>>> Sylvain had suggested that I use the binary protocol as the exceptions
>>> are more explicit so I am trying to determine how this can be done (I don't
>>> see any obvious methods other than the cql ones for calling prepared
>>> statements).
>>>
>>> Regards,
>>>
>>> Stuart
>>>
>>>
>>> On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>>
>>>> Thrift has a prepare_cql call which returns an ID. Then it has an
>>>> exececute_cql call which takes the id and a map or variable bindings.
>>>>
>>>>
>>>> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I just realised that the binary protocol is the low-level thrift api
>>>>> that I was originally using (Cassandra.Client>> get / insert ...). How can
>>>>> a prepared statement be called through the thrift api (i.e. not the cql
>>>>> methods)?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>
>>>>>> Hi Sylvain,
>>>>>>
>>>>>> Thanks for your response. I am handling the
>>>>>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>>>>>> (rather then expecting to build 100000 statements).
>>>>>>
>>>>>> I am not familiar with the binary protocol - which class/methods
>>>>>> should I look at?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Stuart
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <
>>>>>> sylvain@datastax.com> wrote:
>>>>>>
>>>>>>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException)
>>>>>>> are simply returned as InvalidRequestException. The reason for that was a
>>>>>>> mix of not wanting to change the thrift API too much and because we didn't
>>>>>>> knew how to return a lot of different exception with thrift without making
>>>>>>> it horrible to work with. So you'll probably have to parse strings here
>>>>>>> indeed.
>>>>>>>
>>>>>>> This will be cleaner/less fragile if you use the binary protocol as
>>>>>>> exceptions are more fined grained there.
>>>>>>>
>>>>>>> Though taking a step back (and without saying that you shouldn't
>>>>>>> handle the case where a query is not prepared on the node you contact), if
>>>>>>> you're really considering preparing more than 100000 statements, I'd
>>>>>>> suggest that it might be worth benchmarking whether using prepared
>>>>>>> statements in your case is really going to be worth the trouble. Just
>>>>>>> saying.
>>>>>>>
>>>>>>> --
>>>>>>> Sylvain
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>>>
>>>>>>>> Hi Sorin,
>>>>>>>>
>>>>>>>> The PreparedQueryNotFoundException is not thrown from
>>>>>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>>>>>> prepared statements and then re-started cassandra and received the
>>>>>>>> following exception:
>>>>>>>>
>>>>>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>>>>>> found (either the query was not prepared on this host (maybe the host has
>>>>>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>>>>>> 1124421588 has been evicted from the internal cache))
>>>>>>>>
>>>>>>>> The best I have been able to come up with is the following:
>>>>>>>>
>>>>>>>> try {
>>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>>> bindValues, ..);
>>>>>>>> } catch (InvalidRequestException invEx) {
>>>>>>>> String why = invEx.getWhy();
>>>>>>>> CLogger.logger().warning(why);
>>>>>>>> if(why.startsWith("Prepared query with ID")) {
>>>>>>>> rebuildPreparedStatement(preparedStatement);
>>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>>> bindValues, ..);
>>>>>>>> } else {
>>>>>>>> throw invEx;
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> Obviously this is pretty fragile and would break if the cassandra
>>>>>>>> message was changed...but it least it works for now!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Stuart
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <sorinm@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am using Cassandra.Client
>>>>>>>>>> prepare_cql3_query/execute_prepared_cql3_query to create and run
>>>>>>>>>> some
>>>>>>>>>> prepared statements. It is working well but I am unclear as to
>>>>>>>>>> how long
>>>>>>>>>> the server side 'caches' the prepared statements. Should a
>>>>>>>>>> prepared
>>>>>>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>>>>>>> limited testing it seems like I can create some prepared
>>>>>>>>>> statements in
>>>>>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>>>>>> reliable/lasting this is i.e. If I called the prepared statement
>>>>>>>>>> again
>>>>>>>>>> the next day would it still exist? What about if cassandra was
>>>>>>>>>> re-started?
>>>>>>>>>>
>>>>>>>>>> _Background:_
>>>>>>>>>>
>>>>>>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to
>>>>>>>>>> know if
>>>>>>>>>> these could just be set up once. We felt that using the prepared
>>>>>>>>>> statements was easier than escaping values within a CQL statement
>>>>>>>>>> and
>>>>>>>>>> probably more performant.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance for your help.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>>>>>> exception PreparedQueryNotFoundException, code 0x2500.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Sorin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Hi Edward,
My understanding was that thrift supports a number of protocols (binary
being one of them). I don't understand what switching to "binary protocol"
but not using thrift means. Can you point me to any code examples?
Regards,
Stuart
On Tue, Apr 23, 2013 at 4:21 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Having to catch the exception and parse it is a bit ugly, however this is
> close to what someone might do with an SQLException to determine if the
> error was transient etc. If there is an error code it is possible that it
> could be added as an optional property of the InvalidRequestException in
> future versions.
>
> Switching to the "binany protocol" is not a method in thrift, it means
> your not using thrift at all.
>
>
>
>
> On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com>wrote:
>
>> Hi Edward,
>>
>> Thanks for your reply - I was already using the prepare/execute cql
>> methods that you suggested. My problem is that these methods 'mask' the
>> PreparedQueryNotFoundException as an InvalidRequestException. At present I
>> catch the InvalidRequestException (when cassandra has been re-started) and
>> check the message text to figure out if I need to rebuild the prepared
>> queries (rather than building each time I call).
>>
>> Sylvain had suggested that I use the binary protocol as the exceptions
>> are more explicit so I am trying to determine how this can be done (I don't
>> see any obvious methods other than the cql ones for calling prepared
>> statements).
>>
>> Regards,
>>
>> Stuart
>>
>>
>> On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> Thrift has a prepare_cql call which returns an ID. Then it has an
>>> exececute_cql call which takes the id and a map or variable bindings.
>>>
>>>
>>> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I just realised that the binary protocol is the low-level thrift api
>>>> that I was originally using (Cassandra.Client>> get / insert ...). How can
>>>> a prepared statement be called through the thrift api (i.e. not the cql
>>>> methods)?
>>>>
>>>> Cheers,
>>>>
>>>> Stuart
>>>>
>>>>
>>>> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>
>>>>> Hi Sylvain,
>>>>>
>>>>> Thanks for your response. I am handling the
>>>>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>>>>> (rather then expecting to build 100000 statements).
>>>>>
>>>>> I am not familiar with the binary protocol - which class/methods
>>>>> should I look at?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <
>>>>> sylvain@datastax.com> wrote:
>>>>>
>>>>>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException)
>>>>>> are simply returned as InvalidRequestException. The reason for that was a
>>>>>> mix of not wanting to change the thrift API too much and because we didn't
>>>>>> knew how to return a lot of different exception with thrift without making
>>>>>> it horrible to work with. So you'll probably have to parse strings here
>>>>>> indeed.
>>>>>>
>>>>>> This will be cleaner/less fragile if you use the binary protocol as
>>>>>> exceptions are more fined grained there.
>>>>>>
>>>>>> Though taking a step back (and without saying that you shouldn't
>>>>>> handle the case where a query is not prepared on the node you contact), if
>>>>>> you're really considering preparing more than 100000 statements, I'd
>>>>>> suggest that it might be worth benchmarking whether using prepared
>>>>>> statements in your case is really going to be worth the trouble. Just
>>>>>> saying.
>>>>>>
>>>>>> --
>>>>>> Sylvain
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>>
>>>>>>> Hi Sorin,
>>>>>>>
>>>>>>> The PreparedQueryNotFoundException is not thrown from
>>>>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>>>>> prepared statements and then re-started cassandra and received the
>>>>>>> following exception:
>>>>>>>
>>>>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>>>>> found (either the query was not prepared on this host (maybe the host has
>>>>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>>>>> 1124421588 has been evicted from the internal cache))
>>>>>>>
>>>>>>> The best I have been able to come up with is the following:
>>>>>>>
>>>>>>> try {
>>>>>>> client.execute_prepared_cql3_query(psId, bindValues,
>>>>>>> ..);
>>>>>>> } catch (InvalidRequestException invEx) {
>>>>>>> String why = invEx.getWhy();
>>>>>>> CLogger.logger().warning(why);
>>>>>>> if(why.startsWith("Prepared query with ID")) {
>>>>>>> rebuildPreparedStatement(preparedStatement);
>>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>>> bindValues, ..);
>>>>>>> } else {
>>>>>>> throw invEx;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Obviously this is pretty fragile and would break if the cassandra
>>>>>>> message was changed...but it least it works for now!
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Stuart
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>>>>>>>
>>>>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am using Cassandra.Client
>>>>>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and
>>>>>>>>> run some
>>>>>>>>> prepared statements. It is working well but I am unclear as to
>>>>>>>>> how long
>>>>>>>>> the server side 'caches' the prepared statements. Should a
>>>>>>>>> prepared
>>>>>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>>>>>> limited testing it seems like I can create some prepared
>>>>>>>>> statements in
>>>>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>>>>> reliable/lasting this is i.e. If I called the prepared statement
>>>>>>>>> again
>>>>>>>>> the next day would it still exist? What about if cassandra was
>>>>>>>>> re-started?
>>>>>>>>>
>>>>>>>>> _Background:_
>>>>>>>>>
>>>>>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know
>>>>>>>>> if
>>>>>>>>> these could just be set up once. We felt that using the prepared
>>>>>>>>> statements was easier than escaping values within a CQL statement
>>>>>>>>> and
>>>>>>>>> probably more performant.
>>>>>>>>>
>>>>>>>>> Thanks in advance for your help.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Sorin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Edward Capriolo <ed...@gmail.com>.
Having to catch the exception and parse it is a bit ugly, however this is
close to what someone might do with an SQLException to determine if the
error was transient etc. If there is an error code it is possible that it
could be added as an optional property of the InvalidRequestException in
future versions.
Switching to the "binany protocol" is not a method in thrift, it means your
not using thrift at all.
On Tue, Apr 23, 2013 at 11:13 AM, Stuart Broad <st...@moogsoft.com> wrote:
> Hi Edward,
>
> Thanks for your reply - I was already using the prepare/execute cql
> methods that you suggested. My problem is that these methods 'mask' the
> PreparedQueryNotFoundException as an InvalidRequestException. At present I
> catch the InvalidRequestException (when cassandra has been re-started) and
> check the message text to figure out if I need to rebuild the prepared
> queries (rather than building each time I call).
>
> Sylvain had suggested that I use the binary protocol as the exceptions are
> more explicit so I am trying to determine how this can be done (I don't see
> any obvious methods other than the cql ones for calling prepared
> statements).
>
> Regards,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Thrift has a prepare_cql call which returns an ID. Then it has an
>> exececute_cql call which takes the id and a map or variable bindings.
>>
>>
>> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>
>>> Hi all,
>>>
>>> I just realised that the binary protocol is the low-level thrift api
>>> that I was originally using (Cassandra.Client>> get / insert ...). How can
>>> a prepared statement be called through the thrift api (i.e. not the cql
>>> methods)?
>>>
>>> Cheers,
>>>
>>> Stuart
>>>
>>>
>>> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>>
>>>> Hi Sylvain,
>>>>
>>>> Thanks for your response. I am handling the
>>>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>>>> (rather then expecting to build 100000 statements).
>>>>
>>>> I am not familiar with the binary protocol - which class/methods should
>>>> I look at?
>>>>
>>>> Regards,
>>>>
>>>> Stuart
>>>>
>>>>
>>>>
>>>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <
>>>> sylvain@datastax.com> wrote:
>>>>
>>>>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException)
>>>>> are simply returned as InvalidRequestException. The reason for that was a
>>>>> mix of not wanting to change the thrift API too much and because we didn't
>>>>> knew how to return a lot of different exception with thrift without making
>>>>> it horrible to work with. So you'll probably have to parse strings here
>>>>> indeed.
>>>>>
>>>>> This will be cleaner/less fragile if you use the binary protocol as
>>>>> exceptions are more fined grained there.
>>>>>
>>>>> Though taking a step back (and without saying that you shouldn't
>>>>> handle the case where a query is not prepared on the node you contact), if
>>>>> you're really considering preparing more than 100000 statements, I'd
>>>>> suggest that it might be worth benchmarking whether using prepared
>>>>> statements in your case is really going to be worth the trouble. Just
>>>>> saying.
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>>
>>>>>> Hi Sorin,
>>>>>>
>>>>>> The PreparedQueryNotFoundException is not thrown from
>>>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>>>> prepared statements and then re-started cassandra and received the
>>>>>> following exception:
>>>>>>
>>>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>>>> found (either the query was not prepared on this host (maybe the host has
>>>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>>>> 1124421588 has been evicted from the internal cache))
>>>>>>
>>>>>> The best I have been able to come up with is the following:
>>>>>>
>>>>>> try {
>>>>>> client.execute_prepared_cql3_query(psId, bindValues,
>>>>>> ..);
>>>>>> } catch (InvalidRequestException invEx) {
>>>>>> String why = invEx.getWhy();
>>>>>> CLogger.logger().warning(why);
>>>>>> if(why.startsWith("Prepared query with ID")) {
>>>>>> rebuildPreparedStatement(preparedStatement);
>>>>>> client.execute_prepared_cql3_query(psId,
>>>>>> bindValues, ..);
>>>>>> } else {
>>>>>> throw invEx;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Obviously this is pretty fragile and would break if the cassandra
>>>>>> message was changed...but it least it works for now!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Stuart
>>>>>>
>>>>>>
>>>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>>>>>>
>>>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am using Cassandra.Client
>>>>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and run
>>>>>>>> some
>>>>>>>> prepared statements. It is working well but I am unclear as to how
>>>>>>>> long
>>>>>>>> the server side 'caches' the prepared statements. Should a prepared
>>>>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>>>>> limited testing it seems like I can create some prepared statements
>>>>>>>> in
>>>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>>>> reliable/lasting this is i.e. If I called the prepared statement
>>>>>>>> again
>>>>>>>> the next day would it still exist? What about if cassandra was
>>>>>>>> re-started?
>>>>>>>>
>>>>>>>> _Background:_
>>>>>>>>
>>>>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know
>>>>>>>> if
>>>>>>>> these could just be set up once. We felt that using the prepared
>>>>>>>> statements was easier than escaping values within a CQL statement
>>>>>>>> and
>>>>>>>> probably more performant.
>>>>>>>>
>>>>>>>> Thanks in advance for your help.
>>>>>>>>
>>>>>>>>
>>>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Sorin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Hi Edward,
Thanks for your reply - I was already using the prepare/execute cql methods
that you suggested. My problem is that these methods 'mask' the
PreparedQueryNotFoundException as an InvalidRequestException. At present I
catch the InvalidRequestException (when cassandra has been re-started) and
check the message text to figure out if I need to rebuild the prepared
queries (rather than building each time I call).
Sylvain had suggested that I use the binary protocol as the exceptions are
more explicit so I am trying to determine how this can be done (I don't see
any obvious methods other than the cql ones for calling prepared
statements).
Regards,
Stuart
On Tue, Apr 23, 2013 at 4:05 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Thrift has a prepare_cql call which returns an ID. Then it has an
> exececute_cql call which takes the id and a map or variable bindings.
>
>
> On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com>wrote:
>
>> Hi all,
>>
>> I just realised that the binary protocol is the low-level thrift api that
>> I was originally using (Cassandra.Client>> get / insert ...). How can a
>> prepared statement be called through the thrift api (i.e. not the cql
>> methods)?
>>
>> Cheers,
>>
>> Stuart
>>
>>
>> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>>
>>> Hi Sylvain,
>>>
>>> Thanks for your response. I am handling the
>>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>>> (rather then expecting to build 100000 statements).
>>>
>>> I am not familiar with the binary protocol - which class/methods should
>>> I look at?
>>>
>>> Regards,
>>>
>>> Stuart
>>>
>>>
>>>
>>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sylvain@datastax.com
>>> > wrote:
>>>
>>>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException)
>>>> are simply returned as InvalidRequestException. The reason for that was a
>>>> mix of not wanting to change the thrift API too much and because we didn't
>>>> knew how to return a lot of different exception with thrift without making
>>>> it horrible to work with. So you'll probably have to parse strings here
>>>> indeed.
>>>>
>>>> This will be cleaner/less fragile if you use the binary protocol as
>>>> exceptions are more fined grained there.
>>>>
>>>> Though taking a step back (and without saying that you shouldn't handle
>>>> the case where a query is not prepared on the node you contact), if you're
>>>> really considering preparing more than 100000 statements, I'd suggest that
>>>> it might be worth benchmarking whether using prepared statements in your
>>>> case is really going to be worth the trouble. Just saying.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>>
>>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>>>
>>>>> Hi Sorin,
>>>>>
>>>>> The PreparedQueryNotFoundException is not thrown from
>>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>>> prepared statements and then re-started cassandra and received the
>>>>> following exception:
>>>>>
>>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>>> found (either the query was not prepared on this host (maybe the host has
>>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>>> 1124421588 has been evicted from the internal cache))
>>>>>
>>>>> The best I have been able to come up with is the following:
>>>>>
>>>>> try {
>>>>> client.execute_prepared_cql3_query(psId, bindValues,
>>>>> ..);
>>>>> } catch (InvalidRequestException invEx) {
>>>>> String why = invEx.getWhy();
>>>>> CLogger.logger().warning(why);
>>>>> if(why.startsWith("Prepared query with ID")) {
>>>>> rebuildPreparedStatement(preparedStatement);
>>>>> client.execute_prepared_cql3_query(psId,
>>>>> bindValues, ..);
>>>>> } else {
>>>>> throw invEx;
>>>>> }
>>>>> }
>>>>>
>>>>> Obviously this is pretty fragile and would break if the cassandra
>>>>> message was changed...but it least it works for now!
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Stuart
>>>>>
>>>>>
>>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>>>>>
>>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am using Cassandra.Client
>>>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and run
>>>>>>> some
>>>>>>> prepared statements. It is working well but I am unclear as to how
>>>>>>> long
>>>>>>> the server side 'caches' the prepared statements. Should a prepared
>>>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>>>> limited testing it seems like I can create some prepared statements
>>>>>>> in
>>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>>> reliable/lasting this is i.e. If I called the prepared statement
>>>>>>> again
>>>>>>> the next day would it still exist? What about if cassandra was
>>>>>>> re-started?
>>>>>>>
>>>>>>> _Background:_
>>>>>>>
>>>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>>>>>>> these could just be set up once. We felt that using the prepared
>>>>>>> statements was easier than escaping values within a CQL statement and
>>>>>>> probably more performant.
>>>>>>>
>>>>>>> Thanks in advance for your help.
>>>>>>>
>>>>>>>
>>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>>>
>>>>>> Regards,
>>>>>> Sorin
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Edward Capriolo <ed...@gmail.com>.
Thrift has a prepare_cql call which returns an ID. Then it has an
exececute_cql call which takes the id and a map or variable bindings.
On Tue, Apr 23, 2013 at 10:29 AM, Stuart Broad <st...@moogsoft.com> wrote:
> Hi all,
>
> I just realised that the binary protocol is the low-level thrift api that
> I was originally using (Cassandra.Client>> get / insert ...). How can a
> prepared statement be called through the thrift api (i.e. not the cql
> methods)?
>
> Cheers,
>
> Stuart
>
>
> On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com>wrote:
>
>> Hi Sylvain,
>>
>> Thanks for your response. I am handling the
>> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
>> (rather then expecting to build 100000 statements).
>>
>> I am not familiar with the binary protocol - which class/methods should I
>> look at?
>>
>> Regards,
>>
>> Stuart
>>
>>
>>
>> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are
>>> simply returned as InvalidRequestException. The reason for that was a mix
>>> of not wanting to change the thrift API too much and because we didn't knew
>>> how to return a lot of different exception with thrift without making it
>>> horrible to work with. So you'll probably have to parse strings here indeed.
>>>
>>> This will be cleaner/less fragile if you use the binary protocol as
>>> exceptions are more fined grained there.
>>>
>>> Though taking a step back (and without saying that you shouldn't handle
>>> the case where a query is not prepared on the node you contact), if you're
>>> really considering preparing more than 100000 statements, I'd suggest that
>>> it might be worth benchmarking whether using prepared statements in your
>>> case is really going to be worth the trouble. Just saying.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>
>>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>>
>>>> Hi Sorin,
>>>>
>>>> The PreparedQueryNotFoundException is not thrown from
>>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>>> prepared statements and then re-started cassandra and received the
>>>> following exception:
>>>>
>>>> InvalidRequestException(why: Prepared query with ID 1124421588 not
>>>> found (either the query was not prepared on this host (maybe the host has
>>>> been restarted?) or you have prepared more than 100000 queries and queries
>>>> 1124421588 has been evicted from the internal cache))
>>>>
>>>> The best I have been able to come up with is the following:
>>>>
>>>> try {
>>>> client.execute_prepared_cql3_query(psId, bindValues,
>>>> ..);
>>>> } catch (InvalidRequestException invEx) {
>>>> String why = invEx.getWhy();
>>>> CLogger.logger().warning(why);
>>>> if(why.startsWith("Prepared query with ID")) {
>>>> rebuildPreparedStatement(preparedStatement);
>>>> client.execute_prepared_cql3_query(psId,
>>>> bindValues, ..);
>>>> } else {
>>>> throw invEx;
>>>> }
>>>> }
>>>>
>>>> Obviously this is pretty fragile and would break if the cassandra
>>>> message was changed...but it least it works for now!
>>>>
>>>> Cheers,
>>>>
>>>> Stuart
>>>>
>>>>
>>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>>>>
>>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using Cassandra.Client
>>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and run
>>>>>> some
>>>>>> prepared statements. It is working well but I am unclear as to how
>>>>>> long
>>>>>> the server side 'caches' the prepared statements. Should a prepared
>>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>>> limited testing it seems like I can create some prepared statements in
>>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>>> reliable/lasting this is i.e. If I called the prepared statement
>>>>>> again
>>>>>> the next day would it still exist? What about if cassandra was
>>>>>> re-started?
>>>>>>
>>>>>> _Background:_
>>>>>>
>>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>>>>>> these could just be set up once. We felt that using the prepared
>>>>>> statements was easier than escaping values within a CQL statement and
>>>>>> probably more performant.
>>>>>>
>>>>>> Thanks in advance for your help.
>>>>>>
>>>>>>
>>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>>> statements, the least recently used ones will vanish. You'll get the
>>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>>
>>>>> Regards,
>>>>> Sorin
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Hi all,
I just realised that the binary protocol is the low-level thrift api that I
was originally using (Cassandra.Client>> get / insert ...). How can a
prepared statement be called through the thrift api (i.e. not the cql
methods)?
Cheers,
Stuart
On Tue, Apr 23, 2013 at 11:48 AM, Stuart Broad <st...@moogsoft.com> wrote:
> Hi Sylvain,
>
> Thanks for your response. I am handling the
> 'PreparedQueryNotFoundException' more for the case of a cassandra re-start
> (rather then expecting to build 100000 statements).
>
> I am not familiar with the binary protocol - which class/methods should I
> look at?
>
> Regards,
>
> Stuart
>
>
>
> On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are
>> simply returned as InvalidRequestException. The reason for that was a mix
>> of not wanting to change the thrift API too much and because we didn't knew
>> how to return a lot of different exception with thrift without making it
>> horrible to work with. So you'll probably have to parse strings here indeed.
>>
>> This will be cleaner/less fragile if you use the binary protocol as
>> exceptions are more fined grained there.
>>
>> Though taking a step back (and without saying that you shouldn't handle
>> the case where a query is not prepared on the node you contact), if you're
>> really considering preparing more than 100000 statements, I'd suggest that
>> it might be worth benchmarking whether using prepared statements in your
>> case is really going to be worth the trouble. Just saying.
>>
>> --
>> Sylvain
>>
>>
>>
>> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>>
>>> Hi Sorin,
>>>
>>> The PreparedQueryNotFoundException is not thrown from
>>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>>> prepared statements and then re-started cassandra and received the
>>> following exception:
>>>
>>> InvalidRequestException(why: Prepared query with ID 1124421588 not found
>>> (either the query was not prepared on this host (maybe the host has been
>>> restarted?) or you have prepared more than 100000 queries and queries
>>> 1124421588 has been evicted from the internal cache))
>>>
>>> The best I have been able to come up with is the following:
>>>
>>> try {
>>> client.execute_prepared_cql3_query(psId, bindValues, ..);
>>> } catch (InvalidRequestException invEx) {
>>> String why = invEx.getWhy();
>>> CLogger.logger().warning(why);
>>> if(why.startsWith("Prepared query with ID")) {
>>> rebuildPreparedStatement(preparedStatement);
>>> client.execute_prepared_cql3_query(psId, bindValues,
>>> ..);
>>> } else {
>>> throw invEx;
>>> }
>>> }
>>>
>>> Obviously this is pretty fragile and would break if the cassandra
>>> message was changed...but it least it works for now!
>>>
>>> Cheers,
>>>
>>> Stuart
>>>
>>>
>>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>>>
>>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am using Cassandra.Client
>>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and run
>>>>> some
>>>>> prepared statements. It is working well but I am unclear as to how
>>>>> long
>>>>> the server side 'caches' the prepared statements. Should a prepared
>>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>>> limited testing it seems like I can create some prepared statements in
>>>>> one Cassandra.Client and use in another but I am not sure how
>>>>> reliable/lasting this is i.e. If I called the prepared statement again
>>>>> the next day would it still exist? What about if cassandra was
>>>>> re-started?
>>>>>
>>>>> _Background:_
>>>>>
>>>>> I am creating prepared statements for batch updates of pre-defined
>>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>>>>> these could just be set up once. We felt that using the prepared
>>>>> statements was easier than escaping values within a CQL statement and
>>>>> probably more performant.
>>>>>
>>>>> Thanks in advance for your help.
>>>>>
>>>>>
>>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>>> statements, the least recently used ones will vanish. You'll get the
>>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>>
>>>> Regards,
>>>> Sorin
>>>>
>>>>
>>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Hi Sylvain,
Thanks for your response. I am handling the
'PreparedQueryNotFoundException' more for the case of a cassandra re-start
(rather then expecting to build 100000 statements).
I am not familiar with the binary protocol - which class/methods should I
look at?
Regards,
Stuart
On Tue, Apr 23, 2013 at 11:29 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are
> simply returned as InvalidRequestException. The reason for that was a mix
> of not wanting to change the thrift API too much and because we didn't knew
> how to return a lot of different exception with thrift without making it
> horrible to work with. So you'll probably have to parse strings here indeed.
>
> This will be cleaner/less fragile if you use the binary protocol as
> exceptions are more fined grained there.
>
> Though taking a step back (and without saying that you shouldn't handle
> the case where a query is not prepared on the node you contact), if you're
> really considering preparing more than 100000 statements, I'd suggest that
> it might be worth benchmarking whether using prepared statements in your
> case is really going to be worth the trouble. Just saying.
>
> --
> Sylvain
>
>
>
> On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com>wrote:
>
>> Hi Sorin,
>>
>> The PreparedQueryNotFoundException is not thrown from
>> Cassandra.Client>>execute_prepared_cql3_query method. I created some
>> prepared statements and then re-started cassandra and received the
>> following exception:
>>
>> InvalidRequestException(why: Prepared query with ID 1124421588 not found
>> (either the query was not prepared on this host (maybe the host has been
>> restarted?) or you have prepared more than 100000 queries and queries
>> 1124421588 has been evicted from the internal cache))
>>
>> The best I have been able to come up with is the following:
>>
>> try {
>> client.execute_prepared_cql3_query(psId, bindValues, ..);
>> } catch (InvalidRequestException invEx) {
>> String why = invEx.getWhy();
>> CLogger.logger().warning(why);
>> if(why.startsWith("Prepared query with ID")) {
>> rebuildPreparedStatement(preparedStatement);
>> client.execute_prepared_cql3_query(psId, bindValues,
>> ..);
>> } else {
>> throw invEx;
>> }
>> }
>>
>> Obviously this is pretty fragile and would break if the cassandra message
>> was changed...but it least it works for now!
>>
>> Cheers,
>>
>> Stuart
>>
>>
>> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>>
>>> On 2013-04-19 13:57, Stuart Broad wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using Cassandra.Client
>>>> prepare_cql3_query/execute_**prepared_cql3_query to create and run some
>>>> prepared statements. It is working well but I am unclear as to how long
>>>> the server side 'caches' the prepared statements. Should a prepared
>>>> statement be prepared for every new Cassandra.Client? Based on my
>>>> limited testing it seems like I can create some prepared statements in
>>>> one Cassandra.Client and use in another but I am not sure how
>>>> reliable/lasting this is i.e. If I called the prepared statement again
>>>> the next day would it still exist? What about if cassandra was
>>>> re-started?
>>>>
>>>> _Background:_
>>>>
>>>> I am creating prepared statements for batch updates of pre-defined
>>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>>>> these could just be set up once. We felt that using the prepared
>>>> statements was easier than escaping values within a CQL statement and
>>>> probably more performant.
>>>>
>>>> Thanks in advance for your help.
>>>>
>>>>
>>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>>> statements has a size of 100,000. So if you prepare more than 100 thousand
>>> statements, the least recently used ones will vanish. You'll get the
>>> exception PreparedQueryNotFoundException**, code 0x2500.
>>>
>>> Regards,
>>> Sorin
>>>
>>>
>>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Sylvain Lebresne <sy...@datastax.com>.
In thrift, a lot of exceptions (like PreparedQueryNotFoundException) are
simply returned as InvalidRequestException. The reason for that was a mix
of not wanting to change the thrift API too much and because we didn't knew
how to return a lot of different exception with thrift without making it
horrible to work with. So you'll probably have to parse strings here indeed.
This will be cleaner/less fragile if you use the binary protocol as
exceptions are more fined grained there.
Though taking a step back (and without saying that you shouldn't handle the
case where a query is not prepared on the node you contact), if you're
really considering preparing more than 100000 statements, I'd suggest that
it might be worth benchmarking whether using prepared statements in your
case is really going to be worth the trouble. Just saying.
--
Sylvain
On Tue, Apr 23, 2013 at 12:14 PM, Stuart Broad <st...@moogsoft.com> wrote:
> Hi Sorin,
>
> The PreparedQueryNotFoundException is not thrown from
> Cassandra.Client>>execute_prepared_cql3_query method. I created some
> prepared statements and then re-started cassandra and received the
> following exception:
>
> InvalidRequestException(why: Prepared query with ID 1124421588 not found
> (either the query was not prepared on this host (maybe the host has been
> restarted?) or you have prepared more than 100000 queries and queries
> 1124421588 has been evicted from the internal cache))
>
> The best I have been able to come up with is the following:
>
> try {
> client.execute_prepared_cql3_query(psId, bindValues, ..);
> } catch (InvalidRequestException invEx) {
> String why = invEx.getWhy();
> CLogger.logger().warning(why);
> if(why.startsWith("Prepared query with ID")) {
> rebuildPreparedStatement(preparedStatement);
> client.execute_prepared_cql3_query(psId, bindValues,
> ..);
> } else {
> throw invEx;
> }
> }
>
> Obviously this is pretty fragile and would break if the cassandra message
> was changed...but it least it works for now!
>
> Cheers,
>
> Stuart
>
>
> On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com>wrote:
>
>> On 2013-04-19 13:57, Stuart Broad wrote:
>>
>>> Hi,
>>>
>>> I am using Cassandra.Client
>>> prepare_cql3_query/execute_**prepared_cql3_query to create and run some
>>> prepared statements. It is working well but I am unclear as to how long
>>> the server side 'caches' the prepared statements. Should a prepared
>>> statement be prepared for every new Cassandra.Client? Based on my
>>> limited testing it seems like I can create some prepared statements in
>>> one Cassandra.Client and use in another but I am not sure how
>>> reliable/lasting this is i.e. If I called the prepared statement again
>>> the next day would it still exist? What about if cassandra was
>>> re-started?
>>>
>>> _Background:_
>>>
>>> I am creating prepared statements for batch updates of pre-defined
>>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>>> these could just be set up once. We felt that using the prepared
>>> statements was easier than escaping values within a CQL statement and
>>> probably more performant.
>>>
>>> Thanks in advance for your help.
>>>
>>>
>> I've looked in Cassandra's code (v1.2.3). The cache of prepared
>> statements has a size of 100,000. So if you prepare more than 100 thousand
>> statements, the least recently used ones will vanish. You'll get the
>> exception PreparedQueryNotFoundException**, code 0x2500.
>>
>> Regards,
>> Sorin
>>
>>
>>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Stuart Broad <st...@moogsoft.com>.
Hi Sorin,
The PreparedQueryNotFoundException is not thrown from
Cassandra.Client>>execute_prepared_cql3_query method. I created some
prepared statements and then re-started cassandra and received the
following exception:
InvalidRequestException(why: Prepared query with ID 1124421588 not found
(either the query was not prepared on this host (maybe the host has been
restarted?) or you have prepared more than 100000 queries and queries
1124421588 has been evicted from the internal cache))
The best I have been able to come up with is the following:
try {
client.execute_prepared_cql3_query(psId, bindValues, ..);
} catch (InvalidRequestException invEx) {
String why = invEx.getWhy();
CLogger.logger().warning(why);
if(why.startsWith("Prepared query with ID")) {
rebuildPreparedStatement(preparedStatement);
client.execute_prepared_cql3_query(psId, bindValues,
..);
} else {
throw invEx;
}
}
Obviously this is pretty fragile and would break if the cassandra message
was changed...but it least it works for now!
Cheers,
Stuart
On Sun, Apr 21, 2013 at 11:51 AM, Sorin Manolache <so...@gmail.com> wrote:
> On 2013-04-19 13:57, Stuart Broad wrote:
>
>> Hi,
>>
>> I am using Cassandra.Client
>> prepare_cql3_query/execute_**prepared_cql3_query to create and run some
>> prepared statements. It is working well but I am unclear as to how long
>> the server side 'caches' the prepared statements. Should a prepared
>> statement be prepared for every new Cassandra.Client? Based on my
>> limited testing it seems like I can create some prepared statements in
>> one Cassandra.Client and use in another but I am not sure how
>> reliable/lasting this is i.e. If I called the prepared statement again
>> the next day would it still exist? What about if cassandra was
>> re-started?
>>
>> _Background:_
>>
>> I am creating prepared statements for batch updates of pre-defined
>> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
>> these could just be set up once. We felt that using the prepared
>> statements was easier than escaping values within a CQL statement and
>> probably more performant.
>>
>> Thanks in advance for your help.
>>
>>
> I've looked in Cassandra's code (v1.2.3). The cache of prepared statements
> has a size of 100,000. So if you prepare more than 100 thousand statements,
> the least recently used ones will vanish. You'll get the exception
> PreparedQueryNotFoundException**, code 0x2500.
>
> Regards,
> Sorin
>
>
>
Re: Prepared Statement - cache duration (CQL3 - Cassandra 1.2.4)
Posted by Sorin Manolache <so...@gmail.com>.
On 2013-04-19 13:57, Stuart Broad wrote:
> Hi,
>
> I am using Cassandra.Client
> prepare_cql3_query/execute_prepared_cql3_query to create and run some
> prepared statements. It is working well but I am unclear as to how long
> the server side 'caches' the prepared statements. Should a prepared
> statement be prepared for every new Cassandra.Client? Based on my
> limited testing it seems like I can create some prepared statements in
> one Cassandra.Client and use in another but I am not sure how
> reliable/lasting this is i.e. If I called the prepared statement again
> the next day would it still exist? What about if cassandra was re-started?
>
> _Background:_
> I am creating prepared statements for batch updates of pre-defined
> lengths (e.g. 10000, 1000, 500, 250, 50, 10, 1) and wanted to know if
> these could just be set up once. We felt that using the prepared
> statements was easier than escaping values within a CQL statement and
> probably more performant.
>
> Thanks in advance for your help.
>
I've looked in Cassandra's code (v1.2.3). The cache of prepared
statements has a size of 100,000. So if you prepare more than 100
thousand statements, the least recently used ones will vanish. You'll
get the exception PreparedQueryNotFoundException, code 0x2500.
Regards,
Sorin