You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by George Sigletos <si...@textkernel.nl> on 2016/09/28 16:50:06 UTC
TRUNCATE throws OperationTimedOut randomly
Hello,
I keep executing a TRUNCATE command on an empty table and it throws
OperationTimedOut randomly:
cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01
cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01
Having a 3 node cluster running 2.1.14. No connectivity problems. Has
anybody come across the same error?
Thanks,
George
Re: TRUNCATE throws OperationTimedOut randomly
Posted by Romain Hardouin <ro...@yahoo.fr>.
Hi,
@Edward > In older versions you can not control when this call will timeout,truncate_request_timeout_in_ms is available for many years, starting from 1.2. Maybe you have another setting parameter in mind?
@GeorgeTry to put cassandra logs in debug
Best,
Romain
Le Mercredi 28 septembre 2016 20h31, George Sigletos <si...@textkernel.nl> a écrit :
Even when I set a lower request-timeout in order to trigger a timeout, still no WARN or ERROR in the logs
On Wed, Sep 28, 2016 at 8:22 PM, George Sigletos <si...@textkernel.nl> wrote:
Hi Joaquin,
Unfortunately neither WARN nor ERROR found in the system logs across the cluster when executing truncate. Sometimes it executes immediately, other times it takes 25 seconds, given that I have connected with --request-timeout=30 seconds.
The nodes are a bit busy compacting. On a freshly restarted cluster, truncate seems to work without problems.
Some warnings that I see around that time but not exactly when executing truncate are:
WARN [CompactionExecutor:2] 2016-09-28 20:03:29,646 SSTableWriter.java:241 - Compacting large partition system/hints:6f2c3b31-4975- 470b-8f91-e706be89a83a (133819308 bytes
Kind regards,
George
On Wed, Sep 28, 2016 at 7:54 PM, Joaquin Casares <jo...@thelastpickle.com> wrote:
Hi George,
Try grepping for WARN and ERROR on the system.logs across all nodes when you run the command. Could you post any of the recent stacktraces that you see?
Cheers,
Joaquin Casares
ConsultantAustin, TX
Apache Cassandra Consultinghttp://www.thelastpickle.com
On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos <si...@textkernel.nl> wrote:
Thanks a lot for your reply.
I understand that truncate is an expensive operation. But throwing a timeout while truncating a table that is already empty?
A workaround is to set a high --request-timeout when connecting. Even 20 seconds is not always enough
Kind regards,
George
On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo <ed...@gmail.com> wrote:
Truncate does a few things (based on version)
truncate takes snapshots truncate causes a flush
in very old versions truncate causes a schema migration.
In newer versions like cassandra 3.4 you have this knob.
# How long the coordinator should wait for truncates to complete# (This can be much longer, because unless auto_snapshot is disabled# we need to flush first so we can snapshot before removing the data.)truncate_request_timeout_in_ms : 60000
In older versions you can not control when this call will timeout, it is fairly normal that it does!
On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <si...@textkernel.nl> wrote:
Hello,
I keep executing a TRUNCATE command on an empty table and it throws OperationTimedOut randomly:
cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01
cassandra@cqlsh> truncate test.mytable;
OperationTimedOut: errors={}, last_host=cassiebeta-01
Having a 3 node cluster running 2.1.14. No connectivity problems. Has anybody come across the same error?
Thanks,
George
Re: TRUNCATE throws OperationTimedOut randomly
Posted by George Sigletos <si...@textkernel.nl>.
Even when I set a lower request-timeout in order to trigger a timeout,
still no WARN or ERROR in the logs
On Wed, Sep 28, 2016 at 8:22 PM, George Sigletos <si...@textkernel.nl>
wrote:
> Hi Joaquin,
>
> Unfortunately neither WARN nor ERROR found in the system logs across the
> cluster when executing truncate. Sometimes it executes immediately, other
> times it takes 25 seconds, given that I have connected with
> --request-timeout=30 seconds.
>
> The nodes are a bit busy compacting. On a freshly restarted cluster,
> truncate seems to work without problems.
>
> Some warnings that I see around that time but not exactly when executing
> truncate are:
> WARN [CompactionExecutor:2] 2016-09-28 20:03:29,646
> SSTableWriter.java:241 - Compacting large partition
> system/hints:6f2c3b31-4975-470b-8f91-e706be89a83a (133819308 bytes
>
> Kind regards,
> George
>
> On Wed, Sep 28, 2016 at 7:54 PM, Joaquin Casares <
> joaquin@thelastpickle.com> wrote:
>
>> Hi George,
>>
>> Try grepping for WARN and ERROR on the system.logs across all nodes when
>> you run the command. Could you post any of the recent stacktraces that you
>> see?
>>
>> Cheers,
>>
>> Joaquin Casares
>> Consultant
>> Austin, TX
>>
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos <sigletos@textkernel.nl
>> > wrote:
>>
>>> Thanks a lot for your reply.
>>>
>>> I understand that truncate is an expensive operation. But throwing a
>>> timeout while truncating a table that is already empty?
>>>
>>> A workaround is to set a high --request-timeout when connecting. Even 20
>>> seconds is not always enough
>>>
>>> Kind regards,
>>> George
>>>
>>>
>>> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo <ed...@gmail.com>
>>> wrote:
>>>
>>>> Truncate does a few things (based on version)
>>>> truncate takes snapshots
>>>> truncate causes a flush
>>>> in very old versions truncate causes a schema migration.
>>>>
>>>> In newer versions like cassandra 3.4 you have this knob.
>>>>
>>>> # How long the coordinator should wait for truncates to complete
>>>> # (This can be much longer, because unless auto_snapshot is disabled
>>>> # we need to flush first so we can snapshot before removing the data.)
>>>> truncate_request_timeout_in_ms: 60000
>>>>
>>>>
>>>> In older versions you can not control when this call will timeout, it
>>>> is fairly normal that it does!
>>>>
>>>>
>>>> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <
>>>> sigletos@textkernel.nl> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I keep executing a TRUNCATE command on an empty table and it throws
>>>>> OperationTimedOut randomly:
>>>>>
>>>>> cassandra@cqlsh> truncate test.mytable;
>>>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>>> cassandra@cqlsh> truncate test.mytable;
>>>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>>>
>>>>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>>>>> anybody come across the same error?
>>>>>
>>>>> Thanks,
>>>>> George
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: TRUNCATE throws OperationTimedOut randomly
Posted by George Sigletos <si...@textkernel.nl>.
Hi Joaquin,
Unfortunately neither WARN nor ERROR found in the system logs across the
cluster when executing truncate. Sometimes it executes immediately, other
times it takes 25 seconds, given that I have connected with
--request-timeout=30 seconds.
The nodes are a bit busy compacting. On a freshly restarted cluster,
truncate seems to work without problems.
Some warnings that I see around that time but not exactly when executing
truncate are:
WARN [CompactionExecutor:2] 2016-09-28 20:03:29,646 SSTableWriter.java:241
- Compacting large partition
system/hints:6f2c3b31-4975-470b-8f91-e706be89a83a (133819308 bytes
Kind regards,
George
On Wed, Sep 28, 2016 at 7:54 PM, Joaquin Casares <jo...@thelastpickle.com>
wrote:
> Hi George,
>
> Try grepping for WARN and ERROR on the system.logs across all nodes when
> you run the command. Could you post any of the recent stacktraces that you
> see?
>
> Cheers,
>
> Joaquin Casares
> Consultant
> Austin, TX
>
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos <si...@textkernel.nl>
> wrote:
>
>> Thanks a lot for your reply.
>>
>> I understand that truncate is an expensive operation. But throwing a
>> timeout while truncating a table that is already empty?
>>
>> A workaround is to set a high --request-timeout when connecting. Even 20
>> seconds is not always enough
>>
>> Kind regards,
>> George
>>
>>
>> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo <ed...@gmail.com>
>> wrote:
>>
>>> Truncate does a few things (based on version)
>>> truncate takes snapshots
>>> truncate causes a flush
>>> in very old versions truncate causes a schema migration.
>>>
>>> In newer versions like cassandra 3.4 you have this knob.
>>>
>>> # How long the coordinator should wait for truncates to complete
>>> # (This can be much longer, because unless auto_snapshot is disabled
>>> # we need to flush first so we can snapshot before removing the data.)
>>> truncate_request_timeout_in_ms: 60000
>>>
>>>
>>> In older versions you can not control when this call will timeout, it is
>>> fairly normal that it does!
>>>
>>>
>>> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <
>>> sigletos@textkernel.nl> wrote:
>>>
>>>> Hello,
>>>>
>>>> I keep executing a TRUNCATE command on an empty table and it throws
>>>> OperationTimedOut randomly:
>>>>
>>>> cassandra@cqlsh> truncate test.mytable;
>>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>> cassandra@cqlsh> truncate test.mytable;
>>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>>
>>>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>>>> anybody come across the same error?
>>>>
>>>> Thanks,
>>>> George
>>>>
>>>>
>>>
>>
>
Re: TRUNCATE throws OperationTimedOut randomly
Posted by Joaquin Casares <jo...@thelastpickle.com>.
Hi George,
Try grepping for WARN and ERROR on the system.logs across all nodes when
you run the command. Could you post any of the recent stacktraces that you
see?
Cheers,
Joaquin Casares
Consultant
Austin, TX
Apache Cassandra Consulting
http://www.thelastpickle.com
On Wed, Sep 28, 2016 at 12:43 PM, George Sigletos <si...@textkernel.nl>
wrote:
> Thanks a lot for your reply.
>
> I understand that truncate is an expensive operation. But throwing a
> timeout while truncating a table that is already empty?
>
> A workaround is to set a high --request-timeout when connecting. Even 20
> seconds is not always enough
>
> Kind regards,
> George
>
>
> On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
>> Truncate does a few things (based on version)
>> truncate takes snapshots
>> truncate causes a flush
>> in very old versions truncate causes a schema migration.
>>
>> In newer versions like cassandra 3.4 you have this knob.
>>
>> # How long the coordinator should wait for truncates to complete
>> # (This can be much longer, because unless auto_snapshot is disabled
>> # we need to flush first so we can snapshot before removing the data.)
>> truncate_request_timeout_in_ms: 60000
>>
>>
>> In older versions you can not control when this call will timeout, it is
>> fairly normal that it does!
>>
>>
>> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <sigletos@textkernel.nl
>> > wrote:
>>
>>> Hello,
>>>
>>> I keep executing a TRUNCATE command on an empty table and it throws
>>> OperationTimedOut randomly:
>>>
>>> cassandra@cqlsh> truncate test.mytable;
>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>> cassandra@cqlsh> truncate test.mytable;
>>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>>
>>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>>> anybody come across the same error?
>>>
>>> Thanks,
>>> George
>>>
>>>
>>
>
Re: TRUNCATE throws OperationTimedOut randomly
Posted by George Sigletos <si...@textkernel.nl>.
Thanks a lot for your reply.
I understand that truncate is an expensive operation. But throwing a
timeout while truncating a table that is already empty?
A workaround is to set a high --request-timeout when connecting. Even 20
seconds is not always enough
Kind regards,
George
On Wed, Sep 28, 2016 at 6:59 PM, Edward Capriolo <ed...@gmail.com>
wrote:
> Truncate does a few things (based on version)
> truncate takes snapshots
> truncate causes a flush
> in very old versions truncate causes a schema migration.
>
> In newer versions like cassandra 3.4 you have this knob.
>
> # How long the coordinator should wait for truncates to complete
> # (This can be much longer, because unless auto_snapshot is disabled
> # we need to flush first so we can snapshot before removing the data.)
> truncate_request_timeout_in_ms: 60000
>
>
> In older versions you can not control when this call will timeout, it is
> fairly normal that it does!
>
>
> On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <si...@textkernel.nl>
> wrote:
>
>> Hello,
>>
>> I keep executing a TRUNCATE command on an empty table and it throws
>> OperationTimedOut randomly:
>>
>> cassandra@cqlsh> truncate test.mytable;
>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>> cassandra@cqlsh> truncate test.mytable;
>> OperationTimedOut: errors={}, last_host=cassiebeta-01
>>
>> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
>> anybody come across the same error?
>>
>> Thanks,
>> George
>>
>>
>
Re: TRUNCATE throws OperationTimedOut randomly
Posted by Edward Capriolo <ed...@gmail.com>.
Truncate does a few things (based on version)
truncate takes snapshots
truncate causes a flush
in very old versions truncate causes a schema migration.
In newer versions like cassandra 3.4 you have this knob.
# How long the coordinator should wait for truncates to complete
# (This can be much longer, because unless auto_snapshot is disabled
# we need to flush first so we can snapshot before removing the data.)
truncate_request_timeout_in_ms: 60000
In older versions you can not control when this call will timeout, it is
fairly normal that it does!
On Wed, Sep 28, 2016 at 12:50 PM, George Sigletos <si...@textkernel.nl>
wrote:
> Hello,
>
> I keep executing a TRUNCATE command on an empty table and it throws
> OperationTimedOut randomly:
>
> cassandra@cqlsh> truncate test.mytable;
> OperationTimedOut: errors={}, last_host=cassiebeta-01
> cassandra@cqlsh> truncate test.mytable;
> OperationTimedOut: errors={}, last_host=cassiebeta-01
>
> Having a 3 node cluster running 2.1.14. No connectivity problems. Has
> anybody come across the same error?
>
> Thanks,
> George
>
>