You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Anishek Agarwal <an...@gmail.com> on 2015/03/23 13:29:18 UTC
write timeout
Hello,
I am using a single node server class machine with 16 CPUs with 32GB RAM
with a single drive attached to it.
my table structure is as below
CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2
set<text>, lat float, lon float, a bigint, primary key (id, ts));
I am trying to insert 300 entries per partition key with 4000 partition
keys using 25 threads. Configurations
write_request_timeout_in_ms: 5000
concurrent_writes: 32
heap space : 8GB
Client side timeout is 12 sec using datastax java driver.
Consistency level: ONE
With the above configuration i try to run it 10 times to eventually
generate around
300 * 4000 * 10 = 12000000 entries,
When i run this after the first few runs i get a WriteTimeout exception at
client with "1 replica were required but only 0 acknowledged the write"
message.
There are no errors in server log. Why does this error come how do i know
what is the limit I should limit concurrent writes to a single node to.
Looking at iostat disk utilization seems to be at 1-3% when running this.
Please let me know if anything else is required.
Regards,
Anishek
Re: write timeout
Posted by Anishek Agarwal <an...@gmail.com>.
after a few more trials I was able to spot some errors in the system.log,
attaching the log here for reference.
Looks like "test-t1-jb-78" is opened by SSTableReader in some batch but
then later there is compaction that is happening on it where in the file is
not found. Any idea what is happening?
On Tue, Mar 24, 2015 at 9:54 AM, Anishek Agarwal <an...@gmail.com> wrote:
> Bug report posted @ https://issues.apache.org/jira/browse/CASSANDRA-9023
>
> On Mon, Mar 23, 2015 at 11:21 PM, Robert Coli <rc...@eventbrite.com>
> wrote:
>
>> On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <br...@gmail.com>
>> wrote:
>>
>> > My group is seeing the same thing and also can not figure out why its
>> > happening.
>> >
>> > On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com>
>> > wrote:
>> >
>> >> Forgot to mention I am using Cassandra 2.0.13
>> >>
>> >
>> This seems like a rather significant bug in the most recent stable
>> version.
>> In this case, I would tend to file a JIRA first and then ask the mailing
>> list second.
>>
>> Could one or both of you file steps-to-reproduce with a JIRA at
>> http://issues.apache.org?
>>
>> =Rob
>>
>
>
Re: write timeout
Posted by Anishek Agarwal <an...@gmail.com>.
after a few more trials I was able to spot some errors in the system.log,
attaching the log here for reference.
Looks like "test-t1-jb-78" is opened by SSTableReader in some batch but
then later there is compaction that is happening on it where in the file is
not found. Any idea what is happening?
On Tue, Mar 24, 2015 at 9:54 AM, Anishek Agarwal <an...@gmail.com> wrote:
> Bug report posted @ https://issues.apache.org/jira/browse/CASSANDRA-9023
>
> On Mon, Mar 23, 2015 at 11:21 PM, Robert Coli <rc...@eventbrite.com>
> wrote:
>
>> On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <br...@gmail.com>
>> wrote:
>>
>> > My group is seeing the same thing and also can not figure out why its
>> > happening.
>> >
>> > On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com>
>> > wrote:
>> >
>> >> Forgot to mention I am using Cassandra 2.0.13
>> >>
>> >
>> This seems like a rather significant bug in the most recent stable
>> version.
>> In this case, I would tend to file a JIRA first and then ask the mailing
>> list second.
>>
>> Could one or both of you file steps-to-reproduce with a JIRA at
>> http://issues.apache.org?
>>
>> =Rob
>>
>
>
Re: write timeout
Posted by Anishek Agarwal <an...@gmail.com>.
Bug report posted @ https://issues.apache.org/jira/browse/CASSANDRA-9023
On Mon, Mar 23, 2015 at 11:21 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <br...@gmail.com>
> wrote:
>
> > My group is seeing the same thing and also can not figure out why its
> > happening.
> >
> > On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com>
> > wrote:
> >
> >> Forgot to mention I am using Cassandra 2.0.13
> >>
> >
> This seems like a rather significant bug in the most recent stable version.
> In this case, I would tend to file a JIRA first and then ask the mailing
> list second.
>
> Could one or both of you file steps-to-reproduce with a JIRA at
> http://issues.apache.org?
>
> =Rob
>
Re: write timeout
Posted by Anishek Agarwal <an...@gmail.com>.
Bug report posted @ https://issues.apache.org/jira/browse/CASSANDRA-9023
On Mon, Mar 23, 2015 at 11:21 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <br...@gmail.com>
> wrote:
>
> > My group is seeing the same thing and also can not figure out why its
> > happening.
> >
> > On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com>
> > wrote:
> >
> >> Forgot to mention I am using Cassandra 2.0.13
> >>
> >
> This seems like a rather significant bug in the most recent stable version.
> In this case, I would tend to file a JIRA first and then ask the mailing
> list second.
>
> Could one or both of you file steps-to-reproduce with a JIRA at
> http://issues.apache.org?
>
> =Rob
>
Re: write timeout
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <br...@gmail.com> wrote:
> My group is seeing the same thing and also can not figure out why its
> happening.
>
> On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> Forgot to mention I am using Cassandra 2.0.13
>>
>
This seems like a rather significant bug in the most recent stable version.
In this case, I would tend to file a JIRA first and then ask the mailing
list second.
Could one or both of you file steps-to-reproduce with a JIRA at
http://issues.apache.org?
=Rob
Re: write timeout
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Mar 23, 2015 at 7:27 AM, Brian Tarbox <br...@gmail.com> wrote:
> My group is seeing the same thing and also can not figure out why its
> happening.
>
> On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> Forgot to mention I am using Cassandra 2.0.13
>>
>
This seems like a rather significant bug in the most recent stable version.
In this case, I would tend to file a JIRA first and then ask the mailing
list second.
Could one or both of you file steps-to-reproduce with a JIRA at
http://issues.apache.org?
=Rob
Re: write timeout
Posted by Brian Tarbox <br...@gmail.com>.
My group is seeing the same thing and also can not figure out why its
happening.
On Mon, Mar 23, 2015 at 8:36 AM, Anishek Agarwal <an...@gmail.com> wrote:
> Forgot to mention I am using Cassandra 2.0.13
>
> On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I am using a single node server class machine with 16 CPUs with 32GB RAM
>> with a single drive attached to it.
>>
>> my table structure is as below
>>
>> CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2 set<text>, lat float, lon float, a bigint, primary key (id, ts));
>>
>> I am trying to insert 300 entries per partition key with 4000 partition
>> keys using 25 threads. Configurations
>>
>> write_request_timeout_in_ms: 5000
>> concurrent_writes: 32
>> heap space : 8GB
>>
>> Client side timeout is 12 sec using datastax java driver.
>> Consistency level: ONE
>>
>> With the above configuration i try to run it 10 times to eventually
>> generate around
>>
>> 300 * 4000 * 10 = 12000000 entries,
>>
>> When i run this after the first few runs i get a WriteTimeout exception
>> at client with "1 replica were required but only 0 acknowledged the write"
>> message.
>>
>> There are no errors in server log. Why does this error come how do i know
>> what is the limit I should limit concurrent writes to a single node to.
>>
>>
>> Looking at iostat disk utilization seems to be at 1-3% when running this.
>>
>> Please let me know if anything else is required.
>>
>> Regards,
>> Anishek
>>
>>
>
--
http://about.me/BrianTarbox
Re: write timeout
Posted by Anishek Agarwal <an...@gmail.com>.
Forgot to mention I am using Cassandra 2.0.13
On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal <an...@gmail.com> wrote:
> Hello,
>
> I am using a single node server class machine with 16 CPUs with 32GB RAM
> with a single drive attached to it.
>
> my table structure is as below
>
> CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2 set<text>, lat float, lon float, a bigint, primary key (id, ts));
>
> I am trying to insert 300 entries per partition key with 4000 partition
> keys using 25 threads. Configurations
>
> write_request_timeout_in_ms: 5000
> concurrent_writes: 32
> heap space : 8GB
>
> Client side timeout is 12 sec using datastax java driver.
> Consistency level: ONE
>
> With the above configuration i try to run it 10 times to eventually
> generate around
>
> 300 * 4000 * 10 = 12000000 entries,
>
> When i run this after the first few runs i get a WriteTimeout exception at
> client with "1 replica were required but only 0 acknowledged the write"
> message.
>
> There are no errors in server log. Why does this error come how do i know
> what is the limit I should limit concurrent writes to a single node to.
>
>
> Looking at iostat disk utilization seems to be at 1-3% when running this.
>
> Please let me know if anything else is required.
>
> Regards,
> Anishek
>
>
Re: write timeout
Posted by Anishek Agarwal <an...@gmail.com>.
Forgot to mention I am using Cassandra 2.0.13
On Mon, Mar 23, 2015 at 5:59 PM, Anishek Agarwal <an...@gmail.com> wrote:
> Hello,
>
> I am using a single node server class machine with 16 CPUs with 32GB RAM
> with a single drive attached to it.
>
> my table structure is as below
>
> CREATE TABLE t1(id bigint, ts timestamp, cat1 set<text>, cat2 set<text>, lat float, lon float, a bigint, primary key (id, ts));
>
> I am trying to insert 300 entries per partition key with 4000 partition
> keys using 25 threads. Configurations
>
> write_request_timeout_in_ms: 5000
> concurrent_writes: 32
> heap space : 8GB
>
> Client side timeout is 12 sec using datastax java driver.
> Consistency level: ONE
>
> With the above configuration i try to run it 10 times to eventually
> generate around
>
> 300 * 4000 * 10 = 12000000 entries,
>
> When i run this after the first few runs i get a WriteTimeout exception at
> client with "1 replica were required but only 0 acknowledged the write"
> message.
>
> There are no errors in server log. Why does this error come how do i know
> what is the limit I should limit concurrent writes to a single node to.
>
>
> Looking at iostat disk utilization seems to be at 1-3% when running this.
>
> Please let me know if anything else is required.
>
> Regards,
> Anishek
>
>