You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Badrinarayanan S <ba...@fifthgentech.com> on 2011/06/17 20:29:42 UTC
PIG Cassandra - Performance
Hi,
In our production Cassandra systems we are observing the time taken by same
PIG script keeps increasing each and every day. The PIG scripts reads data
for a day at a time from a Cassandra Column Family. The number of rows the
PIG script is expected to return is almost same every day, however every day
the amount of rows we are storing in Cassandra is increasing. We haven't
changed the default setting for multiquery, it is by default enabled.
Could this increase in PIG script execution time be related to the
increasing number of rows in Cassandra every day?
Related to this I was trying to understand the behavior of LOAD statement.
Does LOAD statement reads all the data from Cassandra and then applies the
required filter conditions? If so the increase in execution time could be
attributed to the extra time required to read the ever increasing data in
Cassandra.
We are also working on a suitable archival mechanisms for our data so that
the total number of rows that are stored are always maintained at an optimum
count. This should also help us to maintain almost constant PIG script
execution time every day.
Please advice.
Thanks,
Badri
Re: PIG Cassandra - Performance
Posted by Jeremy Hanna <je...@gmail.com>.
Oh np Badri.
Also fwiw the open-source brisk project - https://github.com/riptano/brisk/ - does a good job integrating cassandra with hadoop and today they released beta 2 of it, which includes pig support in there. Might be worth looking at too. It simplifies operations a lot from what I understand. The explanation of what it does is http://www.datastax.com/brisk
Anyway, so just something else to consider.
Also I started a project called pygmalion to help out with pig + cassandra specifically. You might find it useful and/or want to contribute code/examples to it :). Anyway, that's here: https://github.com/jeromatron/pygmalion/
Jeremy
On Jun 17, 2011, at 9:05 PM, Badrinarayanan S wrote:
> Hi Jeremy,
>
> Thanks. Till we get 1.0 we will also adopt separate CF for analysis
> purposes.
>
> Regards,
> badri
>
> -----Original Message-----
> From: Jeremy Hanna [mailto:jeremy.hanna1234@gmail.com]
> Sent: Saturday, June 18, 2011 12:39 AM
> To: user@pig.apache.org
> Subject: Re: PIG Cassandra - Performance
>
> The way cassandra currently does mapreduce is that it iterates over all the
> rows of the column family. So yes, performance would be related to the
> growing number of rows. You can use the pig FILTER function to filter them
> down, but you are still iterating over all of the rows in that columns
> family.
>
> There is a ticket - CASSANDRA-1600
> (https://issues.apache.org/jira/browse/CASSANDRA-1600) that addresses this
> and allows for subsets of rows to be specified. It will also enable
> mapreducing over secondary indexes in a column family. We had hoped 1600
> would be resolved by now but there was a complication with a dependent
> issue. I have been told that it will definitely be in the next major
> release of Cassandra - 1.0, due out in the beginning of October. From what
> I understand, these updates will then enable both pig and hive to more
> easily push down selects of subsets of data.
>
> Until then, what we've done is set up a separate column family with data
> that we want to analyze that only has a subset of the data. Then when 1.0
> comes out, we'll shift over to use that.
>
> Jeremy
>
> On Jun 17, 2011, at 1:29 PM, Badrinarayanan S wrote:
>
>> Hi,
>>
>>
>>
>> In our production Cassandra systems we are observing the time taken by
> same
>> PIG script keeps increasing each and every day. The PIG scripts reads data
>> for a day at a time from a Cassandra Column Family. The number of rows the
>> PIG script is expected to return is almost same every day, however every
> day
>> the amount of rows we are storing in Cassandra is increasing. We haven't
>> changed the default setting for multiquery, it is by default enabled.
>>
>>
>>
>> Could this increase in PIG script execution time be related to the
>> increasing number of rows in Cassandra every day?
>>
>>
>>
>> Related to this I was trying to understand the behavior of LOAD statement.
>> Does LOAD statement reads all the data from Cassandra and then applies the
>> required filter conditions? If so the increase in execution time could be
>> attributed to the extra time required to read the ever increasing data in
>> Cassandra.
>>
>>
>>
>> We are also working on a suitable archival mechanisms for our data so that
>> the total number of rows that are stored are always maintained at an
> optimum
>> count. This should also help us to maintain almost constant PIG script
>> execution time every day.
>>
>>
>>
>> Please advice.
>>
>>
>>
>> Thanks,
>>
>> Badri
>>
>>
>>
>>
>>
>>
>>
>
RE: PIG Cassandra - Performance
Posted by Badrinarayanan S <ba...@fifthgentech.com>.
Hi Jeremy,
Thanks. Till we get 1.0 we will also adopt separate CF for analysis
purposes.
Regards,
badri
-----Original Message-----
From: Jeremy Hanna [mailto:jeremy.hanna1234@gmail.com]
Sent: Saturday, June 18, 2011 12:39 AM
To: user@pig.apache.org
Subject: Re: PIG Cassandra - Performance
The way cassandra currently does mapreduce is that it iterates over all the
rows of the column family. So yes, performance would be related to the
growing number of rows. You can use the pig FILTER function to filter them
down, but you are still iterating over all of the rows in that columns
family.
There is a ticket - CASSANDRA-1600
(https://issues.apache.org/jira/browse/CASSANDRA-1600) that addresses this
and allows for subsets of rows to be specified. It will also enable
mapreducing over secondary indexes in a column family. We had hoped 1600
would be resolved by now but there was a complication with a dependent
issue. I have been told that it will definitely be in the next major
release of Cassandra - 1.0, due out in the beginning of October. From what
I understand, these updates will then enable both pig and hive to more
easily push down selects of subsets of data.
Until then, what we've done is set up a separate column family with data
that we want to analyze that only has a subset of the data. Then when 1.0
comes out, we'll shift over to use that.
Jeremy
On Jun 17, 2011, at 1:29 PM, Badrinarayanan S wrote:
> Hi,
>
>
>
> In our production Cassandra systems we are observing the time taken by
same
> PIG script keeps increasing each and every day. The PIG scripts reads data
> for a day at a time from a Cassandra Column Family. The number of rows the
> PIG script is expected to return is almost same every day, however every
day
> the amount of rows we are storing in Cassandra is increasing. We haven't
> changed the default setting for multiquery, it is by default enabled.
>
>
>
> Could this increase in PIG script execution time be related to the
> increasing number of rows in Cassandra every day?
>
>
>
> Related to this I was trying to understand the behavior of LOAD statement.
> Does LOAD statement reads all the data from Cassandra and then applies the
> required filter conditions? If so the increase in execution time could be
> attributed to the extra time required to read the ever increasing data in
> Cassandra.
>
>
>
> We are also working on a suitable archival mechanisms for our data so that
> the total number of rows that are stored are always maintained at an
optimum
> count. This should also help us to maintain almost constant PIG script
> execution time every day.
>
>
>
> Please advice.
>
>
>
> Thanks,
>
> Badri
>
>
>
>
>
>
>
Re: PIG Cassandra - Performance
Posted by Jeremy Hanna <je...@gmail.com>.
The way cassandra currently does mapreduce is that it iterates over all the rows of the column family. So yes, performance would be related to the growing number of rows. You can use the pig FILTER function to filter them down, but you are still iterating over all of the rows in that columns family.
There is a ticket - CASSANDRA-1600 (https://issues.apache.org/jira/browse/CASSANDRA-1600) that addresses this and allows for subsets of rows to be specified. It will also enable mapreducing over secondary indexes in a column family. We had hoped 1600 would be resolved by now but there was a complication with a dependent issue. I have been told that it will definitely be in the next major release of Cassandra - 1.0, due out in the beginning of October. From what I understand, these updates will then enable both pig and hive to more easily push down selects of subsets of data.
Until then, what we've done is set up a separate column family with data that we want to analyze that only has a subset of the data. Then when 1.0 comes out, we'll shift over to use that.
Jeremy
On Jun 17, 2011, at 1:29 PM, Badrinarayanan S wrote:
> Hi,
>
>
>
> In our production Cassandra systems we are observing the time taken by same
> PIG script keeps increasing each and every day. The PIG scripts reads data
> for a day at a time from a Cassandra Column Family. The number of rows the
> PIG script is expected to return is almost same every day, however every day
> the amount of rows we are storing in Cassandra is increasing. We haven't
> changed the default setting for multiquery, it is by default enabled.
>
>
>
> Could this increase in PIG script execution time be related to the
> increasing number of rows in Cassandra every day?
>
>
>
> Related to this I was trying to understand the behavior of LOAD statement.
> Does LOAD statement reads all the data from Cassandra and then applies the
> required filter conditions? If so the increase in execution time could be
> attributed to the extra time required to read the ever increasing data in
> Cassandra.
>
>
>
> We are also working on a suitable archival mechanisms for our data so that
> the total number of rows that are stored are always maintained at an optimum
> count. This should also help us to maintain almost constant PIG script
> execution time every day.
>
>
>
> Please advice.
>
>
>
> Thanks,
>
> Badri
>
>
>
>
>
>
>