You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by asethia <se...@gmail.com> on 2016/03/31 17:19:53 UTC
transformation - spark vs cassandra
Hi,
I am working with Cassandra and Spark, would like to know what is best
performance using Cassandra filter based on primary key and cluster key vs
using spark data frame transformation/filters.
for example in spark:
val rdd = sqlContext.read.format("org.apache.spark.sql.cassandra")
.options(Map("keyspace" -> "test", "table" -> "test"))
.load()
and then rdd.filter("cdate ='2016-06-07'").filter("country='USA'").count()
vs
using Cassandra (where cdate is part of primary key and country as cluster
key).
SELECT count(*) FROM test WHERE cdate ='2016-06-07' AND country='USA'
I would like to know when should we use Cassandra simple query vs dataframe
in terms of performance with billion of rows.
Thanks
arun
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/transformation-spark-vs-cassandra-tp26647.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: transformation - spark vs cassandra
Posted by Femi Anthony <fe...@gmail.com>.
Try it out on a smaller subset of data and see which gives the better
performance.
On Thu, Mar 31, 2016 at 12:11 PM, Arun Sethia <se...@gmail.com> wrote:
> Thanks Imre.
>
> But I thought spark-cassandra driver is going to do same internally.
>
> On Thu, Mar 31, 2016 at 10:32 AM, Imre Nagi <im...@gmail.com>
> wrote:
>
>> I think querying by cassandra query language will be better in terms of
>> performance if you want to pull and filter the data from your db, rather
>> than pulling all of the data and do some filtering and transformation by
>> using spark data frame.
>>
>>
>> On 31 Mar 2016 22:19, "asethia" <se...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am working with Cassandra and Spark, would like to know what is best
>>> performance using Cassandra filter based on primary key and cluster key
>>> vs
>>> using spark data frame transformation/filters.
>>>
>>> for example in spark:
>>>
>>> val rdd = sqlContext.read.format("org.apache.spark.sql.cassandra")
>>> .options(Map("keyspace" -> "test", "table" -> "test"))
>>> .load()
>>>
>>> and then rdd.filter("cdate
>>> ='2016-06-07'").filter("country='USA'").count()
>>>
>>> vs
>>>
>>> using Cassandra (where cdate is part of primary key and country as
>>> cluster
>>> key).
>>>
>>> SELECT count(*) FROM test WHERE cdate ='2016-06-07' AND country='USA'
>>>
>>> I would like to know when should we use Cassandra simple query vs
>>> dataframe
>>> in terms of performance with billion of rows.
>>>
>>> Thanks
>>> arun
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/transformation-spark-vs-cassandra-tp26647.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>
--
http://www.femibyte.com/twiki5/bin/view/Tech/
http://www.nextmatrix.com
"Great spirits have always encountered violent opposition from mediocre
minds." - Albert Einstein.
Re: transformation - spark vs cassandra
Posted by Arun Sethia <se...@gmail.com>.
Thanks Imre.
But I thought spark-cassandra driver is going to do same internally.
On Thu, Mar 31, 2016 at 10:32 AM, Imre Nagi <im...@gmail.com> wrote:
> I think querying by cassandra query language will be better in terms of
> performance if you want to pull and filter the data from your db, rather
> than pulling all of the data and do some filtering and transformation by
> using spark data frame.
>
>
> On 31 Mar 2016 22:19, "asethia" <se...@gmail.com> wrote:
>
>> Hi,
>>
>> I am working with Cassandra and Spark, would like to know what is best
>> performance using Cassandra filter based on primary key and cluster key vs
>> using spark data frame transformation/filters.
>>
>> for example in spark:
>>
>> val rdd = sqlContext.read.format("org.apache.spark.sql.cassandra")
>> .options(Map("keyspace" -> "test", "table" -> "test"))
>> .load()
>>
>> and then rdd.filter("cdate ='2016-06-07'").filter("country='USA'").count()
>>
>> vs
>>
>> using Cassandra (where cdate is part of primary key and country as cluster
>> key).
>>
>> SELECT count(*) FROM test WHERE cdate ='2016-06-07' AND country='USA'
>>
>> I would like to know when should we use Cassandra simple query vs
>> dataframe
>> in terms of performance with billion of rows.
>>
>> Thanks
>> arun
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/transformation-spark-vs-cassandra-tp26647.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: transformation - spark vs cassandra
Posted by Imre Nagi <im...@gmail.com>.
I think querying by cassandra query language will be better in terms of
performance if you want to pull and filter the data from your db, rather
than pulling all of the data and do some filtering and transformation by
using spark data frame.
On 31 Mar 2016 22:19, "asethia" <se...@gmail.com> wrote:
> Hi,
>
> I am working with Cassandra and Spark, would like to know what is best
> performance using Cassandra filter based on primary key and cluster key vs
> using spark data frame transformation/filters.
>
> for example in spark:
>
> val rdd = sqlContext.read.format("org.apache.spark.sql.cassandra")
> .options(Map("keyspace" -> "test", "table" -> "test"))
> .load()
>
> and then rdd.filter("cdate ='2016-06-07'").filter("country='USA'").count()
>
> vs
>
> using Cassandra (where cdate is part of primary key and country as cluster
> key).
>
> SELECT count(*) FROM test WHERE cdate ='2016-06-07' AND country='USA'
>
> I would like to know when should we use Cassandra simple query vs dataframe
> in terms of performance with billion of rows.
>
> Thanks
> arun
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/transformation-spark-vs-cassandra-tp26647.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>