You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Martin Gammelsæter <ma...@gmail.com> on 2014/07/02 13:36:09 UTC

How to use groupByKey and CqlPagingInputFormat

Hi!

Total Scala and Spark noob here with a few questions.

I am trying to modify a few of the examples in the spark repo to fit
my needs, but running into a few problems.

I am making an RDD from Cassandra, which I've finally gotten to work,
and trying to do some operations on it. Specifically I am trying to do
a grouping by key for future calculations.
I want the key to be the column "navn" from a certain column family,
but I don't think I understand the returned types. Why are two Maps
returned, instead of one? I'd think that you'd get a list of some kind
with every row, where every element in the list was a map from column
name to the value. So my first question is: What do these maps
represent?

   val casRdd = sc.newAPIHadoopRDD(job.getConfiguration(),
      classOf[CqlPagingInputFormat],
      classOf[java.util.Map[String,ByteBuffer]],
      classOf[java.util.Map[String,ByteBuffer]])

    val navnrevmap = casRdd.map({
      case (key, value) =>
        (ByteBufferUtil.string(value.get("navn")),
ByteBufferUtil.toInt(value.get("revisjon"))
    }).groupByKey()

The second question (probably stemming from my not understanding the
first question) is why am I not allowed to do a groupByKey in the
above code? I understand that the type does not have that function,
but I'm unclear on what I have to do to make it work.

-- 
Best regards,
Martin Gammelsæter

Re: How to use groupByKey and CqlPagingInputFormat

Posted by Martin Gammelsæter <ma...@gmail.com>.
Ah, I see. Thank you!

As we are in the process of building the system we have not tried with
any large amounts of data yet, but when the time comes I'll try both
implementations and do a small benchmark.

On Fri, Jul 4, 2014 at 9:20 PM, Mohammed Guller <mo...@glassbeam.com> wrote:
> As far as I know, there is not much difference, except that the outer parenthesis is redundant. The problem with your original code was that there was mismatch in the opening and closing parenthesis. Sometimes the error messages are misleading :-)
>
> Do you see any performance difference with the Datastax spark driver?
>
> Mohammed
>
> -----Original Message-----
> From: Martin Gammelsæter [mailto:martingammelsaeter@gmail.com]
> Sent: Friday, July 4, 2014 12:43 AM
> To: user@spark.apache.org
> Subject: Re: How to use groupByKey and CqlPagingInputFormat
>
> On Thu, Jul 3, 2014 at 10:29 PM, Mohammed Guller <mo...@glassbeam.com> wrote:
>> Martin,
>>
>> 1) The first map contains the columns in the primary key, which could be a compound primary key containing multiple columns,  and the second map contains all the non-key columns.
>
> Ah, thank you, that makes sense.
>
>> 2) try this fixed code:
>>     val navnrevmap = casRdd.map{
>>       case (key, value) =>
>>         (ByteBufferUtil.string(value.get("navn")),
>>            ByteBufferUtil.toInt(value.get("revisjon")))
>>        }.groupByKey()
>
> I changed from CqlPagingInputFormat to the new Datastax cassandra-spark driver, which is a bit easier to work with, but thanks! I'm curious though, what is the semantic difference between
> map({}) and map{}?



-- 
Mvh.
Martin Gammelsæter
92209139

RE: How to use groupByKey and CqlPagingInputFormat

Posted by Mohammed Guller <mo...@glassbeam.com>.
As far as I know, there is not much difference, except that the outer parenthesis is redundant. The problem with your original code was that there was mismatch in the opening and closing parenthesis. Sometimes the error messages are misleading :-)

Do you see any performance difference with the Datastax spark driver?

Mohammed

-----Original Message-----
From: Martin Gammelsæter [mailto:martingammelsaeter@gmail.com] 
Sent: Friday, July 4, 2014 12:43 AM
To: user@spark.apache.org
Subject: Re: How to use groupByKey and CqlPagingInputFormat

On Thu, Jul 3, 2014 at 10:29 PM, Mohammed Guller <mo...@glassbeam.com> wrote:
> Martin,
>
> 1) The first map contains the columns in the primary key, which could be a compound primary key containing multiple columns,  and the second map contains all the non-key columns.

Ah, thank you, that makes sense.

> 2) try this fixed code:
>     val navnrevmap = casRdd.map{
>       case (key, value) =>
>         (ByteBufferUtil.string(value.get("navn")),
>            ByteBufferUtil.toInt(value.get("revisjon")))
>        }.groupByKey()

I changed from CqlPagingInputFormat to the new Datastax cassandra-spark driver, which is a bit easier to work with, but thanks! I'm curious though, what is the semantic difference between
map({}) and map{}?

Re: How to use groupByKey and CqlPagingInputFormat

Posted by Martin Gammelsæter <ma...@gmail.com>.
On Thu, Jul 3, 2014 at 10:29 PM, Mohammed Guller <mo...@glassbeam.com> wrote:
> Martin,
>
> 1) The first map contains the columns in the primary key, which could be a compound primary key containing multiple columns,  and the second map contains all the non-key columns.

Ah, thank you, that makes sense.

> 2) try this fixed code:
>     val navnrevmap = casRdd.map{
>       case (key, value) =>
>         (ByteBufferUtil.string(value.get("navn")),
>            ByteBufferUtil.toInt(value.get("revisjon")))
>        }.groupByKey()

I changed from CqlPagingInputFormat to the new Datastax
cassandra-spark driver, which is a bit easier to work with, but
thanks! I'm curious though, what is the semantic difference between
map({}) and map{}?

RE: How to use groupByKey and CqlPagingInputFormat

Posted by Mohammed Guller <mo...@glassbeam.com>.
Martin,

1) The first map contains the columns in the primary key, which could be a compound primary key containing multiple columns,  and the second map contains all the non-key columns.
2) try this fixed code:
    val navnrevmap = casRdd.map{
      case (key, value) =>
        (ByteBufferUtil.string(value.get("navn")),
           ByteBufferUtil.toInt(value.get("revisjon")))
       }.groupByKey() 


Mohammed


-----Original Message-----
From: Martin Gammelsæter [mailto:martingammelsaeter@gmail.com] 
Sent: Wednesday, July 2, 2014 4:36 AM
To: user@spark.apache.org
Subject: How to use groupByKey and CqlPagingInputFormat

Hi!

Total Scala and Spark noob here with a few questions.

I am trying to modify a few of the examples in the spark repo to fit my needs, but running into a few problems.

I am making an RDD from Cassandra, which I've finally gotten to work, and trying to do some operations on it. Specifically I am trying to do a grouping by key for future calculations.
I want the key to be the column "navn" from a certain column family, but I don't think I understand the returned types. Why are two Maps returned, instead of one? I'd think that you'd get a list of some kind with every row, where every element in the list was a map from column name to the value. So my first question is: What do these maps represent?

   val casRdd = sc.newAPIHadoopRDD(job.getConfiguration(),
      classOf[CqlPagingInputFormat],
      classOf[java.util.Map[String,ByteBuffer]],
      classOf[java.util.Map[String,ByteBuffer]])

    val navnrevmap = casRdd.map({
      case (key, value) =>
        (ByteBufferUtil.string(value.get("navn")),
ByteBufferUtil.toInt(value.get("revisjon"))
    }).groupByKey()

The second question (probably stemming from my not understanding the first question) is why am I not allowed to do a groupByKey in the above code? I understand that the type does not have that function, but I'm unclear on what I have to do to make it work.

--
Best regards,
Martin Gammelsæter