You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jerry Lam <ch...@gmail.com> on 2014/12/15 20:04:45 UTC
Accessing rows of a row in Spark
Hi spark users,
Do you know how to access rows of row?
I have a SchemaRDD called user and register it as a table with the
following schema:
root
|-- user_id: string (nullable = true)
|-- item: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- item_id: string (nullable = true)
| | |-- name: string (nullable = true)
val items=sqlContext.sql("select items from user where user_id = 1").first
The type of items is org.apache.spark.sql.Row. I want to iterate through
the items and count how many items that user_id = 1 has.
I could not find a method in which I can do that. The farthest I can get to
is to convert items.toSeq. The type information I got back is:
scala> items.toSeq
res57: Seq[Any] = [WrappedArray([1,orange],[2,apple])]
Any suggestion?
Best Regards,
Jerry
Re: Accessing rows of a row in Spark
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Looks like you've got one more layer of containment than you intend -- i.e.
you've got Row[WrappedArray[Row[(Int, String)]] where you want
Row[Row[(Int, String)]]. That's easy to do if somewhere along the line you
did something like `val row = Row(collection)` instead of `val row =
Row.fromSeq(collection)`.
On Mon, Dec 15, 2014 at 11:47 AM, Jerry Lam <ch...@gmail.com> wrote:
>
> Hi Mark,
>
> Thank you for helping out.
>
> The items I got back from Spark SQL has the type information as follows:
>
> scala> items
> res16: org.apache.spark.sql.Row = [WrappedArray([1,orange],[2,apple])]
>
> I tried to iterate the items as you suggested but no luck.
>
> Best Regards,
>
> Jerry
>
>
> On Mon, Dec 15, 2014 at 2:18 PM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>>
>> scala> val items = Row(1 -> "orange", 2 -> "apple")
>>
>> items: org.apache.spark.sql.catalyst.expressions.Row =
>> [(1,orange),(2,apple)]
>>
>>
>> If you literally want an iterator, then this:
>>
>>
>> scala> items.toIterator.count { case (user_id, name) => user_id == 1 }
>>
>> res0: Int = 1
>>
>>
>> ...else:
>>
>>
>> scala> items.count { case (user_id, name) => user_id == 1 }
>>
>> res1: Int = 1
>>
>> On Mon, Dec 15, 2014 at 11:04 AM, Jerry Lam <ch...@gmail.com> wrote:
>>>
>>> Hi spark users,
>>>
>>> Do you know how to access rows of row?
>>>
>>> I have a SchemaRDD called user and register it as a table with the
>>> following schema:
>>>
>>> root
>>> |-- user_id: string (nullable = true)
>>> |-- item: array (nullable = true)
>>> | |-- element: struct (containsNull = false)
>>> | | |-- item_id: string (nullable = true)
>>> | | |-- name: string (nullable = true)
>>>
>>>
>>> val items=sqlContext.sql("select items from user where user_id =
>>> 1").first
>>>
>>> The type of items is org.apache.spark.sql.Row. I want to iterate through
>>> the items and count how many items that user_id = 1 has.
>>>
>>> I could not find a method in which I can do that. The farthest I can get
>>> to is to convert items.toSeq. The type information I got back is:
>>>
>>> scala> items.toSeq
>>> res57: Seq[Any] = [WrappedArray([1,orange],[2,apple])]
>>>
>>> Any suggestion?
>>>
>>> Best Regards,
>>>
>>> Jerry
>>>
>>
Re: Accessing rows of a row in Spark
Posted by Jerry Lam <ch...@gmail.com>.
Hi Mark,
Thank you for helping out.
The items I got back from Spark SQL has the type information as follows:
scala> items
res16: org.apache.spark.sql.Row = [WrappedArray([1,orange],[2,apple])]
I tried to iterate the items as you suggested but no luck.
Best Regards,
Jerry
On Mon, Dec 15, 2014 at 2:18 PM, Mark Hamstra <ma...@clearstorydata.com>
wrote:
>
> scala> val items = Row(1 -> "orange", 2 -> "apple")
>
> items: org.apache.spark.sql.catalyst.expressions.Row =
> [(1,orange),(2,apple)]
>
>
> If you literally want an iterator, then this:
>
>
> scala> items.toIterator.count { case (user_id, name) => user_id == 1 }
>
> res0: Int = 1
>
>
> ...else:
>
>
> scala> items.count { case (user_id, name) => user_id == 1 }
>
> res1: Int = 1
>
> On Mon, Dec 15, 2014 at 11:04 AM, Jerry Lam <ch...@gmail.com> wrote:
>>
>> Hi spark users,
>>
>> Do you know how to access rows of row?
>>
>> I have a SchemaRDD called user and register it as a table with the
>> following schema:
>>
>> root
>> |-- user_id: string (nullable = true)
>> |-- item: array (nullable = true)
>> | |-- element: struct (containsNull = false)
>> | | |-- item_id: string (nullable = true)
>> | | |-- name: string (nullable = true)
>>
>>
>> val items=sqlContext.sql("select items from user where user_id = 1").first
>>
>> The type of items is org.apache.spark.sql.Row. I want to iterate through
>> the items and count how many items that user_id = 1 has.
>>
>> I could not find a method in which I can do that. The farthest I can get
>> to is to convert items.toSeq. The type information I got back is:
>>
>> scala> items.toSeq
>> res57: Seq[Any] = [WrappedArray([1,orange],[2,apple])]
>>
>> Any suggestion?
>>
>> Best Regards,
>>
>> Jerry
>>
>
Re: Accessing rows of a row in Spark
Posted by Mark Hamstra <ma...@clearstorydata.com>.
scala> val items = Row(1 -> "orange", 2 -> "apple")
items: org.apache.spark.sql.catalyst.expressions.Row =
[(1,orange),(2,apple)]
If you literally want an iterator, then this:
scala> items.toIterator.count { case (user_id, name) => user_id == 1 }
res0: Int = 1
...else:
scala> items.count { case (user_id, name) => user_id == 1 }
res1: Int = 1
On Mon, Dec 15, 2014 at 11:04 AM, Jerry Lam <ch...@gmail.com> wrote:
>
> Hi spark users,
>
> Do you know how to access rows of row?
>
> I have a SchemaRDD called user and register it as a table with the
> following schema:
>
> root
> |-- user_id: string (nullable = true)
> |-- item: array (nullable = true)
> | |-- element: struct (containsNull = false)
> | | |-- item_id: string (nullable = true)
> | | |-- name: string (nullable = true)
>
>
> val items=sqlContext.sql("select items from user where user_id = 1").first
>
> The type of items is org.apache.spark.sql.Row. I want to iterate through
> the items and count how many items that user_id = 1 has.
>
> I could not find a method in which I can do that. The farthest I can get
> to is to convert items.toSeq. The type information I got back is:
>
> scala> items.toSeq
> res57: Seq[Any] = [WrappedArray([1,orange],[2,apple])]
>
> Any suggestion?
>
> Best Regards,
>
> Jerry
>