You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Debasish Das <de...@gmail.com> on 2014/05/08 07:09:52 UTC

Schema view of HadoopRDD

Hi,

For each line that we read as textLine from HDFS, we have a schema..if
there is an API that takes the schema as List[Symbol] and maps each token
to the Symbol it will be helpful...

Does RDDs provide a schema view of the dataset on HDFS ?

Thanks.
Deb

Re: Schema view of HadoopRDD

Posted by Flavio Pompermaier <po...@okkam.it>.
Is there any Spark plugin/add-on that facilitate the query to a JSON
content?

Best,
Flavio

On Thu, May 15, 2014 at 6:53 PM, Michael Armbrust <mi...@databricks.com>wrote:

> Here is a link with more info:
> http://people.apache.org/~pwendell/catalyst-docs/sql-programming-guide.html
>
>
> On Wed, May 7, 2014 at 10:09 PM, Debasish Das <de...@gmail.com>wrote:
>
>> Hi,
>>
>> For each line that we read as textLine from HDFS, we have a schema..if
>> there is an API that takes the schema as List[Symbol] and maps each token
>> to the Symbol it will be helpful...
>>
>> Does RDDs provide a schema view of the dataset on HDFS ?
>>
>> Thanks.
>> Deb
>>
>

Re: Schema view of HadoopRDD

Posted by Michael Armbrust <mi...@databricks.com>.
Here is a link with more info:
http://people.apache.org/~pwendell/catalyst-docs/sql-programming-guide.html


On Wed, May 7, 2014 at 10:09 PM, Debasish Das <de...@gmail.com>wrote:

> Hi,
>
> For each line that we read as textLine from HDFS, we have a schema..if
> there is an API that takes the schema as List[Symbol] and maps each token
> to the Symbol it will be helpful...
>
> Does RDDs provide a schema view of the dataset on HDFS ?
>
> Thanks.
> Deb
>

Re: Schema view of HadoopRDD

Posted by Mayur Rustagi <ma...@gmail.com>.
so you can use a input output format & read it whichever way you write...
You can additionally provide variables in hadoop configuration to
configure.



Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Thu, May 8, 2014 at 10:39 AM, Debasish Das <de...@gmail.com>wrote:

> Hi,
>
> For each line that we read as textLine from HDFS, we have a schema..if
> there is an API that takes the schema as List[Symbol] and maps each token
> to the Symbol it will be helpful...
>
> Does RDDs provide a schema view of the dataset on HDFS ?
>
> Thanks.
> Deb
>

Re: Schema view of HadoopRDD

Posted by rxin <rx...@databricks.com>.
The new Spark SQL component is defined for this!




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Re-Schema-view-of-HadoopRDD-tp5627p5723.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Schema view of HadoopRDD

Posted by Mayur Rustagi <ma...@gmail.com>.
I guess what you are trying to do is get a columnar projection on your
data, sparksql maybe a good option for you (especially if your data is
sparse & good for columnar projection).
If you are looking to work with simple key value then you are better off
using Hbase input reader in hadoopIO  & get a pairRDD.

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Thu, May 8, 2014 at 10:51 AM, Debasish Das <de...@gmail.com>wrote:

> Hi,
>
> For each line that we read as textLine from HDFS, we have a schema..if
> there is an API that takes the schema as List[Symbol] and maps each token
> to the Symbol it will be helpful...
>
> One solution is to keep data on hdfs as avro/protobuf serialized objects
> but not sure if that works on HBase input...we are testing HDFS right now
> but finally we will read from a persistent store like hbase...so basically
> the immutableBytes need to be converted to a schema view as well incase we
> don't want to write the whole row as a protobuf...
>
> Does RDDs provide a schema view of the dataset on HDFS / HBase ?
>
> Thanks.
> Deb
>
>

Re: Schema view of HadoopRDD

Posted by Debasish Das <de...@gmail.com>.
Hi,

For each line that we read as textLine from HDFS, we have a schema..if
there is an API that takes the schema as List[Symbol] and maps each token
to the Symbol it will be helpful...

One solution is to keep data on hdfs as avro/protobuf serialized objects
but not sure if that works on HBase input...we are testing HDFS right now
but finally we will read from a persistent store like hbase...so basically
the immutableBytes need to be converted to a schema view as well incase we
don't want to write the whole row as a protobuf...

Does RDDs provide a schema view of the dataset on HDFS / HBase ?

Thanks.
Deb