You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Debasish Das <de...@gmail.com> on 2014/05/08 07:09:52 UTC
Schema view of HadoopRDD
Hi,
For each line that we read as textLine from HDFS, we have a schema..if
there is an API that takes the schema as List[Symbol] and maps each token
to the Symbol it will be helpful...
Does RDDs provide a schema view of the dataset on HDFS ?
Thanks.
Deb
Re: Schema view of HadoopRDD
Posted by Flavio Pompermaier <po...@okkam.it>.
Is there any Spark plugin/add-on that facilitate the query to a JSON
content?
Best,
Flavio
On Thu, May 15, 2014 at 6:53 PM, Michael Armbrust <mi...@databricks.com>wrote:
> Here is a link with more info:
> http://people.apache.org/~pwendell/catalyst-docs/sql-programming-guide.html
>
>
> On Wed, May 7, 2014 at 10:09 PM, Debasish Das <de...@gmail.com>wrote:
>
>> Hi,
>>
>> For each line that we read as textLine from HDFS, we have a schema..if
>> there is an API that takes the schema as List[Symbol] and maps each token
>> to the Symbol it will be helpful...
>>
>> Does RDDs provide a schema view of the dataset on HDFS ?
>>
>> Thanks.
>> Deb
>>
>
Re: Schema view of HadoopRDD
Posted by Michael Armbrust <mi...@databricks.com>.
Here is a link with more info:
http://people.apache.org/~pwendell/catalyst-docs/sql-programming-guide.html
On Wed, May 7, 2014 at 10:09 PM, Debasish Das <de...@gmail.com>wrote:
> Hi,
>
> For each line that we read as textLine from HDFS, we have a schema..if
> there is an API that takes the schema as List[Symbol] and maps each token
> to the Symbol it will be helpful...
>
> Does RDDs provide a schema view of the dataset on HDFS ?
>
> Thanks.
> Deb
>
Re: Schema view of HadoopRDD
Posted by Mayur Rustagi <ma...@gmail.com>.
so you can use a input output format & read it whichever way you write...
You can additionally provide variables in hadoop configuration to
configure.
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Thu, May 8, 2014 at 10:39 AM, Debasish Das <de...@gmail.com>wrote:
> Hi,
>
> For each line that we read as textLine from HDFS, we have a schema..if
> there is an API that takes the schema as List[Symbol] and maps each token
> to the Symbol it will be helpful...
>
> Does RDDs provide a schema view of the dataset on HDFS ?
>
> Thanks.
> Deb
>
Re: Schema view of HadoopRDD
Posted by rxin <rx...@databricks.com>.
The new Spark SQL component is defined for this!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Re-Schema-view-of-HadoopRDD-tp5627p5723.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Schema view of HadoopRDD
Posted by Mayur Rustagi <ma...@gmail.com>.
I guess what you are trying to do is get a columnar projection on your
data, sparksql maybe a good option for you (especially if your data is
sparse & good for columnar projection).
If you are looking to work with simple key value then you are better off
using Hbase input reader in hadoopIO & get a pairRDD.
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Thu, May 8, 2014 at 10:51 AM, Debasish Das <de...@gmail.com>wrote:
> Hi,
>
> For each line that we read as textLine from HDFS, we have a schema..if
> there is an API that takes the schema as List[Symbol] and maps each token
> to the Symbol it will be helpful...
>
> One solution is to keep data on hdfs as avro/protobuf serialized objects
> but not sure if that works on HBase input...we are testing HDFS right now
> but finally we will read from a persistent store like hbase...so basically
> the immutableBytes need to be converted to a schema view as well incase we
> don't want to write the whole row as a protobuf...
>
> Does RDDs provide a schema view of the dataset on HDFS / HBase ?
>
> Thanks.
> Deb
>
>
Re: Schema view of HadoopRDD
Posted by Debasish Das <de...@gmail.com>.
Hi,
For each line that we read as textLine from HDFS, we have a schema..if
there is an API that takes the schema as List[Symbol] and maps each token
to the Symbol it will be helpful...
One solution is to keep data on hdfs as avro/protobuf serialized objects
but not sure if that works on HBase input...we are testing HDFS right now
but finally we will read from a persistent store like hbase...so basically
the immutableBytes need to be converted to a schema view as well incase we
don't want to write the whole row as a protobuf...
Does RDDs provide a schema view of the dataset on HDFS / HBase ?
Thanks.
Deb