You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2013/05/02 19:50:15 UTC
[jira] [Commented] (PIG-3308) Storing data in hive columnar rc
format
[ https://issues.apache.org/jira/browse/PIG-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647749#comment-13647749 ]
Daniel Dai commented on PIG-3308:
---------------------------------
Hit a compilation error:
[javac] /Users/daijy/pig/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/TestHiveColumnarStorage.java:35: package org.apache.hadoop.mapreduce.task does not exist
[javac] import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
Is it for hadoop 2 only?
> Storing data in hive columnar rc format
> ---------------------------------------
>
> Key: PIG-3308
> URL: https://issues.apache.org/jira/browse/PIG-3308
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Affects Versions: 0.10.1
> Reporter: Marcin Czech
> Labels: patch
> Fix For: 0.10.1
>
> Attachments: PIG-3308.patch
>
>
> I've coded HiveColumnarStorage that can store Pig structures as a Hive Columnar RC tables. Code is based on Elephant-bird RCFilePigStorage. The difference is that data are stored in Hive friendly format, so file can be read from Hive.
> Example Pig schema:
> {code}
> f1:tuple (f11: chararray,f12: chararray),f2:map[]
> {code}
> Hive schema:
> {code}
> CREATE TABLE sample_table (f1 struct<f11:string,f12:string>, f2 array<struct<f21:string,f22:string>>)
> PARTITIONED BY (p string)
> STORED AS RCFILE
> {code}
> or as a:
> {code}
> CREATE TABLE sample_table (f1 struct<f11:string,f12:string>, f2 MAP <string,string>)
> PARTITIONED BY (p string)
> STORED AS RCFILE
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira