You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2010/07/30 19:47:16 UTC
[jira] Commented: (PIG-1526) HiveColumnarLoader Partitioning
Support
[ https://issues.apache.org/jira/browse/PIG-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894087#action_12894087 ]
Hadoop QA commented on PIG-1526:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12450900/PIG-1526.patch
against trunk revision 980276.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/367/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/367/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/367/console
This message is automatically generated.
> HiveColumnarLoader Partitioning Support
> ---------------------------------------
>
> Key: PIG-1526
> URL: https://issues.apache.org/jira/browse/PIG-1526
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Gerrit Jansen van Vuuren
> Assignee: Gerrit Jansen van Vuuren
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1526.patch
>
>
> I've made allot improvements on the HiveColumnarLoader:
> -> Added support for LoadMetadata and data path Partitioning
> -> Improved and simplefied column loading
> Data Path Partitioning:
> Hive stores partitions as folders like to /mytable/partition1=[value]/partition2=[value]. That is the table mytable contains 2 partitions [partition1, partition2].
> The HiveColumnarLoader will scan the inputpath /mytable and add to the PigSchema the columns partition2 and partition2.
> These columns can then be used in filtering.
> For example: We've got year,month,day,hour partitions in our data uploads.
> So a table might look like mytable/year=2010/month=02/day=01.
> Loading with the HiveColumnarLoader allows our pig scripts do filter by date using the standard pig Filter operator.
> I've added 2 classes for this:
> -> PathPartitioner
> -> PathPartitionHelper
> These classes are not hive dependent and could be used by any other loader that wants to support partitioning and helps with implementing the LoadMetadata interface.
> For this reason I though it best to put it into the package org.apache.pig.piggybank.storage.partition.
> What would be nice is in the future have the PigStorage also use these 2 classes to provide automatic path partitioning support.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.