You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2013/04/03 23:17:15 UTC
[jira] [Updated] (PIG-3267) HCatStorer fail in limit query
[ https://issues.apache.org/jira/browse/PIG-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-3267:
----------------------------
Attachment: PIG-3267-1.patch
This is due to when we add a Limit job and convert the intermediate storer to temporary storer, we did not nullify the cached storer in POStore. End up we are still using HCatStorer in the intermediate job. Attach patch.
> HCatStorer fail in limit query
> ------------------------------
>
> Key: PIG-3267
> URL: https://issues.apache.org/jira/browse/PIG-3267
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.2, 0.10.1, 0.11.1
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.12
>
> Attachments: PIG-3267-1.patch
>
>
> The following query fail:
> {code}
> data = LOAD 'student.txt' as (name:chararray, age:int, gpa:double);
> data_limited = limit data 10;
> samples = foreach data_limited generate age as number;
> store samples into 'samples' using org.apache.hcatalog.pig.HCatStorer('part_dt=20130101T010000T36');
> {code}
> Error happens before launching the second job. Error message:
> {code}
> Message: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:8020/user/hive/warehouse/samples/part_dt=20130101T010000T36 already exists
> at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:121)
> at org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
> at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecsHelper(PigOutputFormat.java:207)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:188)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:887)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.pig.backend.hadoop20.PigJobControl.mainLoopAction(PigJobControl.java:157)
> at org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:134)
> at java.lang.Thread.run(Thread.java:680)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:257)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira