You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by sh...@accenture.com on 2014/08/04 18:51:31 UTC
Issue with sqoop merge command
Hi,
I am getting a weird error this morning! When running sqoop merge command
Deleted s3://xxx-xx-xxx/data/xxxx/xxxx/xxxx/merged
merge folder exists so deleted 0
Next,
Merge command fails with the error
14/08/04 11:39:24 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3://xx-xx-xx/xx/xxxx/xxx/xxx/merged already exists
14/08/04 11:39:24 ERROR tool.MergeTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3://xx-xx-xx/xx/xxxx/xxxx/xxxx/merged already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
at org.apache.sqoop.mapreduce.JobBase.runJob(JobBase.java:312)
at org.apache.sqoop.mapreduce.MergeJob.runMergeJob(MergeJob.java:138)
at org.apache.sqoop.tool.MergeTool.run(MergeTool.java:57)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
11:39:24: sqoopMergeStatus 1
11:39:24: 0
11:39:24: sqoop Merge Failed for XXXX
Any idea, how to fix it?
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com
Re: Issue with sqoop merge command
Posted by Gwen Shapira <gs...@cloudera.com>.
It says that the output directory exists, can you verify if thats the case?
Sqoop merge takes 3 directory parameters:
--new-data - directory with the new data set to merge (in case of
duplicates we keep the values from this directory)
--onto - directory with existing data set to merge (some of the values
here will be replaced by new values)
--target-dir - this directory should not exist before running Sqoop
merge. The results will go here.
Gwen
On Mon, Aug 4, 2014 at 9:51 AM, <sh...@accenture.com> wrote:
> Hi,
>
>
>
> I am getting a weird error this morning! When running sqoop merge command
>
>
>
> Deleted s3://xxx-xx-xxx/data/xxxx/xxxx/xxxx/merged
>
> merge folder exists so deleted 0
>
>
>
> Next,
>
>
>
> Merge command fails with the error
>
>
>
>
>
> 14/08/04 11:39:24 ERROR security.UserGroupInformation:
> PriviledgedActionException as:hadoop (auth:SIMPLE)
> cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
> s3://xx-xx-xx/xx/xxxx/xxx/xxx/merged already exists
>
> 14/08/04 11:39:24 ERROR tool.MergeTool: Encountered IOException running
> import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output
> directory s3://xx-xx-xx/xx/xxxx/xxxx/xxxx/merged already exists
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
>
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>
> at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
>
> at
> org.apache.sqoop.mapreduce.JobBase.runJob(JobBase.java:312)
>
> at
> org.apache.sqoop.mapreduce.MergeJob.runMergeJob(MergeJob.java:138)
>
> at org.apache.sqoop.tool.MergeTool.run(MergeTool.java:57)
>
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>
>
>
> 11:39:24: sqoopMergeStatus 1
>
> 11:39:24: 0
>
> 11:39:24: sqoop Merge Failed for XXXX
>
>
>
> Any idea, how to fix it?
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed by
> local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
> ______________________________________________________________________________________
>
> www.accenture.com