You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Lizhao.Du (JIRA)" <ji...@apache.org> on 2014/04/18 12:44:15 UTC
[jira] [Updated] (PIG-3904) Pig support windows i18n
[ https://issues.apache.org/jira/browse/PIG-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lizhao.Du updated PIG-3904:
---------------------------
Description:
Utilize Pig run a pig script in Windows (de_DE), it is failed.
The error message showed "Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput", but /tmp/测试/pwInput is exist
in fact. Because of encoding of hadoop adoption is UTF-8. When encoding of client OS pig situated is different with it, hadoop will unrecognize there characters.
Log message as below:
==============================================
ERROR Spring Shell org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Backend error message during job submission
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:270)
... 14 more
I have added a patch to fix it. It works.
was:
Utilize Pig run a pig script in Windows (de_DE), it is failed.
The error message showed "Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput", but /tmp/测试/pwInput is exist
in fact. Because of encoding of hadoop adoption is UTF-8. When encoding of client OS pig situated is different with it, hadoop will unrecognize there characters.
Log message as below:
==============================================
ERROR Spring Shell org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Backend error message during job submission
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:270)
... 14 more
> Pig support windows i18n
> ------------------------
>
> Key: PIG-3904
> URL: https://issues.apache.org/jira/browse/PIG-3904
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.9.2, 0.9.3
> Environment: Windows 7(de_DE/fr_FR/zh_CN)
> Reporter: Lizhao.Du
> Fix For: 0.9.3
>
>
> Utilize Pig run a pig script in Windows (de_DE), it is failed.
> The error message showed "Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput", but /tmp/测试/pwInput is exist
> in fact. Because of encoding of hadoop adoption is UTF-8. When encoding of client OS pig situated is different with it, hadoop will unrecognize there characters.
> Log message as below:
> ==============================================
> ERROR Spring Shell org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Backend error message during job submission
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
> at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.141.73.10:8020/tmp/测试/pwInput
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:270)
> ... 14 more
> I have added a patch to fix it. It works.
--
This message was sent by Atlassian JIRA
(v6.2#6252)