You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2011/05/04 06:11:03 UTC
[jira] [Created] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Pig client uses fs.default.name as provided from the JobTracker instead of local value
--------------------------------------------------------------------------------------
Key: PIG-2034
URL: https://issues.apache.org/jira/browse/PIG-2034
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029049#comment-13029049 ]
Daniel Dai commented on PIG-2034:
---------------------------------
It's really wired that we will use the master setting to submit a job. Is that done by mapreduce as part of job submission protocol? Did you see that in mapreduce job?
> Pig client uses fs.default.name as provided from the JobTracker instead of local value
> --------------------------------------------------------------------------------------
>
> Key: PIG-2034
> URL: https://issues.apache.org/jira/browse/PIG-2034
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Attachments: pig_1304465896181.log
>
>
> When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
> I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029059#comment-13029059 ]
Bill Graham commented on PIG-2034:
----------------------------------
I was wondering the same thing after I filed this. I can't tell if this is from the MR job submission doing this on the client or Pig. Can you tell from the stack trace I provided?
> Pig client uses fs.default.name as provided from the JobTracker instead of local value
> --------------------------------------------------------------------------------------
>
> Key: PIG-2034
> URL: https://issues.apache.org/jira/browse/PIG-2034
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Attachments: pig_1304465896181.log
>
>
> When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
> I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029082#comment-13029082 ]
Bill Graham commented on PIG-2034:
----------------------------------
Yup. I just tried the same with WordCount and got the same failure, so this doesn't seem to be a Pig bug. Do you think this is a valid MapReduce bug?
{code}
Exception in thread "main" java.net.UnknownHostException: unknown host: colo1-hadoop-nn-r0-n0
at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:850)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1419)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1444)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1432)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
at org.apache.hadoop.mapred.JobClient.getFs(JobClient.java:504)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:608)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:802)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1290)
at cnwk.hadoop.mapreduce.WordCount.main(WordCount.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
{code}
> Pig client uses fs.default.name as provided from the JobTracker instead of local value
> --------------------------------------------------------------------------------------
>
> Key: PIG-2034
> URL: https://issues.apache.org/jira/browse/PIG-2034
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Attachments: pig_1304465896181.log
>
>
> When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
> I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029092#comment-13029092 ]
Daniel Dai commented on PIG-2034:
---------------------------------
I would say yes. Client side use server side configuration, that seems counter intuitive.
> Pig client uses fs.default.name as provided from the JobTracker instead of local value
> --------------------------------------------------------------------------------------
>
> Key: PIG-2034
> URL: https://issues.apache.org/jira/browse/PIG-2034
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Attachments: pig_1304465896181.log
>
>
> When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
> I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029073#comment-13029073 ]
Daniel Dai commented on PIG-2034:
---------------------------------
The stack is all from mapreduce. I wonder if you will see the same thing in a pure mapreduce job.
> Pig client uses fs.default.name as provided from the JobTracker instead of local value
> --------------------------------------------------------------------------------------
>
> Key: PIG-2034
> URL: https://issues.apache.org/jira/browse/PIG-2034
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Attachments: pig_1304465896181.log
>
>
> When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
> I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2034) Pig client uses fs.default.name as
provided from the JobTracker instead of local value
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham updated PIG-2034:
-----------------------------
Attachment: pig_1304465896181.log
Attached is an error log that shows what the exception looks like when the servers {{fs.default.name}} can't be resolved by the client when running from the pig shell.
The script that produced this was this:
{code}
raw = LOAD '/user/bob/some_file.txt' AS (a, b, c, d);
A = LIMIT raw 10;
dump A;
{code}
> Pig client uses fs.default.name as provided from the JobTracker instead of local value
> --------------------------------------------------------------------------------------
>
> Key: PIG-2034
> URL: https://issues.apache.org/jira/browse/PIG-2034
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Attachments: pig_1304465896181.log
>
>
> When submitting a Pig job, the client uses the {{fs.default.name}} supplied to it by the JobTracker (via core-site.xml on the master typically) during the staging phase. After that, the client then uses the {{fs.default.name}} from it's local configs. This seems like a bug to me. Expected behavior would be to always use the local value.
> I found this bug when the server configs were set to not use a FQDN for {{fs.default.name}}. This caused the client to fail because it didn't have the same default DNS domain.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira