You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Niels Basjes <Ni...@basjes.nl> on 2015/02/26 17:24:01 UTC
Pig -x tez fails over tez.session.local-resources.pb.file-name ??
Hi,
I have installed tez on an existing hadoop cluster (HDP 2.1.2 with
kerberos installed).
When I run a very basic pig script I get the error you see below.
It effectively fails with
java.io.FileNotFoundException:
tez.session.local-resources.pb.file-name (No such file or directory)
I have done some googleing on that term and I was unable to figure out
what went wrong.
Do you guys have any suggestions what I have done wrong?
Thanks.
Niels Basjes
2015-02-26 17:15:53,310 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is
deprecated. Instead, use fs.defaultFS
2015-02-26 17:15:53,364 INFO [main]
org.apache.tez.dag.app.DAGAppMaster: Created DAGAppMaster for
application appattempt_1423554480020_30727_000002
2015-02-26 17:15:53,534 WARN [main]
org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
applicable
2015-02-26 17:15:53,661 INFO [main]
org.apache.tez.dag.app.DAGAppMaster: Adding session token to
jobTokenSecretManager for application
2015-02-26 17:15:53,667 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.container.AMContainerEventType for class
org.apache.tez.dag.app.rm.container.AMContainerMap
2015-02-26 17:15:53,668 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.node.AMNodeEventType for class
org.apache.tez.dag.app.rm.node.AMNodeMap
2015-02-26 17:15:53,669 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class
org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2015-02-26 17:15:53,670 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.DAGEventType for class
org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2015-02-26 17:15:53,670 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.VertexEventType for class
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2015-02-26 17:15:53,671 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.TaskEventType for class
org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2015-02-26 17:15:53,672 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class
org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2015-02-26 17:15:53,673 INFO [main]
org.apache.tez.dag.app.DAGAppMaster: AM will unregister as this is the
last attempt, currentAttempt=2, maxAttempts=2
2015-02-26 17:15:53,674 INFO [main]
org.apache.tez.dag.app.rm.TaskSchedulerEventHandler: TaskScheduler
notified that it should unregister from RM
2015-02-26 17:15:53,674 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.AMSchedulerEventType for class
org.apache.tez.dag.app.rm.TaskSchedulerEventHandler
2015-02-26 17:15:53,675 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.tez.dag.app.rm.NMCommunicatorEventType for class
org.apache.tez.dag.app.launcher.ContainerLauncherImpl
2015-02-26 17:15:54,586 WARN [main]
org.apache.hadoop.hdfs.BlockReaderLocal: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
2015-02-26 17:15:54,588 INFO [main]
org.apache.hadoop.service.AbstractService: Service
org.apache.tez.dag.app.DAGAppMaster failed in state INITED; cause:
java.io.FileNotFoundException:
tez.session.local-resources.pb.file-name (No such file or directory)
java.io.FileNotFoundException:
tez.session.local-resources.pb.file-name (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:386)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.tez.dag.app.DAGAppMaster$5.run(DAGAppMaster.java:1954)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:1951)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:1790)
2015-02-26 17:15:54,591 INFO [main]
org.apache.tez.dag.history.HistoryEventHandler: Stopping
HistoryEventHandler
2015-02-26 17:15:54,592 FATAL [main]
org.apache.tez.dag.app.DAGAppMaster: Error starting DAGAppMaster
org.apache.hadoop.service.ServiceStateException:
java.io.FileNotFoundException:
tez.session.local-resources.pb.file-name (No such file or directory)
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.tez.dag.app.DAGAppMaster$5.run(DAGAppMaster.java:1954)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:1951)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:1790)
Caused by: java.io.FileNotFoundException:
tez.session.local-resources.pb.file-name (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:386)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 6 more
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Pig -x tez fails over tez.session.local-resources.pb.file-name ??
Posted by Jan van Bemmelen <ja...@tokyoeye.net>.
Hi Niels,
This is probably caused by a discrepancy between the tez version you run on your cluster (0.4.x) and the version that is in your pig release (0.14 is compiled with tez 0.5.2). Tez 0.4 creates a file called tez.session.local-resources.pb.file-name while tez 0.5 creates a file called tez.session.local-resources.pb. This name change was introduced as part of https://issues.apache.org/jira/browse/TEZ-1106 .
I suspect upgrading your cluster to tez 0.5 would solve the issue.
Regards,
Jan van Bemmelen
> On 26 Feb 2015, at 17:24, Niels Basjes <Ni...@basjes.nl> wrote:
>
> Hi,
>
>
> I have installed tez on an existing hadoop cluster (HDP 2.1.2 with
> kerberos installed).
>
> When I run a very basic pig script I get the error you see below.
>
>
> It effectively fails with
>
>
> java.io.FileNotFoundException:
> tez.session.local-resources.pb.file-name (No such file or directory)
>
>
> I have done some googleing on that term and I was unable to figure out
> what went wrong.
>
>
> Do you guys have any suggestions what I have done wrong?
>
>
> Thanks.
>
>
> Niels Basjes
>
>
>
> 2015-02-26 17:15:53,310 INFO [main]
> org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> 2015-02-26 17:15:53,364 INFO [main]
> org.apache.tez.dag.app.DAGAppMaster: Created DAGAppMaster for
> application appattempt_1423554480020_30727_000002
> 2015-02-26 17:15:53,534 WARN [main]
> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where
> applicable
> 2015-02-26 17:15:53,661 INFO [main]
> org.apache.tez.dag.app.DAGAppMaster: Adding session token to
> jobTokenSecretManager for application
> 2015-02-26 17:15:53,667 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.rm.container.AMContainerEventType for class
> org.apache.tez.dag.app.rm.container.AMContainerMap
> 2015-02-26 17:15:53,668 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.rm.node.AMNodeEventType for class
> org.apache.tez.dag.app.rm.node.AMNodeMap
> 2015-02-26 17:15:53,669 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class
> org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
> 2015-02-26 17:15:53,670 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.dag.event.DAGEventType for class
> org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
> 2015-02-26 17:15:53,670 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.dag.event.VertexEventType for class
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
> 2015-02-26 17:15:53,671 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.dag.event.TaskEventType for class
> org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
> 2015-02-26 17:15:53,672 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class
> org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
> 2015-02-26 17:15:53,673 INFO [main]
> org.apache.tez.dag.app.DAGAppMaster: AM will unregister as this is the
> last attempt, currentAttempt=2, maxAttempts=2
> 2015-02-26 17:15:53,674 INFO [main]
> org.apache.tez.dag.app.rm.TaskSchedulerEventHandler: TaskScheduler
> notified that it should unregister from RM
> 2015-02-26 17:15:53,674 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.rm.AMSchedulerEventType for class
> org.apache.tez.dag.app.rm.TaskSchedulerEventHandler
> 2015-02-26 17:15:53,675 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.tez.dag.app.rm.NMCommunicatorEventType for class
> org.apache.tez.dag.app.launcher.ContainerLauncherImpl
> 2015-02-26 17:15:54,586 WARN [main]
> org.apache.hadoop.hdfs.BlockReaderLocal: The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.
> 2015-02-26 17:15:54,588 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.tez.dag.app.DAGAppMaster failed in state INITED; cause:
> java.io.FileNotFoundException:
> tez.session.local-resources.pb.file-name (No such file or directory)
> java.io.FileNotFoundException:
> tez.session.local-resources.pb.file-name (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:146)
> at java.io.FileInputStream.<init>(FileInputStream.java:101)
> at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:386)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.tez.dag.app.DAGAppMaster$5.run(DAGAppMaster.java:1954)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
> at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:1951)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:1790)
> 2015-02-26 17:15:54,591 INFO [main]
> org.apache.tez.dag.history.HistoryEventHandler: Stopping
> HistoryEventHandler
> 2015-02-26 17:15:54,592 FATAL [main]
> org.apache.tez.dag.app.DAGAppMaster: Error starting DAGAppMaster
> org.apache.hadoop.service.ServiceStateException:
> java.io.FileNotFoundException:
> tez.session.local-resources.pb.file-name (No such file or directory)
> at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at org.apache.tez.dag.app.DAGAppMaster$5.run(DAGAppMaster.java:1954)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
> at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:1951)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:1790)
> Caused by: java.io.FileNotFoundException:
> tez.session.local-resources.pb.file-name (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:146)
> at java.io.FileInputStream.<init>(FileInputStream.java:101)
> at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:386)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 6 more
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes