You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by "Chen Xi (JIRA)" <ji...@apache.org> on 2017/05/15 07:47:04 UTC
[jira] [Created] (TEZ-3728) When using HDFS federation, setting
tez.simple.history.logging.dir to viewfs:// causes AM to fail
Chen Xi created TEZ-3728:
----------------------------
Summary: When using HDFS federation, setting tez.simple.history.logging.dir to viewfs:// causes AM to fail
Key: TEZ-3728
URL: https://issues.apache.org/jira/browse/TEZ-3728
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.8.5
Environment: hive1.1.0
hadoop2.6.0
Reporter: Chen Xi
tez.simple.history.logging.dir does not support viewfs schema.
In tez-site.xml we have:
{noformat}
<property>
<name>tez.simple.history.logging.dir</name>
<value>viewfs://clusterX/history-tez</value>
</property>
{noformat}
TezAM throws such exception:
{noformat}
2017-05-15 15:34:00,809 [INFO] [main] |impl.SimpleHistoryLoggingService|: Using configured log file location for SimpleHistoryLoggingService logDirPath=viewfs://clusterX/history-tez
2017-05-15 15:34:00,812 [INFO] [main] |impl.SimpleHistoryLoggingService|: Initializing SimpleHistoryLoggingService, logFileLocation=hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001, maxErrors=10
2017-05-15 15:34:00,812 [INFO] [main] |recovery.RecoveryService|: RecoveryService initialized with recoveryPath=hdfs://ns2/tmp/hive/op/_tez_session_dir/6b76d2f7-2f00-4f0c-9a15-4ad8478cf32e/.tez/application_1494315174208_1108/recovery/1, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2017-05-15 15:34:00,814 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:APP_LAUNCHED]: applicationId=application_1494315174208_1108, appSubmitTime=1494833615647, launchTime=1494833639649
2017-05-15 15:34:00,815 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1494315174208_1108_000001, appSubmitTime=1494833615647, launchTime=1494833639649
2017-05-15 15:34:00,818 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |impl.SimpleHistoryLoggingService|: Starting SimpleHistoryLoggingService
2017-05-15 15:34:00,819 [INFO] [ServiceThread:org.apache.tez.dag.history.HistoryEventHandler] |service.AbstractService|: Service org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService failed in state STARTED; cause: java.lang.IllegalArgumentException: Wrong FS: hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001, expected: viewfs://clusterX/
java.lang.IllegalArgumentException: Wrong FS: hdfs://ns2/history-tez/history.txt.appattempt_1494315174208_1108_000001, expected: viewfs://clusterX/
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.getUriPath(ViewFileSystem.java:117)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:615)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService.serviceStart(SimpleHistoryLoggingService.java:96)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.tez.dag.history.HistoryEventHandler.serviceStart(HistoryEventHandler.java:105)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1838)
at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1859)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)