You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Ayush Saxena (Jira)" <ji...@apache.org> on 2023/12/21 06:25:00 UTC
[jira] [Commented] (TEZ-4415) Hadoop archives created with Tez miss index files
[ https://issues.apache.org/jira/browse/TEZ-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799281#comment-17799281 ]
Ayush Saxena commented on TEZ-4415:
-----------------------------------
Last time I tried on our internal build. Today I tried on Hadoop-3.3.6 & Tez-0.10.2 & *it works*
{noformat}
ayushsaxena@ayushsaxena apache-tez-0.10.2-bin % $HADOOP_HOME/bin/hadoop archive -D mapreduce.framework.name=yarn-tez -archiveName abc.har -p /out /haro
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/ayushsaxena/code/cluster/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/ayushsaxena/code/cluster/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2023-12-21 11:48:49,203 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-12-21 11:48:49,667 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2023-12-21 11:48:49,931 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2023-12-21 11:48:49,944 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2023-12-21 11:48:50,011 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/ayushsaxena/.staging/job_1703099201534_0008
2023-12-21 11:48:50,524 INFO mapreduce.JobSubmitter: number of splits:1
2023-12-21 11:48:50,607 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1703099201534_0008
2023-12-21 11:48:50,607 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-12-21 11:48:50,685 INFO client.YARNRunner: Number of stages: 2
2023-12-21 11:48:50,704 INFO conf.Configuration: resource-types.xml not found
2023-12-21 11:48:50,704 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-12-21 11:48:50,968 INFO counters.Limits: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2023-12-21 11:48:50,968 INFO counters.Limits: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120
2023-12-21 11:48:50,968 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.10.2, revision=72977b8720b2337ab0a0a3bf3b12e1c57900fa69, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2022-07-08T16:21:10Z, buildUser=laszlobodor, buildJavaVersion=1.8.0_292 ]
2023-12-21 11:48:50,976 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2023-12-21 11:48:50,977 INFO client.TezClient: Submitting DAG application with id: application_1703099201534_0008
2023-12-21 11:48:50,978 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: hdfs://localhost:52714/apps/tez/tez.tar.gz
2023-12-21 11:48:50,978 INFO client.TezClientUtils: Using tez.lib.uris.classpath value from configuration: null
2023-12-21 11:48:50,986 INFO client.TezClient: Tez system stage directory hdfs://localhost:52714/tmp/hadoop-yarn/staging/ayushsaxena/.staging/job_1703099201534_0008/.tez/application_1703099201534_0008 doesn't exist and is created
2023-12-21 11:48:51,075 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1703099201534_0008, dagName=hadoop-archives-3.3.6.jar
2023-12-21 11:48:51,103 INFO impl.YarnClientImpl: Submitted application application_1703099201534_0008
2023-12-21 11:48:51,105 INFO client.TezClient: The url to track the Tez AM: http://localhost:8088/proxy/application_1703099201534_0008/
2023-12-21 11:48:54,254 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2023-12-21 11:48:54,274 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1703099201534_0008/
2023-12-21 11:48:54,275 INFO mapreduce.Job: Running job: job_1703099201534_0008
2023-12-21 11:48:55,292 INFO mapreduce.Job: Job job_1703099201534_0008 running in uber mode : false
2023-12-21 11:48:55,296 INFO mapreduce.Job: map 0% reduce 0%
2023-12-21 11:48:58,326 INFO mapreduce.Job: map 100% reduce 100%
2023-12-21 11:48:58,330 INFO mapreduce.Job: Job job_1703099201534_0008 completed successfully
2023-12-21 11:48:58,339 INFO mapreduce.Job: Counters: 0
ayushsaxena@ayushsaxena apache-tez-0.10.2-bin % hdfs dfs -ls /haro/abc.har
zsh: command not found: hdfs
ayushsaxena@ayushsaxena apache-tez-0.10.2-bin % $HADOOP_HOME/bin/hdfs dfs -ls /haro/abc.har
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/ayushsaxena/code/cluster/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/ayushsaxena/code/cluster/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2023-12-21 11:49:53,241 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 ayushsaxena supergroup 0 2023-12-21 11:48 /haro/abc.har/_SUCCESS
-rw-r--r-- 3 ayushsaxena supergroup 8179 2023-12-21 11:48 /haro/abc.har/part-0{noformat}
> Hadoop archives created with Tez miss index files
> -------------------------------------------------
>
> Key: TEZ-4415
> URL: https://issues.apache.org/jira/browse/TEZ-4415
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.2
> Reporter: Christophe Préaud
> Priority: Minor
>
> When a hadoop archive is created with Tez, the _index and _masterindex files are not created:
> {code:java}
> # create hadoop archive with Tez
> hadoop archive -D mapreduce.framework.name=yarn-tez -archiveName data.har -p /user/preaudc/data /user/preaudc
> (...)
> 22/05/23 13:04:39 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.9.2, revision=10cb3519bd34389210e6511a2ba291b52dcda081, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2019-03-19T20:44:07Z ]
> (...)
> # _index and _masterindex files are not created
> hdfs dfs -ls /user/preaudc/data.har
> Found 2 items
> -rw-r--r-- 3 preaudc preaudc 0 2022-05-23 13:06 /user/preaudc/data.har/_SUCCESS
> -rw-r--r-- 3 preaudc preaudc 2537147461 2022-05-23 13:06 /user/preaudc/data.har/part-0
> # the hadoop archive is thus unreadable
> hdfs dfs -ls har:/user/preaudc/data.har
> ls: Invalid path for the Har Filesystem. No index file in har:/user/preaudc/data.har{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)