You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Elek, Marton (JIRA)" <ji...@apache.org> on 2019/05/14 11:53:00 UTC
[jira] [Commented] (HDDS-1525) Mapreduce failure when using Hadoop
2.7.5
[ https://issues.apache.org/jira/browse/HDDS-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16839340#comment-16839340 ]
Elek, Marton commented on HDDS-1525:
------------------------------------
Thank you very much to report the problem [~Sammi]
The suggested way to run mapreduce job is to put the right ozonefs jar (and only that one) to the classpath. See https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html for more details.
Background: We use project specific classpath: for example s3g daemon has different classpath than ozone. One of the reason for that is the ability to support different versions from the same jars. For example s3g uses newer jaxrs stack than the core hadoop. share/ozone/lib contains all of the jar files sometimes the same jar file with different version.
TLDR: adding all the jar files may cause conflicts.
Unfortunately I couldn't say that this bug report is invalid, I think the ozonefs can be broken on trunk.
Would you be so kind to help to test it with the following approach:
1. Add only the share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-0.4.0-alpha.jar to the classpath
2. Use org.apache.hadoop.fs.ozone.BasicOzoneFileSystem as the HCFS implementation
> Mapreduce failure when using Hadoop 2.7.5
> -----------------------------------------
>
> Key: HDDS-1525
> URL: https://issues.apache.org/jira/browse/HDDS-1525
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
> Affects Versions: 0.4.0
> Reporter: Sammi Chen
> Priority: Major
> Attachments: teragen.log
>
>
> Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, while teragen failed.
> When add -verbose:class to java options, it shows that class KeyProvider is loaded twice by different classloader while it is only loaded once when execute "hdfs dfs -ls /"
> All jars under share/ozone/lib are added into hadoop classpath except ozone file system current lib jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org