You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Sangjin Lee (JIRA)" <ji...@apache.org> on 2016/05/05 22:17:12 UTC
[jira] [Commented] (YARN-5045) hbase unit tests fail due to
dependency issues
[ https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273218#comment-15273218 ]
Sangjin Lee commented on YARN-5045:
-----------------------------------
This is basically a *diamond dependency problem*. Timeline service depends on HBase 1.0.1 which in turn depends on hadoop-common *2.5.1*. However, timeline service itself depends on hadoop-common *trunk*. In maven, the trunk version is chosen as the hadoop dependency version is managed.
We knew there is potential risk of this diamond problem when the timeline service is developed on trunk (where the backward incompatible changes can happen) while we also depend on HBase which relies on a released version of hadoop. This is a first manifestation of the issue.
There are several ways we can address this issue, but there are drawbacks on all of the approaches.
(1) we can try to resurrect the metrics v.1 classes (that were removed in HADOOP-12504)
We could add back those classes under {{src/test/java}} of the timelineservice project. While this might be an easy way to get around this specific problem, there are issues. First, it is in poor taste to resurrect code we're getting rid of to work around this problem. Also, it does nothing to handle potential future issues that may arise. What if classes' shapes change instead of being removed?
(2) isolate the hbase unit test code and (en)force the hadoop version that HBase relies on
We could try to override the default version of hadoop-common (trunk: 3.0.0-SNAPSHOT) while running the unit tests. It turns out this requires creating a separate project. Prior to maven 3, there used to be a capability of choosing a different version for the test scope and test scope only. However, maven 3 or later seems to have removed that. Thus, for a given project, there can only be one dependency version regardless of scope.
We could try to isolate the HBase-related unit tests into their own project and enforce 2.5.1 in that project. This has a decent chance of success.
Still there are caveats. For this to work, the portion of the timeline service code that exercises HBase has to ensure it works fine against hadoop-common 2.5.1 as well as trunk. I'm not sure how best to enforce this state.
Also, I thought of moving the non-test code that relies on HBase into this new project as well, but I'm moving away from that idea. The main problem is that we need to prevent the hadoop-common 2.5.1 jar from being incorporated into the hadoop distribution tarball. If we have the non-test code in here, the worry is hadoop-common 2.5.1 may sneak into the tarball.
(3) pull out timeline service entirely out of the hadoop project
This is truly an out-of-the-box idea. Technically it might be feasible to set up the HBase-related part of the timeline service as a separate project from hadoop. Then the hadoop-proper part of the timeline service would interact with the HBase-related part of the timeline service over the wire. That way, hadoop can still be shielded from the thorny HBase dependency issues.
While there is some possibility this may work technically, I think there are several major issues, ranging from logistics of setting up a separate project independent of hadoop, performance implications, etc. I think this is a long shot at best.
Out of these options, it seems to me (2) is the least bad option. I am working on making that work. I'd love to hear your thoughts if you have better suggestions.
> hbase unit tests fail due to dependency issues
> ----------------------------------------------
>
> Key: YARN-5045
> URL: https://issues.apache.org/jira/browse/YARN-5045
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: YARN-2928
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Blocker
>
> After the 5/4 rebase, the hbase unit tests in the timeline service project are failing:
> {noformat}
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage Time elapsed: 5.103 sec <<< ERROR!
> java.io.IOException: Shutting down
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
> at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153)
> at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
> at org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
> at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978)
> at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938)
> at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
> at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750)
> at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87)
> {noformat}
> The root cause is that the hbase mini server depends on hadoop common's {{MetricsServlet}} which has been removed in the trunk (HADOOP-12504):
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/metrics/MetricsServlet
> at org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> at org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> ... 26 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org