You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@gmail.com> on 2019/05/19 12:56:59 UTC
Review Request 70677: RANGER-2432: Upgrade Hadoop Version to 3.1.1
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70677/
-----------------------------------------------------------
Review request for ranger.
Bugs: RANGER-2432
https://issues.apache.org/jira/browse/RANGER-2432
Repository: ranger
Description
-------
**Following libraries can be upgraded:**
Hadoop version from 3.0.0 to 3.1.1
HBase version from 2.0.0 to 2.0.2
Hive version from 3.0.0 to 3.1.0
Atlas version from 1.1.0 to 2.0.0
**Note:** Disabled Hbase test cases as setup() in HBaseRangerAuthorizationTest is failing. It could be due to any of the issues mentioned in https://issues.apache.org/jira/browse/HBASE-21688. If its so, then probably fix is not available in the released hbase versions as I have tried all released Hbase > 2.0.0. Also have tried with all the combination of hadoop > 3.0.0
It appears this issue is due to changes in hadoop after 3.0.0 release but not fixed from hbase end in released version.
I could not find any work around to get them working as earlier. If you have any approach to fix these testcases please leave your comment here or in the Jira.
**Stacktrace after the testcase failure:**
Thread 1 (main):
State: RUNNABLE
Blocked count: 50
Waited count: 499
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341)
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258)
org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778)
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 207.69 s <<< FAILURE! - in org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
[ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest Time elapsed: 207.69 s <<< ERROR!
java.io.IOException: Shutting down
at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
Caused by: java.lang.RuntimeException: Master not initialized after 200000ms seconds
at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
Diffs
-----
hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java 172b310f1
hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java 62381743f
pom.xml 40b053bd2
Diff: https://reviews.apache.org/r/70677/diff/1/
Testing
-------
Tested Ranger Build, Ranger installation and Ranger admin service, zone, policy, user and group crud operation.
Thanks,
Pradeep Agrawal
Re: Review Request 70677: RANGER-2432: Upgrade Hadoop Version to 3.1.1
Posted by Mehul Parikh <me...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70677/#review215565
-----------------------------------------------------------
Ship it!
Ship It!
- Mehul Parikh
On May 19, 2019, 12:56 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70677/
> -----------------------------------------------------------
>
> (Updated May 19, 2019, 12:56 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2432
> https://issues.apache.org/jira/browse/RANGER-2432
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Following libraries can be upgraded:**
>
> Hadoop version from 3.0.0 to 3.1.1
> HBase version from 2.0.0 to 2.0.2
> Hive version from 3.0.0 to 3.1.0
> Atlas version from 1.1.0 to 2.0.0
>
>
> **Note:** Disabled Hbase test cases as setup() in HBaseRangerAuthorizationTest is failing. It could be due to any of the issues mentioned in https://issues.apache.org/jira/browse/HBASE-21688. If its so, then probably fix is not available in the released hbase versions as I have tried all released Hbase > 2.0.0. Also have tried with all the combination of hadoop > 3.0.0
> It appears this issue is due to changes in hadoop after 3.0.0 release but not fixed from hbase end in released version.
>
> I could not find any work around to get them working as earlier. If you have any approach to fix these testcases please leave your comment here or in the Jira.
>
> **Stacktrace after the testcase failure:**
>
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 50
> Waited count: 499
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
> org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341)
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258)
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778)
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 207.69 s <<< FAILURE! - in org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest Time elapsed: 207.69 s <<< ERROR!
> java.io.IOException: Shutting down
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> Caused by: java.lang.RuntimeException: Master not initialized after 200000ms seconds
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
>
>
> Diffs
> -----
>
> hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java 172b310f1
> hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java 62381743f
> pom.xml 40b053bd2
> security-admin/pom.xml 76bb317e5
> src/main/assembly/admin-web.xml c862155f3
>
>
> Diff: https://reviews.apache.org/r/70677/diff/3/
>
>
> Testing
> -------
>
> Tested Ranger Build, Ranger installation and Ranger admin service, zone, policy, user and group crud operation.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 70677: RANGER-2432: Upgrade Hadoop Version to 3.1.1
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On May 24, 2019, 3:18 p.m., Velmurugan Periasamy wrote:
> > Ship It!
>
> Abhay Kulkarni wrote:
> About description of this JIRA: In my testing for hbase and hadoop up I found the following to be the cause of the failure.
>
> 2019-05-23 16:23:56,805 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Formatting using clusterid: testClusterID
> 2019-05-23 16:23:58,033 WARN [main] impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
> 2019-05-23 16:23:58,186 WARN [main] http.HttpRequestLog: Jetty request log can only be enabled using Log4j
> 2019-05-23 16:23:58,863 WARN [main] common.MetricsLoggerTask: Metrics logging will not be async since the logger is not log4j
> 2019-05-23 16:23:58,947 WARN [main] http.HttpRequestLog: Jetty request log can only be enabled using Log4j
> 2019-05-23 16:23:59,224 WARN [main] common.MetricsLoggerTask: Metrics logging will not be async since the logger is not log4j
> 2019-05-23 16:23:59,722 ERROR [main] server.ZooKeeperServer: ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
> 2019-05-23 16:24:01,269 WARN [Thread-107] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
> 2019-05-23 16:24:02,216 ERROR [Thread-107] config.RangerConfiguration: addResourceIfReadable(ranger-hbase-audit.xml): couldn't find resource file location
> 2019-05-23 16:24:02,302 WARN [RS:0;10.22.9.38:52417] regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying.
> 2019-05-23 16:24:02,916 WARN [master/10.22.9.38:52415] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
> 2019-05-23 16:24:03,933 WARN [master/10.22.9.38:52415] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
> 2019-05-23 16:24:04,946 WARN [master/10.22.9.38:52415] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
> 2019-05-23 16:24:05,324 WARN [RS:0;10.22.9.38:52417] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
> 2019-05-23 16:24:08,521 ERROR [RS:0;10.22.9.38:52417] regionserver.HRegionServer: ***** ABORTING region server 10.22.9.38,52417,1558653841129: Unhandled: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected *****
> java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
> at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:768)
> at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:848)
> at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:843)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:856)
> at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:168)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165)
> at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:764)
> at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:494)
> at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:251)
> at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:73)
> at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:48)
> at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:276)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1290)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1172)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:989)
> at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183)
> at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129)
> at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
> at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164)
> at java.lang.Thread.run(Thread.java:748)
> 2019-05-23 16:24:08,524 ERROR [RS:0;10.22.9.38:52417] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]
>
>
> Also, Hbase JIRA which addresses this is : https://issues.apache.org/jira/browse/HBASE-22394
Complete stacktrace after setting test cases log level(ranger.root.logger=WARN,console) in the log4j.properties of ranger hbase plugins : https://gist.github.com/pradeepagrawal8184/798bd00df71797ae74fe7aa4d53f3017
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70677/#review215499
-----------------------------------------------------------
On May 19, 2019, 12:56 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70677/
> -----------------------------------------------------------
>
> (Updated May 19, 2019, 12:56 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2432
> https://issues.apache.org/jira/browse/RANGER-2432
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Following libraries can be upgraded:**
>
> Hadoop version from 3.0.0 to 3.1.1
> HBase version from 2.0.0 to 2.0.2
> Hive version from 3.0.0 to 3.1.0
> Atlas version from 1.1.0 to 2.0.0
>
>
> **Note:** Disabled Hbase test cases as setup() in HBaseRangerAuthorizationTest is failing. It could be due to any of the issues mentioned in https://issues.apache.org/jira/browse/HBASE-21688. If its so, then probably fix is not available in the released hbase versions as I have tried all released Hbase > 2.0.0. Also have tried with all the combination of hadoop > 3.0.0
> It appears this issue is due to changes in hadoop after 3.0.0 release but not fixed from hbase end in released version.
>
> I could not find any work around to get them working as earlier. If you have any approach to fix these testcases please leave your comment here or in the Jira.
>
> **Stacktrace after the testcase failure:**
>
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 50
> Waited count: 499
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
> org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341)
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258)
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778)
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 207.69 s <<< FAILURE! - in org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest Time elapsed: 207.69 s <<< ERROR!
> java.io.IOException: Shutting down
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> Caused by: java.lang.RuntimeException: Master not initialized after 200000ms seconds
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
>
>
> Diffs
> -----
>
> hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java 172b310f1
> hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java 62381743f
> pom.xml 40b053bd2
> security-admin/pom.xml 76bb317e5
> src/main/assembly/admin-web.xml c862155f3
>
>
> Diff: https://reviews.apache.org/r/70677/diff/3/
>
>
> Testing
> -------
>
> Tested Ranger Build, Ranger installation and Ranger admin service, zone, policy, user and group crud operation.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 70677: RANGER-2432: Upgrade Hadoop Version to 3.1.1
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
> On May 24, 2019, 3:18 p.m., Velmurugan Periasamy wrote:
> > Ship It!
About description of this JIRA: In my testing for hbase and hadoop up I found the following to be the cause of the failure.
2019-05-23 16:23:56,805 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: testClusterID
2019-05-23 16:23:58,033 WARN [main] impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2019-05-23 16:23:58,186 WARN [main] http.HttpRequestLog: Jetty request log can only be enabled using Log4j
2019-05-23 16:23:58,863 WARN [main] common.MetricsLoggerTask: Metrics logging will not be async since the logger is not log4j
2019-05-23 16:23:58,947 WARN [main] http.HttpRequestLog: Jetty request log can only be enabled using Log4j
2019-05-23 16:23:59,224 WARN [main] common.MetricsLoggerTask: Metrics logging will not be async since the logger is not log4j
2019-05-23 16:23:59,722 ERROR [main] server.ZooKeeperServer: ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
2019-05-23 16:24:01,269 WARN [Thread-107] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
2019-05-23 16:24:02,216 ERROR [Thread-107] config.RangerConfiguration: addResourceIfReadable(ranger-hbase-audit.xml): couldn't find resource file location
2019-05-23 16:24:02,302 WARN [RS:0;10.22.9.38:52417] regionserver.HRegionServer: reportForDuty failed; sleeping and then retrying.
2019-05-23 16:24:02,916 WARN [master/10.22.9.38:52415] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
2019-05-23 16:24:03,933 WARN [master/10.22.9.38:52415] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
2019-05-23 16:24:04,946 WARN [master/10.22.9.38:52415] assignment.AssignmentManager: No servers available; cannot place 1 unassigned regions.
2019-05-23 16:24:05,324 WARN [RS:0;10.22.9.38:52417] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
2019-05-23 16:24:08,521 ERROR [RS:0;10.22.9.38:52417] regionserver.HRegionServer: ***** ABORTING region server 10.22.9.38,52417,1558653841129: Unhandled: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected *****
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.hdfs.protocol.HdfsFileStatus, but class was expected
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:768)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$400(FanOutOneBlockAsyncDFSOutputHelper.java:118)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:848)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$16.doCall(FanOutOneBlockAsyncDFSOutputHelper.java:843)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createOutput(FanOutOneBlockAsyncDFSOutputHelper.java:856)
at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:168)
at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165)
at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:764)
at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:494)
at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:251)
at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:73)
at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:48)
at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:276)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1290)
at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:989)
at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183)
at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129)
at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164)
at java.lang.Thread.run(Thread.java:748)
2019-05-23 16:24:08,524 ERROR [RS:0;10.22.9.38:52417] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]
Also, Hbase JIRA which addresses this is : https://issues.apache.org/jira/browse/HBASE-22394
- Abhay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70677/#review215499
-----------------------------------------------------------
On May 19, 2019, 12:56 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70677/
> -----------------------------------------------------------
>
> (Updated May 19, 2019, 12:56 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2432
> https://issues.apache.org/jira/browse/RANGER-2432
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Following libraries can be upgraded:**
>
> Hadoop version from 3.0.0 to 3.1.1
> HBase version from 2.0.0 to 2.0.2
> Hive version from 3.0.0 to 3.1.0
> Atlas version from 1.1.0 to 2.0.0
>
>
> **Note:** Disabled Hbase test cases as setup() in HBaseRangerAuthorizationTest is failing. It could be due to any of the issues mentioned in https://issues.apache.org/jira/browse/HBASE-21688. If its so, then probably fix is not available in the released hbase versions as I have tried all released Hbase > 2.0.0. Also have tried with all the combination of hadoop > 3.0.0
> It appears this issue is due to changes in hadoop after 3.0.0 release but not fixed from hbase end in released version.
>
> I could not find any work around to get them working as earlier. If you have any approach to fix these testcases please leave your comment here or in the Jira.
>
> **Stacktrace after the testcase failure:**
>
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 50
> Waited count: 499
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
> org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341)
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258)
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778)
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 207.69 s <<< FAILURE! - in org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest Time elapsed: 207.69 s <<< ERROR!
> java.io.IOException: Shutting down
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> Caused by: java.lang.RuntimeException: Master not initialized after 200000ms seconds
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
>
>
> Diffs
> -----
>
> hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java 172b310f1
> hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java 62381743f
> pom.xml 40b053bd2
> security-admin/pom.xml 76bb317e5
> src/main/assembly/admin-web.xml c862155f3
>
>
> Diff: https://reviews.apache.org/r/70677/diff/3/
>
>
> Testing
> -------
>
> Tested Ranger Build, Ranger installation and Ranger admin service, zone, policy, user and group crud operation.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 70677: RANGER-2432: Upgrade Hadoop Version to 3.1.1
Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70677/#review215499
-----------------------------------------------------------
Ship it!
Ship It!
- Velmurugan Periasamy
On May 19, 2019, 12:56 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70677/
> -----------------------------------------------------------
>
> (Updated May 19, 2019, 12:56 p.m.)
>
>
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2432
> https://issues.apache.org/jira/browse/RANGER-2432
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Following libraries can be upgraded:**
>
> Hadoop version from 3.0.0 to 3.1.1
> HBase version from 2.0.0 to 2.0.2
> Hive version from 3.0.0 to 3.1.0
> Atlas version from 1.1.0 to 2.0.0
>
>
> **Note:** Disabled Hbase test cases as setup() in HBaseRangerAuthorizationTest is failing. It could be due to any of the issues mentioned in https://issues.apache.org/jira/browse/HBASE-21688. If its so, then probably fix is not available in the released hbase versions as I have tried all released Hbase > 2.0.0. Also have tried with all the combination of hadoop > 3.0.0
> It appears this issue is due to changes in hadoop after 3.0.0 release but not fixed from hbase end in released version.
>
> I could not find any work around to get them working as earlier. If you have any approach to fix these testcases please leave your comment here or in the Jira.
>
> **Stacktrace after the testcase failure:**
>
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 50
> Waited count: 499
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.hadoop.hbase.util.Threads$PrintThreadInfoLazyHolder$1.printThreadInfo(Threads.java:294)
> org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:341)
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:258)
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:115)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1038)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:984)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:855)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:849)
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:778)
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 207.69 s <<< FAILURE! - in org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest Time elapsed: 207.69 s <<< ERROR!
> java.io.IOException: Shutting down
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
> Caused by: java.lang.RuntimeException: Master not initialized after 200000ms seconds
> at org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.setup(HBaseRangerAuthorizationTest.java:100)
>
>
> Diffs
> -----
>
> hbase-agent/src/test/java/org/apache/ranger/authorization/hbase/HBaseRangerAuthorizationTest.java 172b310f1
> hive-agent/src/test/java/org/apache/ranger/services/hive/HIVERangerAuthorizerTest.java 62381743f
> pom.xml 40b053bd2
> security-admin/pom.xml 76bb317e5
> src/main/assembly/admin-web.xml c862155f3
>
>
> Diff: https://reviews.apache.org/r/70677/diff/3/
>
>
> Testing
> -------
>
> Tested Ranger Build, Ranger installation and Ranger admin service, zone, policy, user and group crud operation.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>