You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2022/02/01 05:56:00 UTC

[jira] [Commented] (HUDI-1180) Upgrade HBase to 2.x

    [ https://issues.apache.org/jira/browse/HUDI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485055#comment-17485055 ] 

Ethan Guo commented on HUDI-1180:
---------------------------------

WIP PR here: https://github.com/apache/hudi/pull/4695

> Upgrade HBase to 2.x
> --------------------
>
>                 Key: HUDI-1180
>                 URL: https://issues.apache.org/jira/browse/HUDI-1180
>             Project: Apache Hudi
>          Issue Type: Task
>          Components: writer-core
>    Affects Versions: 0.9.0
>            Reporter: Wenning Ding
>            Assignee: Ethan Guo
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> Trying to upgrade HBase to 2.3.3 but ran into several issues.
> According to the Hadoop version support matrix: [http://hbase.apache.org/book.html#hadoop], also need to upgrade Hadoop to 2.8.5+.
>  
> There are several API conflicts between HBase 2.2.3 and HBase 1.2.3, we need to resolve this first. After resolving conflicts, I am able to compile it but then I ran into a tricky jetty version issue during the testing:
> {code:java}
> [ERROR] TestHBaseIndex.testDelete()  Time elapsed: 4.705 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR] TestHBaseIndex.testSimpleTagLocationAndUpdate()  Time elapsed: 0.174 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR] TestHBaseIndex.testSimpleTagLocationAndUpdateWithRollback()  Time elapsed: 0.076 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR] TestHBaseIndex.testSmallBatchSize()  Time elapsed: 0.122 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR] TestHBaseIndex.testTagLocationAndDuplicateUpdate()  Time elapsed: 0.16 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR] TestHBaseIndex.testTotalGetsBatching()  Time elapsed: 1.771 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR] TestHBaseIndex.testTotalPutsBatching()  Time elapsed: 0.082 s  <<< ERROR!
> java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> 34206 [Thread-260] WARN  org.apache.hadoop.hdfs.server.datanode.DirectoryScanner  - DirectoryScanner: shutdown has been called
> 34240 [BP-1058834949-10.0.0.2-1597189606506 heartbeating to localhost/127.0.0.1:55924] WARN  org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager  - IncrementalBlockReportManager interrupted
> 34240 [BP-1058834949-10.0.0.2-1597189606506 heartbeating to localhost/127.0.0.1:55924] WARN  org.apache.hadoop.hdfs.server.datanode.DataNode  - Ending block pool service for: Block pool BP-1058834949-10.0.0.2-1597189606506 (Datanode Uuid cb7bd8aa-5d79-4955-b1ec-bdaf7f1b6431) service to localhost/127.0.0.1:55924
> 34246 [refreshUsed-/private/var/folders/98/mxq3vc_n6l5728rf1wmcwrqs52lpwg/T/temp1791820148926982977/dfs/data/data1/current/BP-1058834949-10.0.0.2-1597189606506] WARN  org.apache.hadoop.fs.CachingGetSpaceUsed  - Thread Interrupted waiting to refresh disk information: sleep interrupted
> 34247 [refreshUsed-/private/var/folders/98/mxq3vc_n6l5728rf1wmcwrqs52lpwg/T/temp1791820148926982977/dfs/data/data2/current/BP-1058834949-10.0.0.2-1597189606506] WARN  org.apache.hadoop.fs.CachingGetSpaceUsed  - Thread Interrupted waiting to refresh disk information: sleep interrupted
> 37192 [HBase-Metrics2-1] WARN  org.apache.hadoop.metrics2.impl.MetricsConfig  - Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties
> 43904 [master/iad1-ws-cor-r12:0:becomeActiveMaster-SendThread(localhost:58768)] WARN  org.apache.zookeeper.ClientCnxn  - Session 0x173dfeb0c8b0004 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Errors: 
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [ERROR]   org.eclipse.jetty.server.session.SessionHandler.setHttpOnly(Z)V
> [INFO] 
> [ERROR] Tests run: 10, Failures: 0, Errors: 7, Skipped: 0
> [INFO] 
> {code}
> Basically currently Hudi and it's dependency Javalin depend on Jetty 9.4.x but Hbase depends on jetty 9.3.x. And they have incompatible APIs which could not be easily resolved.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)