You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/29 09:39:31 UTC

[GitHub] [hudi] xicm opened a new pull request, #5464: Hudi 3980

xicm opened a new pull request, #5464:
URL: https://github.com/apache/hudi/pull/5464

   ## What is the purpose of the pull request
   
   Add configurations in HoodieHBaseIndexConfig.java to support kerberos hbase connection.
   
   ## Brief change log
   
   Add configureations, getters and setters in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieHBaseIndexConfig.java,hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
   
   Modify HbaseConnection in hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/hbase/SparkHoodieHBaseIndex.java
   
   ## Verify this pull request
   
   I tested in my local cluster, spark 3.1.1, scala 2.12.10.
   ```
   df.write.format("org.apache.hudi").
         options(getQuickstartWriteConfigs).
         option(PRECOMBINE_FIELD.key, "ts").
         option(RECORDKEY_FIELD.key, "uuid").
         option(PARTITIONPATH_FIELD.key, "partitionpath").
         option(TBL_NAME.key, tableName).
         option(INDEX_TYPE.key, "HBASE").
         option(ZKQUORUM.key, zk_quorum).
         option(ZKPORT.key, "2181").
         option(ZK_NODE_PATH.key, zk_node_path).
         option(SECURITY_AUTHENTICATION.key(), "kerberos").
         option(KERBEROS_USER_KEYTAB.key(), keytab). // "keytab" is the name of keytab file, it shoud be add with --files 
         option(KERBEROS_USER_PRINCIPAL.key(), principal).
         option(REGIONSERVER_PRINCIPAL.key(), server_principal).
         option(MASTER_PRINCIPAL.key(), master_principal).
         option(TABLENAME.key(), tableName).
         option(EMBEDDED_TIMELINE_SERVER_ENABLE.key, false).
         mode(Overwrite).
         save(tablePath)
   ```
   Problems you may encounter during testing
   1. NullPointException about hbase index configureations, see https://github.com/apache/hudi/pull/5308
   2. some ClassNotFoundExceptions I described at https://issues.apache.org/jira/browse/HUDI-3983
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1118218555

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387) 
   * 636d10e4d61c76f04dae08da247651fcd019b7ad UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1113137976

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1118251914

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436",
       "triggerID" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "triggerType" : "PUSH"
     }, {
       "hash" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387) 
   * 636d10e4d61c76f04dae08da247651fcd019b7ad Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436) 
   * 523c84e2eae12e127b2be09be70cabe87984f603 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1118411526

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436",
       "triggerID" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "triggerType" : "PUSH"
     }, {
       "hash" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8439",
       "triggerID" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 523c84e2eae12e127b2be09be70cabe87984f603 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8439) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan merged pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #5464:
URL: https://github.com/apache/hudi/pull/5464


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1113135368

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1118350469

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436",
       "triggerID" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "triggerType" : "PUSH"
     }, {
       "hash" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8439",
       "triggerID" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 636d10e4d61c76f04dae08da247651fcd019b7ad Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436) 
   * 523c84e2eae12e127b2be09be70cabe87984f603 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8439) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1113233630

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1118220174

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436",
       "triggerID" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387) 
   * 636d10e4d61c76f04dae08da247651fcd019b7ad Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1118256507

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387",
       "triggerID" : "578d03bf52a55c34f5f162728db64ab3c6cf129e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436",
       "triggerID" : "636d10e4d61c76f04dae08da247651fcd019b7ad",
       "triggerType" : "PUSH"
     }, {
       "hash" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8439",
       "triggerID" : "523c84e2eae12e127b2be09be70cabe87984f603",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 578d03bf52a55c34f5f162728db64ab3c6cf129e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8387) 
   * 636d10e4d61c76f04dae08da247651fcd019b7ad Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8436) 
   * 523c84e2eae12e127b2be09be70cabe87984f603 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8439) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1126696936

   cool, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1123842814

   @xicm : can you confirm one thing for me. for existing users who is not leverage kerberos, no additional configs are required when they migrate to 0.12 for eg right? I mean, after this patch is landed. if yes, we are good to go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xicm commented on pull request #5464: [HUDI-3980] Suport kerberos hbase index

Posted by GitBox <gi...@apache.org>.
xicm commented on PR #5464:
URL: https://github.com/apache/hudi/pull/5464#issuecomment-1124447910

   hi, @nsivabalan thanks for review. No additional configs are required when Kerberos is not enabled in the hbase cluster.
   we check whether Kerberos is enabled based on SECURITY_AUTHENTICATION. Its default value is simple, and Kerberos-related configuration is not required.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org