You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/03 06:22:40 UTC

[GitHub] [hudi] pengzhiwei2018 opened a new pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

pengzhiwei2018 opened a new pull request #3387:
URL: https://github.com/apache/hudi/pull/3387


   
   ## What is the purpose of the pull request
   
   Use HMS to sync hive meta for spark sql to remove the dependency for json and calcite.
   
   ## Brief change log
   
   *(for example:)*
     - *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682094930



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -347,6 +346,8 @@ object DataSourceWriteOptions {
     .defaultValue("false")
     .withDocumentation("")
 
+  // We should use HIVE_SYNC_MODE instead of this config from 0.9.0

Review comment:
       can we add `deprecatedAfter(0.9.0)` ?

##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       +1 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-892775157


   my bad. I checked out the master. I did not realize we had sync mode even before this patch. Ignore my previous question. I see this code block.
   
   ```
   if (!StringUtils.isNullOrEmpty(cfg.syncMode)) {
           switch (cfg.syncMode.toLowerCase()) {
             case "hms":
               ddlExecutor = new HMSDDLExecutor(configuration, cfg, fs);
               break;
             case "hiveql":
               ddlExecutor = new HiveQueryDDLExecutor(cfg, fs, configuration);
               break;
             case "jdbc":
               ddlExecutor = new JDBCExecutor(cfg, fs);
               break;
             default:
               throw new HoodieHiveSyncException("Invalid sync mode given " + cfg.syncMode);
           }
         } else {
           ddlExecutor = cfg.useJdbc ? new JDBCExecutor(cfg, fs) : new HiveQueryDDLExecutor(cfg, fs, configuration);
         }
   ```
   
   So, basically hive sync mode has higher precedence. If not set, we will look into useJdbc. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9a3d8ec68c6a9bf36ecf68b4f2721200180105ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321) 
   * 656226de7302806a140efdec95c42e5961e85650 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682621582



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       Currently `jdbc` is the default value for spark sql. But for spark datasource, If we set `jdbc` as the default here, the "useJdbc"  config will not work. see the logical in `HoodieHiveClient`. So it would affect the old spark datasource job.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 656226de7302806a140efdec95c42e5961e85650 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * 5dd343e30f7560d258887843a811373feb1a6931 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * 5dd343e30f7560d258887843a811373feb1a6931 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395) 
   * b63e015a5795413f21a2c3b96189d1bac832b568 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r681472924



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +398,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()
+    .withDocumentation("Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.")

Review comment:
       Yes, good suggestion!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682249809



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       There have some compatibility problem. If set the HIVE_SYNC_MODE a default value , the `useJdbc` will not work which will affect old hudi table job.
   But I agree to add a Enum for the constant.

##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -347,6 +346,8 @@ object DataSourceWriteOptions {
     .defaultValue("false")
     .withDocumentation("")
 
+  // We should use HIVE_SYNC_MODE instead of this config from 0.9.0

Review comment:
       make sense.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc9a650f2cda42a587ed969de0d8fd3c5f391df7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351) 
   * 5f5eb86f46e4bca94d0afaef7db2909d57c63d8c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682577407



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       what I meant is, lets make jdbc as default (since it spans spark datasource, spark-sql etc). From within spark-sql, may be we can make hms as default(if user does not explicitly set it)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r681938258



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       Also, can we introduce an ENUM for diff modes.

##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       wondering if we can set jdbc as default. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 656226de7302806a140efdec95c42e5961e85650 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326) 
   * cc9a650f2cda42a587ed969de0d8fd3c5f391df7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9a3d8ec68c6a9bf36ecf68b4f2721200180105ff UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356) 
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * a5fb3f243e544f91d61c9dd55da80b26c08c4c4c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400",
       "triggerID" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "77761d33e31dddaacc02487cd03491a41d51491f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "77761d33e31dddaacc02487cd03491a41d51491f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * b63e015a5795413f21a2c3b96189d1bac832b568 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400) 
   * 77761d33e31dddaacc02487cd03491a41d51491f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc9a650f2cda42a587ed969de0d8fd3c5f391df7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351) 
   * 5f5eb86f46e4bca94d0afaef7db2909d57c63d8c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * a5fb3f243e544f91d61c9dd55da80b26c08c4c4c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r681471029



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +398,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()
+    .withDocumentation("Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql.")

Review comment:
       Should also deprecate the `use_jdbc` flag.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f5eb86f46e4bca94d0afaef7db2909d57c63d8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354) 
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9a3d8ec68c6a9bf36ecf68b4f2721200180105ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f5eb86f46e4bca94d0afaef7db2909d57c63d8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356) 
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f5eb86f46e4bca94d0afaef7db2909d57c63d8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354) 
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400",
       "triggerID" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * b63e015a5795413f21a2c3b96189d1bac832b568 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682736531



##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveSyncMode.java
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.ddl;
+
+import java.util.Locale;
+
+public enum HiveSyncMode {
+  HMS, HIVEQL, JDBC;

Review comment:
       java docs please

##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       sounds good




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682621582



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       Currently `jdbc` is the default value for spark sql. But for spark datasource, If we set `jdbc` as the default here, the "useJdbc"  config will not work. see the logical in `HoodieHiveClient`. So it would affect the old spark datasource job.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9a3d8ec68c6a9bf36ecf68b4f2721200180105ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400",
       "triggerID" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * 5dd343e30f7560d258887843a811373feb1a6931 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395) 
   * b63e015a5795413f21a2c3b96189d1bac832b568 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * a5fb3f243e544f91d61c9dd55da80b26c08c4c4c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385) 
   * 5dd343e30f7560d258887843a811373feb1a6931 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400",
       "triggerID" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "77761d33e31dddaacc02487cd03491a41d51491f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1407",
       "triggerID" : "77761d33e31dddaacc02487cd03491a41d51491f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * b63e015a5795413f21a2c3b96189d1bac832b568 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400) 
   * 77761d33e31dddaacc02487cd03491a41d51491f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1407) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1400",
       "triggerID" : "b63e015a5795413f21a2c3b96189d1bac832b568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "77761d33e31dddaacc02487cd03491a41d51491f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1407",
       "triggerID" : "77761d33e31dddaacc02487cd03491a41d51491f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * 77761d33e31dddaacc02487cd03491a41d51491f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1407) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cc9a650f2cda42a587ed969de0d8fd3c5f391df7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682577407



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       what I meant, is lets make jdbc as default (since it spans spark datasource, spark-sql etc). From within spark-sql, may be we can make hms as default(if user does not explicitly set it)

##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       what I meant is, lets make jdbc as default (since it spans spark datasource, spark-sql etc). From within spark-sql, may be we can make hms as default(if user does not explicitly set it)

##########
File path: hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveSyncMode.java
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.ddl;
+
+import java.util.Locale;
+
+public enum HiveSyncMode {
+  HMS, HIVEQL, JDBC;

Review comment:
       java docs please

##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       sounds good




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9a3d8ec68c6a9bf36ecf68b4f2721200180105ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321) 
   * 656226de7302806a140efdec95c42e5961e85650 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan merged pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3387:
URL: https://github.com/apache/hudi/pull/3387


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5dd343e30f7560d258887843a811373feb1a6931",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395",
       "triggerID" : "5dd343e30f7560d258887843a811373feb1a6931",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * a5fb3f243e544f91d61c9dd55da80b26c08c4c4c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385) 
   * 5dd343e30f7560d258887843a811373feb1a6931 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1395) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-892775157


   my bad. I checked out the master. I did not realize we had sync mode even before this patch. Ignore my previous question. I see this code block.
   
   ```
   if (!StringUtils.isNullOrEmpty(cfg.syncMode)) {
           switch (cfg.syncMode.toLowerCase()) {
             case "hms":
               ddlExecutor = new HMSDDLExecutor(configuration, cfg, fs);
               break;
             case "hiveql":
               ddlExecutor = new HiveQueryDDLExecutor(cfg, fs, configuration);
               break;
             case "jdbc":
               ddlExecutor = new JDBCExecutor(cfg, fs);
               break;
             default:
               throw new HoodieHiveSyncException("Invalid sync mode given " + cfg.syncMode);
           }
         } else {
           ddlExecutor = cfg.useJdbc ? new JDBCExecutor(cfg, fs) : new HiveQueryDDLExecutor(cfg, fs, configuration);
         }
   ```
   
   So, basically hive sync mode has higher precedence. If not set, we will look into useJdbc. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f5eb86f46e4bca94d0afaef7db2909d57c63d8c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354) 
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#discussion_r682577407



##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -399,6 +400,11 @@ object DataSourceWriteOptions {
     .defaultValue(1000)
     .withDocumentation("The number of partitions one batch when synchronous partitions to hive.")
 
+  val HIVE_SYNC_MODE: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.datasource.hive_sync.mode")
+    .noDefaultValue()

Review comment:
       what I meant, is lets make jdbc as default (since it spans spark datasource, spark-sql etc). From within spark-sql, may be we can make hms as default(if user does not explicitly set it)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3387: [HUDI-2233] Use HMS To Sync Hive Meta For Spark Sql

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3387:
URL: https://github.com/apache/hudi/pull/3387#issuecomment-891570386


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1321",
       "triggerID" : "9a3d8ec68c6a9bf36ecf68b4f2721200180105ff",
       "triggerType" : "PUSH"
     }, {
       "hash" : "656226de7302806a140efdec95c42e5961e85650",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1326",
       "triggerID" : "656226de7302806a140efdec95c42e5961e85650",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1351",
       "triggerID" : "cc9a650f2cda42a587ed969de0d8fd3c5f391df7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1354",
       "triggerID" : "5f5eb86f46e4bca94d0afaef7db2909d57c63d8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dae0d69eade3ba95d39e37c1851a56534f80e007",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356",
       "triggerID" : "e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6043d6a54b7e2d70a071f556b4eb3da8e3992e2c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385",
       "triggerID" : "a5fb3f243e544f91d61c9dd55da80b26c08c4c4c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dae0d69eade3ba95d39e37c1851a56534f80e007 UNKNOWN
   * e2e35cec3fb48b01bf0a19bf4f2913bd7cb45a2b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1356) 
   * 6043d6a54b7e2d70a071f556b4eb3da8e3992e2c UNKNOWN
   * a5fb3f243e544f91d61c9dd55da80b26c08c4c4c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1385) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org