You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2023/05/18 07:38:56 UTC

[GitHub] [hudi] voonhous opened a new pull request, #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

voonhous opened a new pull request, #8752:
URL: https://github.com/apache/hudi/pull/8752

   …eltaStreamer entrypoints
   
   ### Change Logs
   
   In Hudi 0.10.X, Hudi assumes that hoodie table properties contains the `hoodie.datasource.write.hive_style_partitioning` configuration value. 
   
   As such `hoodie.properties` should contain this config for all Hudi tables created by Hudi > 0.10.X. However, in DeltaStreamer entrypoints, this configuration is not written to `hoodie.properties`, which will cause incorrect query results via Spark.
   
   This PR fixes the aforementioned assumptions by ensuring that  `hoodie.datasource.write.hive_style_partitioning`  is written to `hoodie.properties`.
   
   ### Impact
   
   _Describe any public API or user-facing feature change or any performance impact._
   
   None
   
   ### Risk level (write none, low medium or high below)
   
   _If medium or high, explain what verification was done to mitigate the risks._
   
   Low
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change_
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552689739

   Also, we may need add an improvement for how tables are initialised in the future so that all entrypoints will use a common `HoodieTableMetaClient` initialiser.
   
   Instead of the current implementation where all entrypoints are dictating their own means of creating a `HoodieTableMetaClient`.
   
   Once that is done, we will also need a configuration validation component to check the configs of a table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552939562

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189",
       "triggerID" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * 6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183) 
   * f9b5f2d4727ffabc20e3e28e78e49009f6fa221e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #8752:
URL: https://github.com/apache/hudi/pull/8752#discussion_r1197514595


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -495,6 +498,8 @@ public Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> readFromSource(
           .setPartitionMetafileUseBaseFormat(props.getBoolean(HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.key(),
               HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.defaultValue()))
           .setShouldDropPartitionColumns(isDropPartitionColumns())
+          .setHiveStylePartitioningEnable(props.getBoolean(HIVE_STYLE_PARTITIONING_ENABLE.key(),

Review Comment:
   Nope, not written to `hoodie.properties`.
   
   ```properties
   #Properties saved on 2023-05-18T07:48:15.181Z
   #Thu May 18 15:48:15 SGT 2023
   hoodie.table.precombine.field=timestamp
   hoodie.datasource.write.drop.partition.columns=false
   hoodie.table.partition.fields=partition_path
   hoodie.table.type=COPY_ON_WRITE
   hoodie.archivelog.folder=archived
   hoodie.table.cdc.enabled=false
   hoodie.populate.meta.fields=true
   hoodie.partition.metafile.use.base.format=false
   hoodie.table.version=5
   hoodie.timeline.layout.version=1
   hoodie.table.base.file.format=PARQUET
   hoodie.table.recordkey.fields=_row_key
   hoodie.table.name=hoodie_trips
   hoodie.table.keygenerator.class=org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer$TestGenerator
   hoodie.datasource.write.hive_style_partitioning=true
   hoodie.table.checksum=224269229
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #8752:
URL: https://github.com/apache/hudi/pull/8752#discussion_r1197509863


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -495,6 +498,8 @@ public Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> readFromSource(
           .setPartitionMetafileUseBaseFormat(props.getBoolean(HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.key(),
               HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.defaultValue()))
           .setShouldDropPartitionColumns(isDropPartitionColumns())
+          .setHiveStylePartitioningEnable(props.getBoolean(HIVE_STYLE_PARTITIONING_ENABLE.key(),

Review Comment:
   In `DeltaSync`, does `urlEncodePartitioningEnabled` write to hoodie.properties or not? If not, I think we can add it as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552892692

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * 6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183) 
   * f9b5f2d4727ffabc20e3e28e78e49009f6fa221e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #8752:
URL: https://github.com/apache/hudi/pull/8752#discussion_r1197509863


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -495,6 +498,8 @@ public Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> readFromSource(
           .setPartitionMetafileUseBaseFormat(props.getBoolean(HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.key(),
               HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.defaultValue()))
           .setShouldDropPartitionColumns(isDropPartitionColumns())
+          .setHiveStylePartitioningEnable(props.getBoolean(HIVE_STYLE_PARTITIONING_ENABLE.key(),

Review Comment:
   Does `urlEncodePartitioningEnabled` write to hoodie.properties or not? If not, I think we can add it as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] TengHuo commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552728285

   According the the code in HoodieSparkSqlWriter, there are few more config when building HoodieTableConfig.
   
   https://github.com/apache/hudi/blob/720c9d76c1d281b7a1305a205d195a4f179ce36a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#LL203C13-L203C13
   
   I think it's better we can align DeltaSync according to HoodieSparkSqlWriter.
   
   BTW: totally agree, we should use a unique entrypoint for this HoodieTableConfig builder. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #8752:
URL: https://github.com/apache/hudi/pull/8752#discussion_r1197514595


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -495,6 +498,8 @@ public Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> readFromSource(
           .setPartitionMetafileUseBaseFormat(props.getBoolean(HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.key(),
               HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.defaultValue()))
           .setShouldDropPartitionColumns(isDropPartitionColumns())
+          .setHiveStylePartitioningEnable(props.getBoolean(HIVE_STYLE_PARTITIONING_ENABLE.key(),

Review Comment:
   Nope, not written to `hoodie.properties`.
   
   
   ```properties
   #Properties saved on 2023-05-18T07:48:15.181Z
   #Thu May 18 15:48:15 SGT 2023
   hoodie.table.precombine.field=timestamp
   hoodie.datasource.write.drop.partition.columns=false
   hoodie.table.partition.fields=partition_path
   hoodie.table.type=COPY_ON_WRITE
   hoodie.archivelog.folder=archived
   hoodie.table.cdc.enabled=false
   hoodie.populate.meta.fields=true
   hoodie.partition.metafile.use.base.format=false
   hoodie.table.version=5
   hoodie.timeline.layout.version=1
   hoodie.table.base.file.format=PARQUET
   hoodie.table.recordkey.fields=_row_key
   hoodie.table.name=hoodie_trips
   hoodie.table.keygenerator.class=org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer$TestGenerator
   hoodie.table.checksum=224269229
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1553924948

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552690137

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552745846

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * 6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #8752:
URL: https://github.com/apache/hudi/pull/8752#discussion_r1197514595


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -495,6 +498,8 @@ public Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> readFromSource(
           .setPartitionMetafileUseBaseFormat(props.getBoolean(HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.key(),
               HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.defaultValue()))
           .setShouldDropPartitionColumns(isDropPartitionColumns())
+          .setHiveStylePartitioningEnable(props.getBoolean(HIVE_STYLE_PARTITIONING_ENABLE.key(),

Review Comment:
   Nope, not written to `hoodie.properties`. (The contents is how a `hoodie.properties` file will look like WITHOUT the the changes made in this PR.
   
   
   ```properties
   #Properties saved on 2023-05-18T07:48:15.181Z
   #Thu May 18 15:48:15 SGT 2023
   hoodie.table.precombine.field=timestamp
   hoodie.datasource.write.drop.partition.columns=false
   hoodie.table.partition.fields=partition_path
   hoodie.table.type=COPY_ON_WRITE
   hoodie.archivelog.folder=archived
   hoodie.table.cdc.enabled=false
   hoodie.populate.meta.fields=true
   hoodie.partition.metafile.use.base.format=false
   hoodie.table.version=5
   hoodie.timeline.layout.version=1
   hoodie.table.base.file.format=PARQUET
   hoodie.table.recordkey.fields=_row_key
   hoodie.table.name=hoodie_trips
   hoodie.table.keygenerator.class=org.apache.hudi.utilities.deltastreamer.TestHoodieDeltaStreamer$TestGenerator
   hoodie.table.checksum=224269229
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1553931378

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189",
       "triggerID" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17201",
       "triggerID" : "1553924948",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * f9b5f2d4727ffabc20e3e28e78e49009f6fa221e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17201) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552699962

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * 6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552793531

   > According the the code in HoodieSparkSqlWriter, there are few more config when building HoodieTableConfig.
   > 
   > https://github.com/apache/hudi/blob/720c9d76c1d281b7a1305a205d195a4f179ce36a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#LL203C13-L203C13
   > 
   > I think it's better we can align DeltaSync according to HoodieSparkSqlWriter.
   > 
   > BTW: totally agree, we should use a unique entrypoint for this HoodieTableConfig builder.
   
   Missing configs that are in `HoodieSparkSqlWriter.scala`, but not in `DeltaSync.java`.
   
   # In initializeEmptyTable()
   
   ```
   .setCommitTimezone()
   .setDatabaseName()
   .set(timestampKeyGeneratorConfigs)
   ```
   
   # In readFromSource()
   ```
   .setCDCEnabled()
   .setCDCSupplementalLoggingMode()
   .setCommitTimezone()
   .setDatabaseName()
   .setPreCombineField()
   .setRecordKeyFields()
   .set(timestampKeyGeneratorConfigs)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1553800662

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189",
       "triggerID" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * f9b5f2d4727ffabc20e3e28e78e49009f6fa221e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 merged pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8752:
URL: https://github.com/apache/hudi/pull/8752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552661379

   @boneanxs for visibility


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #8752:
URL: https://github.com/apache/hudi/pull/8752#discussion_r1197521391


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -495,6 +498,8 @@ public Pair<SchemaProvider, Pair<String, JavaRDD<HoodieRecord>>> readFromSource(
           .setPartitionMetafileUseBaseFormat(props.getBoolean(HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.key(),
               HoodieTableConfig.PARTITION_METAFILE_USE_BASE_FORMAT.defaultValue()))
           .setShouldDropPartitionColumns(isDropPartitionColumns())
+          .setHiveStylePartitioningEnable(props.getBoolean(HIVE_STYLE_PARTITIONING_ENABLE.key(),

Review Comment:
   @boneanxs Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1552860947

   Apologies again. 
   
   Fixed the checkstyle violations and re-executed a mvn clean install on my end.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8752:
URL: https://github.com/apache/hudi/pull/8752#issuecomment-1554323189

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7762747d22f8ffade79936aa3465db3ce89045db",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17183",
       "triggerID" : "6707bc8a2d23cb62a9dfefb5fed8ee3ea86445e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189",
       "triggerID" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f9b5f2d4727ffabc20e3e28e78e49009f6fa221e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17201",
       "triggerID" : "1553924948",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 7762747d22f8ffade79936aa3465db3ce89045db UNKNOWN
   * f9b5f2d4727ffabc20e3e28e78e49009f6fa221e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17189) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17201) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 closed pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed pull request #8752: [HUDI-6236] write hive_style_partitioning_enable to table config in D…
URL: https://github.com/apache/hudi/pull/8752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org