You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2024/05/20 02:12:11 UTC

[PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Zouxxyy opened a new pull request, #11260:
URL: https://github.com/apache/hudi/pull/11260

   ### Change Logs
   
   e.g.
   When the actual partition structure is `dt=2021-12-09/hh=1`, but mistakenly wrote `dt=2021-12-09`, the partition `dt=2021-12-09` will be written to the metadata of replacecommit.
   
   If we configure HMS partition synchronization at the same time, an error will always be reported:
   
   <img width="1425" alt="image" src="https://github.com/apache/hudi/assets/37108074/6a856771-598b-4e09-bff3-a365695e8528">
   
   ### Impact
   
   Filter wrong partitions when using `hoodie.datasource.write.partitions.to.delete`
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   
   None
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119587362

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996",
       "triggerID" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11ffaae39f099bed8905bb717419a9b72c9a91ea",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "11ffaae39f099bed8905bb717419a9b72c9a91ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8afd3edc4075f2cc7d24063475499627ce362380 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996) 
   * 11ffaae39f099bed8905bb717419a9b72c9a91ea UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119676130

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996",
       "triggerID" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11ffaae39f099bed8905bb717419a9b72c9a91ea",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23998",
       "triggerID" : "11ffaae39f099bed8905bb717419a9b72c9a91ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 11ffaae39f099bed8905bb717419a9b72c9a91ea Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23998) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119550892

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8afd3edc4075f2cc7d24063475499627ce362380 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119577233

   > Should we just supplement the secondary partition paths so that the whole `dt=2021-12-09` partiton can be deleted, this might be the original intention of the user.
   
   yes, can just config  `hoodie.datasource.write.partitions.to.delete` = `dt=2021-12-09/*`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119676691

   > I mean when user decpare the partition path as `dt=2021-12-09`, we handle it the same way as `dt=2021-12-09/*`.
   
   This looks good, but for example, such as third-level partitions, dt, hh, mm, if user configures `dt=1/mm=1` do we need to help the user configure as `dt=1/hh=*/mm=1`?
   
   I think simple is OK, user can either specify * or just full path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy closed pull request #11260: [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete
URL: https://github.com/apache/hudi/pull/11260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119898699

   > This looks good, but for example, such as third-level partitions, dt, hh, mm, if user configures `dt=1/mm=1` do we need to help the user configure as `dt=1/hh=*/mm=1`?
   
   I'm +1 on this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119610628

   > > Should we just supplement the secondary partition paths so that the whole `dt=2021-12-09` partiton can be deleted, this might be the original intention of the user.
   > 
   > yes, can just config `hoodie.datasource.write.partitions.to.delete` = `dt=2021-12-09/*`
   
   I mean when user decpare the partition path as `dt=2021-12-09`, we handle it the same way as `dt=2021-12-09/*`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119581897

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996",
       "triggerID" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8afd3edc4075f2cc7d24063475499627ce362380 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119564569

   > When the actual partition structure is dt=2021-12-09/hh=1, but mistakenly config dt=2021-12-09 in hoodie.datasource.write.partitions.to.delete, then, the partition dt=2021-12-09 will be written to the metadata of replacecommit.
   
   Should we just supplement the secondary partition paths so that the whole `dt=2021-12-09` partiton can be deleted, this might be the original intention of the user.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2119592157

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996",
       "triggerID" : "8afd3edc4075f2cc7d24063475499627ce362380",
       "triggerType" : "PUSH"
     }, {
       "hash" : "11ffaae39f099bed8905bb717419a9b72c9a91ea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23998",
       "triggerID" : "11ffaae39f099bed8905bb717419a9b72c9a91ea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8afd3edc4075f2cc7d24063475499627ce362380 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23996) 
   * 11ffaae39f099bed8905bb717419a9b72c9a91ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23998) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #11260:
URL: https://github.com/apache/hudi/pull/11260#issuecomment-2123793925

   @danny0405 When `hive_style_partitioning` is false, such as 2016/03/15, it is difficult to automatically add * to identify them.
   Besides I think that for this type of configuration that requires manual setting partitions, the full path and the * are sufficient, because there is no engine to parsing them, and this is just how it was used currently.
   If we want to use other more flexible solutions, can use SQL instead of manual configuration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-7781] Filter wrong partitions when using hoodie.datasource.write.partitions.to.delete [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #11260:
URL: https://github.com/apache/hudi/pull/11260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org