You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2025/02/06 08:01:30 UTC

[PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

lokeshj1703 opened a new pull request, #12792:
URL: https://github.com/apache/hudi/pull/12792

   ### Change Logs
   
   The PR adds a config which fails the entire job if an invaild file is detected. The config is disabled by default.
   
   ### Impact
   
   NA
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   
   NA
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2642979626

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419",
       "triggerID" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "879d2968dd5d60b8637aeea937bf481bfa1235fc",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "879d2968dd5d60b8637aeea937bf481bfa1235fc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e28163512176e15032ebd92fecdd0633c4bcf4f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419) 
   * 879d2968dd5d60b8637aeea937bf481bfa1235fc UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2639123713

   @nsivabalan Can you please take a look?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2639099366

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419",
       "triggerID" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e28163512176e15032ebd92fecdd0633c4bcf4f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2639332895

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419",
       "triggerID" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e28163512176e15032ebd92fecdd0633c4bcf4f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #12792:
URL: https://github.com/apache/hudi/pull/12792


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2643185125

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419",
       "triggerID" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "879d2968dd5d60b8637aeea937bf481bfa1235fc",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3473",
       "triggerID" : "879d2968dd5d60b8637aeea937bf481bfa1235fc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 879d2968dd5d60b8637aeea937bf481bfa1235fc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3473) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2642983856

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419",
       "triggerID" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "879d2968dd5d60b8637aeea937bf481bfa1235fc",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3473",
       "triggerID" : "879d2968dd5d60b8637aeea937bf481bfa1235fc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e28163512176e15032ebd92fecdd0633c4bcf4f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3419) 
   * 879d2968dd5d60b8637aeea937bf481bfa1235fc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3473) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2639096368

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e28163512176e15032ebd92fecdd0633c4bcf4f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e28163512176e15032ebd92fecdd0633c4bcf4f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8967] Add a config to fail job if duplicate data files detected during reconcileAgainstMarkers [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12792:
URL: https://github.com/apache/hudi/pull/12792#issuecomment-2639618686

   @lokeshj1703 Can you add some context in which scenario the user needs to enable it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org