You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "1032851561 (via GitHub)" <gi...@apache.org> on 2023/03/15 09:27:31 UTC

[GitHub] [hudi] 1032851561 opened a new pull request, #8194: [HUDI-5937] Archived log file never rollover

1032851561 opened a new pull request, #8194:
URL: https://github.com/apache/hudi/pull/8194

   ### Change Logs
   
   Archived log file never rollover. `HoodieLogFormatWriter` is always closed after archive , so `HoodieLogFormatWriter#output.getPos()`  will get 0 and currentSize can never reach the sizeThreshold(default 512M).
   
   ### env
   
   - hudi 0.11.1
   - flink 1.14.5
   - hdfs 2.10.2
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8194:
URL: https://github.com/apache/hudi/pull/8194#issuecomment-1471015862

   @1032851561 : is it possible to write a test. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a diff in pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8194:
URL: https://github.com/apache/hudi/pull/8194#discussion_r1137911092


##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java:
##########
@@ -216,9 +216,10 @@ private int getLogBlockLength(int contentLength, int headerLength, int footerLen
 
   private void rolloverIfNeeded() throws IOException {
     // Roll over if the size is past the threshold
-    if (getCurrentSize() > sizeThreshold) {
-      LOG.info("CurrentSize " + getCurrentSize() + " has reached threshold " + sizeThreshold
-          + ". Rolling over to the next version");
+    long currentSize = fs.getFileStatus(logFile.getPath()).getLen();
+    if (currentSize > sizeThreshold) {

Review Comment:
   @prashantwason @suryaprasanna : Can you guys review this patch. surprised that rolliover isn't working. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8194:
URL: https://github.com/apache/hudi/pull/8194#issuecomment-1469933876

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "76c0a05dd43fa881874853feeeccf73b1844a9c6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15732",
       "triggerID" : "76c0a05dd43fa881874853feeeccf73b1844a9c6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 76c0a05dd43fa881874853feeeccf73b1844a9c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15732) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] 1032851561 closed pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "1032851561 (via GitHub)" <gi...@apache.org>.
1032851561 closed pull request #8194: [HUDI-5937] Archived log file never rollover 
URL: https://github.com/apache/hudi/pull/8194


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8194:
URL: https://github.com/apache/hudi/pull/8194#issuecomment-1469662525

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "76c0a05dd43fa881874853feeeccf73b1844a9c6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "76c0a05dd43fa881874853feeeccf73b1844a9c6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 76c0a05dd43fa881874853feeeccf73b1844a9c6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8194:
URL: https://github.com/apache/hudi/pull/8194#issuecomment-1469673462

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "76c0a05dd43fa881874853feeeccf73b1844a9c6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15732",
       "triggerID" : "76c0a05dd43fa881874853feeeccf73b1844a9c6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 76c0a05dd43fa881874853feeeccf73b1844a9c6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15732) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] 1032851561 commented on a diff in pull request #8194: [HUDI-5937] Archived log file never rollover

Posted by "1032851561 (via GitHub)" <gi...@apache.org>.
1032851561 commented on code in PR #8194:
URL: https://github.com/apache/hudi/pull/8194#discussion_r1138123092


##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java:
##########
@@ -216,9 +216,10 @@ private int getLogBlockLength(int contentLength, int headerLength, int footerLen
 
   private void rolloverIfNeeded() throws IOException {
     // Roll over if the size is past the threshold
-    if (getCurrentSize() > sizeThreshold) {
-      LOG.info("CurrentSize " + getCurrentSize() + " has reached threshold " + sizeThreshold
-          + ". Rolling over to the next version");
+    long currentSize = fs.getFileStatus(logFile.getPath()).getLen();
+    if (currentSize > sizeThreshold) {

Review Comment:
   Sorry, I found it has been fixed in version 0.12.1 or above by [HUDI-4780](https://issues.apache.org/jira/browse/HUDI-4780). 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org