You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/30 22:06:05 UTC

[GitHub] [hudi] nsivabalan opened a new pull request, #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

nsivabalan opened a new pull request, #6546:
URL: https://github.com/apache/hudi/pull/6546

   ### Change Logs
   
   Fixed flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat. When generating data, test wasn't generating data to all partitions. during validation we validate that partition meta file exists for one of the 3 partitions that HoodieTestDataGenerator generates. 
   
   ### Impact
   
   will fix CI stability
   
   **Risk level: low**
   
   _Choose one. If medium or high, explain what verification was done to mitigate the risks._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan merged pull request #6546: [HUDI-4438] Fix flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #6546:
URL: https://github.com/apache/hudi/pull/6546


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6546:
URL: https://github.com/apache/hudi/pull/6546#issuecomment-1232245948

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7f3810447e0e279491030e4fe66453af789bb1a8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11062",
       "triggerID" : "7f3810447e0e279491030e4fe66453af789bb1a8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7f3810447e0e279491030e4fe66453af789bb1a8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11062) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on a diff in pull request #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6546:
URL: https://github.com/apache/hudi/pull/6546#discussion_r959004054


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/commit/TestCopyOnWriteActionExecutor.java:
##########
@@ -532,7 +532,7 @@ public void testPartitionMetafileFormat(boolean partitionMetafileUseBaseFormat)
     writeClient.startCommitWithTime(instantTime);
 
     // Insert new records
-    final JavaRDD<HoodieRecord> inputRecords = generateTestRecordsForBulkInsert(jsc, 10);
+    final JavaRDD<HoodieRecord> inputRecords = generateTestRecordsForBulkInsert(jsc, 50);
     writeClient.bulkInsert(inputRecords, instantTime);

Review Comment:
   ok would changing to 50 guarantee records generated for the required partitions? are we able to specify partitions to generate records?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a diff in pull request #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6546:
URL: https://github.com/apache/hudi/pull/6546#discussion_r959107402


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/commit/TestCopyOnWriteActionExecutor.java:
##########
@@ -532,7 +532,7 @@ public void testPartitionMetafileFormat(boolean partitionMetafileUseBaseFormat)
     writeClient.startCommitWithTime(instantTime);
 
     // Insert new records
-    final JavaRDD<HoodieRecord> inputRecords = generateTestRecordsForBulkInsert(jsc, 10);
+    final JavaRDD<HoodieRecord> inputRecords = generateTestRecordsForBulkInsert(jsc, 50);
     writeClient.bulkInsert(inputRecords, instantTime);

Review Comment:
   after my fix, I ran it for 200 times locally and could not reproduce the flakiness.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on pull request #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6546:
URL: https://github.com/apache/hudi/pull/6546#issuecomment-1232442582

   ![Screen Shot 2022-08-30 at 11 18 28 PM](https://user-images.githubusercontent.com/2701446/187591825-77d0897c-cd7f-4cad-ba75-c66aff4f0ca1.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a diff in pull request #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6546:
URL: https://github.com/apache/hudi/pull/6546#discussion_r959106750


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/commit/TestCopyOnWriteActionExecutor.java:
##########
@@ -532,7 +532,7 @@ public void testPartitionMetafileFormat(boolean partitionMetafileUseBaseFormat)
     writeClient.startCommitWithTime(instantTime);
 
     // Insert new records
-    final JavaRDD<HoodieRecord> inputRecords = generateTestRecordsForBulkInsert(jsc, 10);
+    final JavaRDD<HoodieRecord> inputRecords = generateTestRecordsForBulkInsert(jsc, 50);
     writeClient.bulkInsert(inputRecords, instantTime);

Review Comment:
   already we instantiate w/ 3 partitions. in all our tests we generate 50 to 100 records and it generally works out. just that, with 10 records, 1 in 25 to 30 runs, sometimes not all 3 partitions gets the record. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6546: [HUDI-4438] Fixing flaky TestCopyOnWriteActionExecutor#testPartitionMetafileFormat

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6546:
URL: https://github.com/apache/hudi/pull/6546#issuecomment-1232242626

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "7f3810447e0e279491030e4fe66453af789bb1a8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7f3810447e0e279491030e4fe66453af789bb1a8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7f3810447e0e279491030e4fe66453af789bb1a8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org