You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/09 15:09:51 UTC

[GitHub] [hudi] xushiyan opened a new pull request, #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

xushiyan opened a new pull request, #7171:
URL: https://github.com/apache/hudi/pull/7171

   ### Change Logs
   
   Make starting HDFS optional in `UtilitiesTestBase` and update subclasses' usage.
   
   ### Impact
   
   NA
   
   ### Risk level
   
   none
   
   ### Documentation Update
   
   NA
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7171:
URL: https://github.com/apache/hudi/pull/7171#issuecomment-1309459685

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bba82cddcfad42d2e2fe698d8a27077536257bfc",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12909",
       "triggerID" : "bba82cddcfad42d2e2fe698d8a27077536257bfc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a1766f041bf9b8bac2927b1aa076916361a00b6 UNKNOWN
   * bba82cddcfad42d2e2fe698d8a27077536257bfc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12909) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7171:
URL: https://github.com/apache/hudi/pull/7171#issuecomment-1309183756

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a1766f041bf9b8bac2927b1aa076916361a00b6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan merged pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #7171:
URL: https://github.com/apache/hudi/pull/7171


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan closed pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
xushiyan closed pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests
URL: https://github.com/apache/hudi/pull/7171


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7171:
URL: https://github.com/apache/hudi/pull/7171#issuecomment-1308960169

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a1766f041bf9b8bac2927b1aa076916361a00b6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on a diff in pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #7171:
URL: https://github.com/apache/hudi/pull/7171#discussion_r1018635258


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -250,7 +250,7 @@ static HoodieDeltaStreamer.Config makeConfig(String basePath, WriteOperationType
       cfg.operation = op;
       cfg.enableHiveSync = enableHiveSync;
       cfg.sourceOrderingField = sourceOrderingField;
-      cfg.propsFilePath = dfsBasePath + "/" + propsFilename;
+      cfg.propsFilePath = UtilitiesTestBase.basePath + "/" + propsFilename;

Review Comment:
   because there is another local var named the same. don't want to rename more variables further.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7171:
URL: https://github.com/apache/hudi/pull/7171#issuecomment-1309272029

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bba82cddcfad42d2e2fe698d8a27077536257bfc",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12909",
       "triggerID" : "bba82cddcfad42d2e2fe698d8a27077536257bfc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a1766f041bf9b8bac2927b1aa076916361a00b6 UNKNOWN
   * bba82cddcfad42d2e2fe698d8a27077536257bfc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12909) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7171:
URL: https://github.com/apache/hudi/pull/7171#issuecomment-1309266099

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a1766f041bf9b8bac2927b1aa076916361a00b6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bba82cddcfad42d2e2fe698d8a27077536257bfc",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bba82cddcfad42d2e2fe698d8a27077536257bfc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a1766f041bf9b8bac2927b1aa076916361a00b6 UNKNOWN
   * bba82cddcfad42d2e2fe698d8a27077536257bfc UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan commented on a diff in pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7171:
URL: https://github.com/apache/hudi/pull/7171#discussion_r1018623450


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -250,7 +250,7 @@ static HoodieDeltaStreamer.Config makeConfig(String basePath, WriteOperationType
       cfg.operation = op;
       cfg.enableHiveSync = enableHiveSync;
       cfg.sourceOrderingField = sourceOrderingField;
-      cfg.propsFilePath = dfsBasePath + "/" + propsFilename;
+      cfg.propsFilePath = UtilitiesTestBase.basePath + "/" + propsFilename;

Review Comment:
   basePath is protected in UtilitiesTestBase and TestHoodieDeltastreamer extends from it. so why prefixing with class name ? 



##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java:
##########
@@ -272,7 +272,7 @@ static HoodieDeltaStreamer.Config makeConfigForHudiIncrSrc(String srcBasePath, S
       cfg.sourceClassName = HoodieIncrSource.class.getName();
       cfg.operation = op;
       cfg.sourceOrderingField = "timestamp";
-      cfg.propsFilePath = dfsBasePath + "/test-downstream-source.properties";
+      cfg.propsFilePath = UtilitiesTestBase.basePath + "/test-downstream-source.properties";

Review Comment:
   same here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] xushiyan commented on pull request #7171: [HUDI-5145] Avoid starting HDFS in hudi-utilities tests

Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #7171:
URL: https://github.com/apache/hudi/pull/7171#issuecomment-1309749162

   The test run time has good improvement. 
   
   **in master version**
   
   UT 
   ```
   [INFO] hudi-utilities_2.11 ................................ SUCCESS [08:32 min]
   ```
   FT
   ```
   [INFO] hudi-utilities_2.11 ................................ SUCCESS [59:44 min]
   ```
   
   **With this patch**
   
   UT
   ```
   [INFO] hudi-utilities_2.11 ................................ SUCCESS [07:14 min]
   ```
   FT
   ```
   [INFO] hudi-utilities_2.11 ................................ SUCCESS [38:23 min]
   ```
   
   Overall it has about 30% drop!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org