You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2025/02/13 10:41:03 UTC

[PR] Fix the HoodieCompactor schedule parameter alias name [hudi]

rangareddy opened a new pull request, #12836:
URL: https://github.com/apache/hudi/pull/12836

   ### Change Logs
   
   Fixed the **schedule** alias name from **sc** to **sch**.
   
   **Before fixing the issue:**
   
   ```sh
   % spark-submit \
     --class org.apache.hudi.utilities.HoodieCompactor \
     packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
     --base-path /tmp/hudi/hudi-test-topic \
     --table-name hudi-test-topic
   ```
   ```sh
   25/02/13 15:33:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
   Exception in thread "main" org.apache.hudi.com.beust.jcommander.ParameterException: Found the option -sc multiple times
     at org.apache.hudi.com.beust.jcommander.JCommander.addDescription(JCommander.java:627)
     at org.apache.hudi.com.beust.jcommander.JCommander.createDescriptions(JCommander.java:594)
     at org.apache.hudi.com.beust.jcommander.JCommander.<init>(JCommander.java:249)
     at org.apache.hudi.utilities.HoodieCompactor.main(HoodieCompactor.java:173)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:498)
     at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
     at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
     at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
     at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
     at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
     at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   25/02/13 15:33:59 INFO ShutdownHookManager: Shutdown hook called
   ```
   
   **After fixing the issue:**
   
   ```sh
   % spark-submit \                
     --class org.apache.hudi.utilities.HoodieCompactor \
     packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
     --base-path /tmp/hudi/hudi-test-topic \
     --table-name hudi-test-topic
   ```
   ```sh
   .....
   25/02/13 15:56:52 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /tmp/hudi/hudi-test-topic
   25/02/13 15:56:52 INFO HoodieTableConfig: Loading table properties from /tmp/hudi/hudi-test-topic/.hoodie/hoodie.properties
   25/02/13 15:56:52 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1) from /tmp/hudi/hudi-test-topic
   25/02/13 15:56:52 INFO HoodieTableMetaClient: Loading Active commit timeline for /tmp/hudi/hudi-test-topic
   25/02/13 15:56:52 INFO ActiveTimelineV1: Loaded instants upto : Option{val=[20250213153114368__20250213153136000__commit__COMPLETED]}
   25/02/13 15:56:52 INFO HoodieCompactor: HoodieCompactorConfig {
      --base-path /tmp/hudi/hudi-test-topic, 
      --table-name hudi-test-topic, 
      --instant-time null, 
      --parallelism 200, 
      --schema-file null, 
      --spark-master null, 
      --spark-memory null, 
      --skipClean true, 
      --retry 0, 
      --schedule false, 
      --mode execute, 
      --strategy org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy, 
      --props null, 
      --hoodie-conf []
   }
   25/02/13 15:56:52 INFO HoodieCompactor: Running Mode: [execute]; Do compaction
   25/02/13 15:56:52 INFO HoodieCompactor: Schema --> : {"type":"record","name":"hudi__test__topic_record","namespace":"hoodie.hudi__test__topic","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"field1","type":["null","string"],"default":null},{"name":"field2","type":["null","string"],"default":null},{"name":"field3","type":["null","string"],"default":null},{"name":"field4","type":"boolean"},{"name":"field5","type":"long"},{"name":"ts","type":"long"},{"name":"partition","type":["null","string"],"default":null},{"name":"fruits","type":["null","string"],"default":null}]}
   ....
   25/02/13 15:56:53 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /tmp/hudi/hudi-test-topic
   25/02/13 15:56:53 INFO HoodieTableConfig: Loading table properties from /tmp/hudi/hudi-test-topic/.hoodie/hoodie.properties
   25/02/13 15:56:53 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1) from /tmp/hudi/hudi-test-topic
   25/02/13 15:56:53 INFO HoodieTableMetaClient: Loading Active commit timeline for /tmp/hudi/hudi-test-topic
   25/02/13 15:56:53 INFO ActiveTimelineV1: Loaded instants upto : Option{val=[20250213153114368__20250213153136000__commit__COMPLETED]}
   25/02/13 15:56:53 INFO HoodieCompactor: There is no scheduled compaction in the table.
   25/02/13 15:56:53 INFO BaseHoodieClient: Stopping Timeline service !!
   25/02/13 15:56:53 INFO EmbeddedTimelineService: Closing Timeline server
   25/02/13 15:56:53 INFO TimelineService: Closing Timeline Service
   25/02/13 15:56:53 INFO Javalin: Stopping Javalin ...
   25/02/13 15:56:53 INFO Javalin: Javalin has stopped
   25/02/13 15:56:53 INFO TimelineService: Closed Timeline Service
   25/02/13 15:56:53 INFO EmbeddedTimelineService: Closed Timeline server
   25/02/13 15:56:53 INFO TransactionManager: Transaction manager closed
   ....
   25/02/13 15:56:53 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
   25/02/13 15:56:53 INFO SparkContext: Successfully stopped SparkContext
   25/02/13 15:56:58 INFO ShutdownHookManager: Shutdown hook called
   ```
   
   ### Impact
   
   Corrected the **schedule** alias name.
   
   ### Risk level (write none, low medium or high below)
   
   none
   
   ### Documentation Update
   
   NA
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9016] Fix the HoodieCompactor schedule parameter alias name [hudi]

Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun merged PR #12836:
URL: https://github.com/apache/hudi/pull/12836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix the HoodieCompactor schedule parameter alias name [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12836:
URL: https://github.com/apache/hudi/pull/12836#issuecomment-2656193791

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5e6d947d3fccf3d4310f69c0ec43fac063c6f22f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix the HoodieCompactor schedule parameter alias name [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12836:
URL: https://github.com/apache/hudi/pull/12836#issuecomment-2656197481

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612",
       "triggerID" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5e6d947d3fccf3d4310f69c0ec43fac063c6f22f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix the HoodieCompactor schedule parameter alias name [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12836:
URL: https://github.com/apache/hudi/pull/12836#issuecomment-2656455444

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612",
       "triggerID" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5e6d947d3fccf3d4310f69c0ec43fac063c6f22f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org