You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2025/02/13 10:41:03 UTC
[PR] Fix the HoodieCompactor schedule parameter alias name [hudi]
rangareddy opened a new pull request, #12836:
URL: https://github.com/apache/hudi/pull/12836
### Change Logs
Fixed the **schedule** alias name from **sc** to **sch**.
**Before fixing the issue:**
```sh
% spark-submit \
--class org.apache.hudi.utilities.HoodieCompactor \
packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
--base-path /tmp/hudi/hudi-test-topic \
--table-name hudi-test-topic
```
```sh
25/02/13 15:33:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" org.apache.hudi.com.beust.jcommander.ParameterException: Found the option -sc multiple times
at org.apache.hudi.com.beust.jcommander.JCommander.addDescription(JCommander.java:627)
at org.apache.hudi.com.beust.jcommander.JCommander.createDescriptions(JCommander.java:594)
at org.apache.hudi.com.beust.jcommander.JCommander.<init>(JCommander.java:249)
at org.apache.hudi.utilities.HoodieCompactor.main(HoodieCompactor.java:173)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
25/02/13 15:33:59 INFO ShutdownHookManager: Shutdown hook called
```
**After fixing the issue:**
```sh
% spark-submit \
--class org.apache.hudi.utilities.HoodieCompactor \
packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-1.1.0-SNAPSHOT.jar \
--base-path /tmp/hudi/hudi-test-topic \
--table-name hudi-test-topic
```
```sh
.....
25/02/13 15:56:52 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /tmp/hudi/hudi-test-topic
25/02/13 15:56:52 INFO HoodieTableConfig: Loading table properties from /tmp/hudi/hudi-test-topic/.hoodie/hoodie.properties
25/02/13 15:56:52 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1) from /tmp/hudi/hudi-test-topic
25/02/13 15:56:52 INFO HoodieTableMetaClient: Loading Active commit timeline for /tmp/hudi/hudi-test-topic
25/02/13 15:56:52 INFO ActiveTimelineV1: Loaded instants upto : Option{val=[20250213153114368__20250213153136000__commit__COMPLETED]}
25/02/13 15:56:52 INFO HoodieCompactor: HoodieCompactorConfig {
--base-path /tmp/hudi/hudi-test-topic,
--table-name hudi-test-topic,
--instant-time null,
--parallelism 200,
--schema-file null,
--spark-master null,
--spark-memory null,
--skipClean true,
--retry 0,
--schedule false,
--mode execute,
--strategy org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy,
--props null,
--hoodie-conf []
}
25/02/13 15:56:52 INFO HoodieCompactor: Running Mode: [execute]; Do compaction
25/02/13 15:56:52 INFO HoodieCompactor: Schema --> : {"type":"record","name":"hudi__test__topic_record","namespace":"hoodie.hudi__test__topic","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"field1","type":["null","string"],"default":null},{"name":"field2","type":["null","string"],"default":null},{"name":"field3","type":["null","string"],"default":null},{"name":"field4","type":"boolean"},{"name":"field5","type":"long"},{"name":"ts","type":"long"},{"name":"partition","type":["null","string"],"default":null},{"name":"fruits","type":["null","string"],"default":null}]}
....
25/02/13 15:56:53 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /tmp/hudi/hudi-test-topic
25/02/13 15:56:53 INFO HoodieTableConfig: Loading table properties from /tmp/hudi/hudi-test-topic/.hoodie/hoodie.properties
25/02/13 15:56:53 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1) from /tmp/hudi/hudi-test-topic
25/02/13 15:56:53 INFO HoodieTableMetaClient: Loading Active commit timeline for /tmp/hudi/hudi-test-topic
25/02/13 15:56:53 INFO ActiveTimelineV1: Loaded instants upto : Option{val=[20250213153114368__20250213153136000__commit__COMPLETED]}
25/02/13 15:56:53 INFO HoodieCompactor: There is no scheduled compaction in the table.
25/02/13 15:56:53 INFO BaseHoodieClient: Stopping Timeline service !!
25/02/13 15:56:53 INFO EmbeddedTimelineService: Closing Timeline server
25/02/13 15:56:53 INFO TimelineService: Closing Timeline Service
25/02/13 15:56:53 INFO Javalin: Stopping Javalin ...
25/02/13 15:56:53 INFO Javalin: Javalin has stopped
25/02/13 15:56:53 INFO TimelineService: Closed Timeline Service
25/02/13 15:56:53 INFO EmbeddedTimelineService: Closed Timeline server
25/02/13 15:56:53 INFO TransactionManager: Transaction manager closed
....
25/02/13 15:56:53 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
25/02/13 15:56:53 INFO SparkContext: Successfully stopped SparkContext
25/02/13 15:56:58 INFO ShutdownHookManager: Shutdown hook called
```
### Impact
Corrected the **schedule** alias name.
### Risk level (write none, low medium or high below)
none
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9016] Fix the HoodieCompactor schedule parameter alias name [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun merged PR #12836:
URL: https://github.com/apache/hudi/pull/12836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the HoodieCompactor schedule parameter alias name [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12836:
URL: https://github.com/apache/hudi/pull/12836#issuecomment-2656193791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e6d947d3fccf3d4310f69c0ec43fac063c6f22f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the HoodieCompactor schedule parameter alias name [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12836:
URL: https://github.com/apache/hudi/pull/12836#issuecomment-2656197481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612",
"triggerID" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e6d947d3fccf3d4310f69c0ec43fac063c6f22f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix the HoodieCompactor schedule parameter alias name [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12836:
URL: https://github.com/apache/hudi/pull/12836#issuecomment-2656455444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612",
"triggerID" : "5e6d947d3fccf3d4310f69c0ec43fac063c6f22f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e6d947d3fccf3d4310f69c0ec43fac063c6f22f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3612)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org