You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:29:03 UTC
[I] Create "Type" configs for current configs that take in classes [hudi]
hudi-bot opened a new issue, #15780:
URL: https://github.com/apache/hudi/issues/15780
Take for example `hoodie.datasource.write.keygenerator.type` and its counterpart `hoodie.datasource.write.keygenerator.class`. We should make configs like keygenerator.type for configs currently like keygenerator.class. Take for example `hoodie.datasource.write.record.merger.impls`. If you want HoodieAvroRecordMerger you need to set the value to org.apache.hudi.common.model.HoodieAvroRecordMerger. However, if you want HoodieSparkRecordMerger, the full class path is org.apache.hudi.HoodieSparkRecordMerger. Differences like this are an easy cause for mistakes.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5761
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5738
- Fix version(s):
- 1.1.0
---
## Comments
03/Apr/23 13:35;zxcoccer;Hi, I am trying to complete this issue, but there is some confusion in the process of completing the issue, how to get the `*_type` value elegantly, the method I can think of now is to correspond to a method for each `_type` Value, like this
{code:java}
switch (keyGeneratorTypeEnum){
case SIMPLE: return new SimpleAvroKeyGenerator(props);
case COMPLEX: return new ComplexAvroKeyGenerator(props);
case TIMESTAMP: return new TimestampBasedAvroKeyGenerator(props);
case NON_PARTITION: return new NonpartitionedAvroKeyGenerator(props);
case GLOBAL_DELETE: return new GlobalAvroDeleteKeyGenerator(props);
}
{code}
By the way, can you assign this ticket to me? I will try my best to complete it;;;
---
03/Apr/23 15:39;jonvex;Ok so keygenerator.type is now not such a good example because https://issues.apache.org/jira/browse/HUDI-5929 . So I will give a different example. In HoodieBootstrapConfig.java we have config "hoodie.bootstrap.mode.selector". We have selectors that are already created:
org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector
org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector
org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector
But you need to read the code to figure that out and it is long to type out. We want to create a new enum
{code:java}
public enum BootstrapSelectorType {
REGEX,
FULL_RECORD,
METADATA_ONLY,
CUSTOM
}
{code}
And create a new config "hoodie.bootstrap.mode.type" that takes in those config values
Then, in HoodieWriteConfig.java there is a method
{code:java}
public String getBootstrapModeSelectorClass() {
return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME);
} {code}
We want to replace it with something like
{code:java}
public String getBootstrapModeSelectorClass() {
switch (getString(HoodieBootstrapConfig.MODE_SELECTOR_TYPE_NAME)) {
case REGEX: return "org.apache.hudi.client.bootstrap.selector.BootstrapRegexModeSelector";
case FULL_RECORD: return "org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector";
case METADATA_ONLY: return "
org.apache.hudi.client.bootstrap.selector.FullRecordBootstrapModeSelector";
case CUSTOM: return getString(HoodieBootstrapConfig.MODE_SELECTOR_CLASS_NAME);
}
}
{code}
Then repeat this with all other configs that take in class:
hoodie.datasource.write.payload.class
hoodie.datasource.write.record.merger.impls
hoodie.writestatus.class
hoodie.fileid.prefix.provider.class
for example. There are probably more than that.
To get assigned tickets, you need to email [dev@hudi.apache.org|mailto:dev@hudi.apache.org] and ask to be added as a contributor.
;;;
---
05/Apr/23 11:38;zxcoccer;Thank you very much for your reply. I will complete this task soon.:);;;
---
24/Apr/23 18:27;jonvex;Taking this over because there is a lot of documentation required;;;
---
24/Apr/23 18:40;jonvex;Additional classes to look at:
h3. -hoodie.datasource.hive_sync.partition_extractor_class-
h3. -hoodie.meta.sync.client.tool.class-
h3. hoodie.precommit.validators
h3. hoodie.filesystem.operation.retry.exceptions
h3. hoodie.bulkinsert.user.defined.partitioner.class
h3. hoodie.write.concurrency.early.conflict.detection.strategy
h3. hoodie.compaction.strategy
h3. hoodie.clustering.execution.strategy.class
h3. hoodie.clustering.plan.strategy.class
h3. hoodie.clustering.updates.strategy
h3. hoodie.bootstrap.full.input.provider
h3. hoodie.bootstrap.index.class
h3. hoodie.bootstrap.partitionpath.translator.class
h4. hoodie.write.commit.callback.class
h4. hoodie.write.lock.conflict.resolution.strategy
h4. hoodie.write.lock.provider
h4. hoodie.index.hbase.qps.allocator.class
h3. hoodie.meta.sync.datahub.dataset.identifier.class
h3. hoodie.schemaprovider.class
h3. hoodie.filesystem.view.remote.retry.exceptions[|https://hudi.apache.org/docs/configurations#hoodiefilesystemviewremoteretryexceptions]
;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create "Type" configs for current configs that take in classes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15780:
URL: https://github.com/apache/hudi/issues/15780#issuecomment-3630157833
### Linked PR(s)
- https://github.com/apache/hudi/pull/8566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org