[jira] [Created] (HUDI-3640) Use default key generator class if not provided by user in TwoToThreeUpgradeHandler

["Ethan Guo (Jira)" <ji...@apache.org>]

Ethan Guo created HUDI-3640:
-------------------------------

             Summary: Use default key generator class if not provided by user in TwoToThreeUpgradeHandler
                 Key: HUDI-3640
                 URL: https://issues.apache.org/jira/browse/HUDI-3640
             Project: Apache Hudi
          Issue Type: Task
            Reporter: Ethan Guo
            Assignee: Ethan Guo
             Fix For: 0.11.0


When upgrading from table version 2 to 3, the logic checks if the key generator class is provided by user config.  We should still use default SimpleKeyGenerator if not provided, instead of throwing an error in the upgrade step.
{code:java}
ValidationUtils.checkState(keyGenClassName != null, String.format("Missing config: %s or %s",
        HoodieTableConfig.KEY_GENERATOR_CLASS_NAME, HoodieWriteConfig.KEYGENERATOR_CLASS_NAME)); {code}
 

 
{code:java}
22/03/14 12:28:10 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to exception
java.lang.IllegalStateException: Missing config: Key: 'hoodie.table.keygenerator.class' , default: null description: Key Generator class property for the hoodie table since version: version is not defined deprecated after: version is not defined) or Key: 'hoodie.datasource.write.keygenerator.class' , default: null description: Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. since version: version is not defined deprecated after: version is not defined)
    at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:67)
    at org.apache.hudi.table.upgrade.TwoToThreeUpgradeHandler.upgrade(TwoToThreeUpgradeHandler.java:50)
    at org.apache.hudi.table.upgrade.UpgradeDowngrade.upgrade(UpgradeDowngrade.java:145)
    at org.apache.hudi.table.upgrade.UpgradeDowngrade.run(UpgradeDowngrade.java:118)
    at org.apache.hudi.client.BaseHoodieWriteClient.tryUpgrade(BaseHoodieWriteClient.java:1407)
    at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1302)
    at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:151)
    at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:577)
    at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:656)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
22/03/14 12:28:10 INFO HoodieDeltaStreamer: Delta Sync shutdown. Error ?true
22/03/14 12:28:10 WARN HoodieDeltaStreamer: Gracefully shutting down compactor
22/03/14 12:28:16 INFO AsyncCompactService: Compactor shutting down properly!!
22/03/14 12:28:16 INFO HoodieDeltaStreamer: DeltaSync shutdown. Closing write client. Error?true
22/03/14 12:28:16 ERROR HoodieAsyncService: Service shutdown with error
java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: Missing config: Key: 'hoodie.table.keygenerator.class' , default: null description: Key Generator class property for the hoodie table since version: version is not defined deprecated after: version is not defined) or Key: 'hoodie.datasource.write.keygenerator.class' , default: null description: Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` extract a key out of incoming records. since version: version is not defined deprecated after: version is not defined)
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
    at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:103)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:182)
    at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:179)
    at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:530)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)