You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/01 17:38:28 UTC

[GitHub] [incubator-hudi] WTa-hash commented on issue #1581: [SUPPORT] Hive Metastore not in sync with Hudi Dataset using DataSource API

WTa-hash commented on issue #1581:
URL: https://github.com/apache/incubator-hudi/issues/1581#issuecomment-622486217


   This issue is also present in Hudi-0.6.0-SNAPSHOT (as of 2020-MAY-01).
   
   Hudi DeltaStreamer (run-once mode) seems to update the Hudi metastore with no issues when processing new data changes.
   
   DeltaStreamer Configs:
   `val hudiOptions:HoodieDeltaStreamer.Config = new HoodieDeltaStreamer.Config();
   hudiOptions.tableType = hudiStorage;
   hudiOptions.sourceOrderingField = precombineKey;
   hudiOptions.sourceClassName = classOf[ParquetDFSSource].getName;
   hudiOptions.targetBasePath = targetFileLocation;
   hudiOptions.targetTableName = tableName;
   hudiOptions.transformerClassNames = Arrays.asList(classOf[AWSDmsTransformer].getName);
   hudiOptions.payloadClassName = classOf[AWSDmsAvroPayload].getName;
   hudiOptions.enableHiveSync = true;
   val addlHudiOptions:ArrayList[String] = new ArrayList[String]();
   addlHudiOptions.add("hoodie.datasource.write.recordkey.field=" + primaryKey);
   addlHudiOptions.add("hoodie.datasource.write.partitionpath.field=" + partitionKey);
   addlHudiOptions.add("hoodie.deltastreamer.source.dfs.root=" + sourceFileLocation);
   addlHudiOptions.add("hoodie.datasource.hive_sync.database=" + hiveDatabase);
   addlHudiOptions.add("hoodie.datasource.hive_sync.table=" + tableName);
   addlHudiOptions.add("hoodie.datasource.hive_sync.partition_fields=" + partitionKey);
   addlHudiOptions.add("hoodie.datasource.hive_sync.partition_extractor_class=" + classOf[MultiPartKeysValueExtractor].getName);
   hudiOptions.configs = addlHudiOptions;`
   
   Run DeltaStreamer
   `val deltaStreamer:HoodieDeltaStreamer = new HoodieDeltaStreamer(hudiOptions, new JavaSparkContext(sc));
   deltaStreamer.sync();
   `
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org