[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1566: [HUDI-603]: DeltaStreamer can now fetch schema before every run in continuous mode

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #1566:
URL: https://github.com/apache/incubator-hudi/pull/1566#discussion_r416319065



##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -162,18 +162,23 @@ public DeltaSync(HoodieDeltaStreamer.Config cfg, SparkSession sparkSession, Sche
     this.fs = fs;
     this.onInitializingHoodieWriteClient = onInitializingHoodieWriteClient;
     this.props = props;
-    this.schemaProvider = schemaProvider;
 
     refreshTimeline();
-
     this.transformer = UtilHelpers.createTransformer(cfg.transformerClassNames);
     this.keyGenerator = DataSourceUtils.createKeyGenerator(props);
-
-    this.formatAdapter = new SourceFormatAdapter(
-        UtilHelpers.createSource(cfg.sourceClassName, props, jssc, sparkSession, schemaProvider));
-
     this.conf = conf;
+    refreshSchemaProvider(schemaProvider);
+  }
 
+  /**
+   * Very useful when DeltaStreamer is running in continuous mode.
+   * @param schemaProvider
+   * @throws IOException
+   */
+  public void refreshSchemaProvider(SchemaProvider schemaProvider) throws IOException {

Review comment:
       right.. embedded server re-use is important. let's scratch my idea. 
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org