[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #4800: Pinot Ingestion Jobs Refactor with Pinot Spark Support

[GitBox <gi...@apache.org>]

mcvsubbu commented on a change in pull request #4800: Pinot Ingestion Jobs Refactor with Pinot Spark Support
URL: https://github.com/apache/incubator-pinot/pull/4800#discussion_r345930218
 
 

 ##########
 File path: pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
 ##########
 @@ -42,84 +38,31 @@
 import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
 import org.apache.pinot.common.config.SegmentsValidationAndRetentionConfig;
 import org.apache.pinot.common.config.TableConfig;
-import org.apache.pinot.common.data.Schema;
 import org.apache.pinot.common.utils.StringUtil;
 import org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper;
-import org.apache.pinot.hadoop.utils.JobPreparationHelper;
-import org.apache.pinot.hadoop.utils.PushLocation;
+import org.apache.pinot.hadoop.utils.PinotHadoopJobPreparationHelper;
+import org.apache.pinot.ingestion.common.JobConfigConstants;
+import org.apache.pinot.ingestion.jobs.SegmentCreationJob;
+import org.apache.pinot.ingestion.utils.JobPreparationHelper;
 
 
-public class SegmentCreationJob extends BaseSegmentJob {
-  protected static final String APPEND = "APPEND";
-
-  protected final Path _inputPattern;
-  protected final Path _outputDir;
-  protected final Path _stagingDir;
-  protected final String _rawTableName;
-
-  // Optional
-  protected final Path _depsJarDir;
-  protected final Path _schemaFile;
-  protected final String _defaultPermissionsMask;
-  protected final List<PushLocation> _pushLocations;
-
+public class HadoopSegmentCreationJob extends SegmentCreationJob {
   // Output Directory FileSystem
   protected FileSystem _outputDirFileSystem;
 
-  public SegmentCreationJob(Properties properties) {
+  public HadoopSegmentCreationJob(Properties properties) {
 
 Review comment:
   Renaming this class is a backwards-incoompatible. Open source community can extend (and indeed, we do extend in Linkedin) this class to create their custom hadoop jobs. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org