[GitHub] [samza] kw2542 opened a new pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 opened a new pull request #1428:
URL: https://github.com/apache/samza/pull/1428


   Issues: app.main.class is only set for Beam apps which causes different workflow on AM when launching a job.
   Changes:
   1. Introduce DefaultApplicationMain to capture launch workflow for High/Low level jobs so on AM, all jobs are launched in the same way: ClusterBasedJobCoordinatorRunner#main -> app.main.class -> JobCoordinatorLaunchUtil
   2. Update ApplicationConfig#getAppMainClass to default to DefaultApplicationMain
   
   Tests:
   1. Unit Tests
   2. Deployed hello samza job successfully with the change following instructions on http://samza.apache.org/startup/hello-samza/latest/
   
   API Changes: None
   Upgrade Instructions: None
   Usage Instructions: None


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat merged pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat merged pull request #1428:
URL: https://github.com/apache/samza/pull/1428






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r486518598



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Does it it mean you pass options that are dynamic and are not known static list? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] kw2542 commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487278983



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat merged pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat merged pull request #1428:
URL: https://github.com/apache/samza/pull/1428






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] kw2542 commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r486503376



##########
File path: samza-core/src/main/java/org/apache/samza/config/ApplicationConfig.java
##########
@@ -104,8 +105,8 @@ public ApplicationMode getAppMode() {
     return Optional.ofNullable(get(APP_MAIN_ARGS));

Review comment:
       This particular method is not being used as the current toArgs() method is checking the key name directly. We can remove this method. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487284863



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r486433394



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       why do we need this?

##########
File path: samza-core/src/main/java/org/apache/samza/config/ApplicationConfig.java
##########
@@ -104,8 +105,8 @@ public ApplicationMode getAppMode() {
     return Optional.ofNullable(get(APP_MAIN_ARGS));

Review comment:
       Looks like this is unused. Do you where this is used/will be used?
   Now that main class is no longer optional, wondering if this should be too?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] kw2542 commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487278983



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] kw2542 commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487278983



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   

##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2. Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] kw2542 commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r486502251



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       This is needed because submission configs are now passed to this class through arguments, in order for the main class to load full config, we need to parse the arguments. This is similar to Beam application where arguments are being parsed in the main class as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat merged pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat merged pull request #1428:
URL: https://github.com/apache/samza/pull/1428


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487284863



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] kw2542 commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

kw2542 commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487278983



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       To answer your original questions, submission configs is a static list of configs predefined as they will impact how AM works, plus any extra config overrides user wants to supply in order to override the configs fetched in the `ConfigLoader`. 
   
   The complete workflow will be:
   
   1. submission configs are first passed to ClusterBasedJobCoordinatorRunner as System env variable and is retrieved with
   `final String submissionEnv = System.getenv(ShellCommandConfig.ENV_SUBMISSION_CONFIG);`
   
   2.Afterwards, when ClusterBasedJobCoordinatorRunner invokes app.main.class, the above submission configs will be converted to `String[] args` as this is the only parameter that main() method takes.
   
   3. Inside main() method, including both Beam and High/Low, args will be used to load full job config with `ConfigLoaderFactory`
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r487284863



##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/DefaultApplicationMain.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.samza.clustermanager;
+
+import com.google.common.annotations.VisibleForTesting;
+import joptsimple.OptionSet;
+import org.apache.samza.application.ApplicationUtil;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunnerMain;
+import org.apache.samza.util.ConfigUtil;
+
+
+public class DefaultApplicationMain {
+
+  public static void main(String[] args) {
+    run(args);
+  }
+
+  @VisibleForTesting
+  static void run(String[] args) {
+    // This branch is ONLY for Yarn deployments, standalone apps uses offspring
+    final ApplicationRunnerMain.ApplicationRunnerCommandLine cmdLine = new ApplicationRunnerMain.ApplicationRunnerCommandLine();
+    cmdLine.parser().allowsUnrecognizedOptions();

Review comment:
       Thanks for the explanation 👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [samza] mynameborat commented on a change in pull request #1428: SAMZA-2589: Consolidate Beam and High/Low Samza Apps launch workflow

[GitBox <gi...@apache.org>]

mynameborat commented on a change in pull request #1428:
URL: https://github.com/apache/samza/pull/1428#discussion_r486518731



##########
File path: samza-core/src/main/java/org/apache/samza/config/ApplicationConfig.java
##########
@@ -104,8 +105,8 @@ public ApplicationMode getAppMode() {
     return Optional.ofNullable(get(APP_MAIN_ARGS));

Review comment:
       sounds good to me. If you can remove it, that will be great.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org