You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "pratapaditya04 (via GitHub)" <gi...@apache.org> on 2025/04/02 18:02:22 UTC

[PR] Azkaban id deprecation work [gobblin]

pratapaditya04 opened a new pull request, #4108:
URL: https://github.com/apache/gobblin/pull/4108

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-XXX
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2048311774


##########
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java:
##########
@@ -385,7 +385,11 @@ private boolean isCurrentTimeInRange() {
    */
   private static List<? extends Tag<?>> addAdditionalMetadataTags(Properties jobProps) {
     List<Tag<?>> metadataTags = Lists.newArrayList();
-    String jobExecutionId = jobProps.getProperty(AZKABAN_FLOW_EXEC_ID, "");
+    String jobExecutionId = jobProps.getProperty(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, "");

Review Comment:
   addded



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtilsTest.java:
##########
@@ -77,6 +78,26 @@ public void testSubmitNextNodesSuccess() throws URISyntaxException, IOException
     }
     Mockito.verifyNoMoreInteractions(dagManagementStateStore);
   }
+  @Test

Review Comment:
   added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2055553678


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   The  com.typesafe.config maintains config in hierarichal structure, so it maintains same paths, in the same structure, for it to distinguish .```
   withValue(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, ConfigValueFactory.fromAnyRef(gaasJobExecutionIdHash))
             .withoutPath())
   ```
   it would have been needed to add the config like this, but
   In general, it is not recommended to keep two configuration keys with the same base name (e.g., a.key1 and a.key1.hash) because this can lead to ambiguity and conflicts within the configuration management system.
   



##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   The  com.typesafe.config maintains config in hierarichal structure, so it maintains same paths, in the same structure, for it to distinguish .```
   withValue(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, ConfigValueFactory.fromAnyRef(gaasJobExecutionIdHash))
             .withoutPath())
   ```
   ```
   it would have been needed to add the config like this, but
   In general, it is not recommended to keep two configuration keys with the same base name (e.g., a.key1 and a.key1.hash) because this can lead to ambiguity and conflicts within the configuration management system.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2052423286


##########
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java:
##########
@@ -385,7 +385,11 @@ private boolean isCurrentTimeInRange() {
    */
   private static List<? extends Tag<?>> addAdditionalMetadataTags(Properties jobProps) {
     List<Tag<?>> metadataTags = Lists.newArrayList();
-    String jobExecutionId = jobProps.getProperty(AZKABAN_FLOW_EXEC_ID, "");
+    String jobExecutionId = jobProps.getProperty(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, "");

Review Comment:
   Yeah , we will use the same class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2055251377


##########
gobblin-utility/src/main/java/org/apache/gobblin/util/AzkabanTags.java:
##########
@@ -38,6 +38,7 @@ public class AzkabanTags {
       .put(ConfigurationKeys.AZKABAN_FLOW_ID, "azkabanFlowId")
       .put(ConfigurationKeys.AZKABAN_JOB_ID, "azkabanJobId")
       .put(ConfigurationKeys.AZKABAN_EXEC_ID, "azkabanExecId")
+      .put(ConfigurationKeys.GAAS_JOB_EXEC_ID,"gaasJobExecId")

Review Comment:
   nit: let's put a space after ,



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtilsTest.java:
##########
@@ -78,6 +80,27 @@ public void testSubmitNextNodesSuccess() throws URISyntaxException, IOException
     Mockito.verifyNoMoreInteractions(dagManagementStateStore);
   }
 
+  @Test
+  public void testGaaSJobExecutionIdInjection() throws URISyntaxException, IOException {
+    Dag.DagId dagId = new Dag.DagId("testFlowGroup", "testFlowName", 2345678);
+    List<JobExecutionPlan> jobExecutionPlans = getJobExecutionPlans();
+    List<Dag.DagNode<JobExecutionPlan>> dagNodeList = jobExecutionPlans.stream()
+        .map(Dag.DagNode<JobExecutionPlan>::new)
+        .collect(Collectors.toList());
+    Dag<JobExecutionPlan> dag = new Dag<>(dagNodeList);
+    Mockito.doNothing().when(dagManagementStateStore).addJobDagAction(Mockito.anyString(), Mockito.anyString(), Mockito.anyLong(), Mockito.anyString(), Mockito.any());
+    DagProcUtils.submitNextNodes(dagManagementStateStore, dag, dagId);
+    // Assertion to test that GaaS job execution Id has been successfully injected
+    for(JobExecutionPlan jobExecutionPlan : jobExecutionPlans) {
+      final String gaasJobExecutionId = ConfigUtils.getString(jobExecutionPlan.getJobSpec().getConfig(),ConfigurationKeys.GAAS_JOB_EXEC_ID,"");
+      final Long gaasJobExecutionIdHash = Long.parseLong(ConfigUtils.getString(jobExecutionPlan.getJobSpec().getConfig(),ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH,""));

Review Comment:
   nit: space after comma



##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   why does the rename work?



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtilsTest.java:
##########
@@ -78,6 +80,27 @@ public void testSubmitNextNodesSuccess() throws URISyntaxException, IOException
     Mockito.verifyNoMoreInteractions(dagManagementStateStore);
   }
 
+  @Test
+  public void testGaaSJobExecutionIdInjection() throws URISyntaxException, IOException {
+    Dag.DagId dagId = new Dag.DagId("testFlowGroup", "testFlowName", 2345678);
+    List<JobExecutionPlan> jobExecutionPlans = getJobExecutionPlans();
+    List<Dag.DagNode<JobExecutionPlan>> dagNodeList = jobExecutionPlans.stream()
+        .map(Dag.DagNode<JobExecutionPlan>::new)
+        .collect(Collectors.toList());
+    Dag<JobExecutionPlan> dag = new Dag<>(dagNodeList);
+    Mockito.doNothing().when(dagManagementStateStore).addJobDagAction(Mockito.anyString(), Mockito.anyString(), Mockito.anyLong(), Mockito.anyString(), Mockito.any());
+    DagProcUtils.submitNextNodes(dagManagementStateStore, dag, dagId);
+    // Assertion to test that GaaS job execution Id has been successfully injected
+    for(JobExecutionPlan jobExecutionPlan : jobExecutionPlans) {
+      final String gaasJobExecutionId = ConfigUtils.getString(jobExecutionPlan.getJobSpec().getConfig(),ConfigurationKeys.GAAS_JOB_EXEC_ID,"");

Review Comment:
   nit: can we use `StringUtils.EMPTY` instead of "" ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2052384155


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/Help.java:
##########
@@ -104,9 +103,9 @@ public static String calcPerExecQualifierWithOptFlowExecId(FileSystemJobStateful
   public static String calcPerExecQualifier(Config workerConfig) {
     String userToProxy = workerConfig.hasPath(USER_TO_PROXY_KEY)
         ? workerConfig.getString(USER_TO_PROXY_KEY) : "";
-    String azFlowExecId = workerConfig.hasPath(AZKABAN_FLOW_EXEC_ID_KEY)
-        ? workerConfig.getString(AZKABAN_FLOW_EXEC_ID_KEY) : UUID.randomUUID().toString();
-    return userToProxy + "_" + azFlowExecId;
+    String gaasFlowExecId = workerConfig.hasPath(ConfigurationKeys.GAAS_JOB_EXEC_ID)
+        ? workerConfig.getString(ConfigurationKeys.GAAS_JOB_EXEC_ID) : UUID.randomUUID().toString();

Review Comment:
   I believe this code is getting used only in executor where the jar will not be updated during deployment, unlike service where the deployment will affect currently running jobs.
   Can you please confirm if that is the intention behind not adding a fallback here?
   Or is this getting used somewhere in service and we are still not adding a fallback for some reason?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2055553678


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   The  com.typesafe.config maintains config in hierarichal structure, so it maintains same paths, in the same structure, for it to distinguish .```
   withValue(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, ConfigValueFactory.fromAnyRef(gaasJobExecutionIdHash))
             .withoutPath())
   ```
   ```
   it would have been needed to add the config like this " .withoutPath())", but
   In general, it is not recommended to keep two configuration keys with the same base name (e.g., a.key1 and a.key1.hash) because this can lead to ambiguity and conflicts within the configuration management system.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2055553678


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   The  com.typesafe.config maintains config in hierarichal structure, so it maintains same paths, in the same structure, for it to distinguish .```
   withValue(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, ConfigValueFactory.fromAnyRef(gaasJobExecutionIdHash))
             .withoutPath("basePath"))
   ```
   ```
   it would have been needed to add the config like this " .withoutPath("basePath))", but
   In general, it is not recommended to keep two configuration keys with the same base name (e.g., a.key1 and a.key1.hash) because this can lead to ambiguity and conflicts within the configuration management system.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "Blazer-007 (via GitHub)" <gi...@apache.org>.
Blazer-007 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2047076545


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for to store has of gaas.job.execid, to be used for jobExecutionId for backwards compatibility

Review Comment:
   NIT : `has` -> `hash`



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtilsTest.java:
##########
@@ -77,6 +78,26 @@ public void testSubmitNextNodesSuccess() throws URISyntaxException, IOException
     }
     Mockito.verifyNoMoreInteractions(dagManagementStateStore);
   }
+  @Test

Review Comment:
   NIT: insert single blank line



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/spec/JobExecutionPlan.java:
##########
@@ -116,6 +117,10 @@ private static JobSpec buildJobSpec(FlowSpec flowSpec, Config jobConfig, Long fl
 
       String jobName = ConfigUtils.getString(jobConfig, ConfigurationKeys.JOB_NAME_KEY, "");
       String edgeId = ConfigUtils.getString(jobConfig, FlowGraphConfigurationKeys.FLOW_EDGE_ID_KEY, "");
+      final UUID gaasJobExecutionUUID = UUID.randomUUID();
+      final String gaasJobExecutionId = gaasJobExecutionUUID.toString(); // Creating a unique Identifier for JobExecution

Review Comment:
   NIT: this can be merged into single line



##########
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java:
##########
@@ -385,7 +385,11 @@ private boolean isCurrentTimeInRange() {
    */
   private static List<? extends Tag<?>> addAdditionalMetadataTags(Properties jobProps) {
     List<Tag<?>> metadataTags = Lists.newArrayList();
-    String jobExecutionId = jobProps.getProperty(AZKABAN_FLOW_EXEC_ID, "");
+    String jobExecutionId = jobProps.getProperty(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, "");

Review Comment:
   can you update the associated comment below as well on line 414?
   
   `//Use azkaban.flow.execid as the jobExecutionId`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2052422167


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/Help.java:
##########
@@ -104,9 +103,9 @@ public static String calcPerExecQualifierWithOptFlowExecId(FileSystemJobStateful
   public static String calcPerExecQualifier(Config workerConfig) {
     String userToProxy = workerConfig.hasPath(USER_TO_PROXY_KEY)
         ? workerConfig.getString(USER_TO_PROXY_KEY) : "";
-    String azFlowExecId = workerConfig.hasPath(AZKABAN_FLOW_EXEC_ID_KEY)
-        ? workerConfig.getString(AZKABAN_FLOW_EXEC_ID_KEY) : UUID.randomUUID().toString();
-    return userToProxy + "_" + azFlowExecId;
+    String gaasFlowExecId = workerConfig.hasPath(ConfigurationKeys.GAAS_JOB_EXEC_ID)
+        ? workerConfig.getString(ConfigurationKeys.GAAS_JOB_EXEC_ID) : UUID.randomUUID().toString();

Review Comment:
   It is only being in executor, which has a separate deployment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2055553678


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   The  com.typesafe.config maintains config in hierarichal structure, so it maintains same paths, in the same structure, for it to distinguish .withValue(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, ConfigValueFactory.fromAnyRef(gaasJobExecutionIdHash))
             .withoutPath())
   ```
   it would have been needed to add the config like this, but
   In general, it is not recommended to keep two configuration keys with the same base name (e.g., a.key1 and a.key1.hash) because this can lead to ambiguity and conflicts within the configuration management system.
   ```
   



##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for storing hash of gaas.job.execid, to be used as jobExecutionId(integer) for backwards compatibility
+  public static final String GAAS_JOB_EXEC_ID_HASH = "gaas.job.executionid.hash";

Review Comment:
   The  com.typesafe.config maintains config in hierarichal structure, so it maintains same paths, in the same structure, for it to distinguish .```
   withValue(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, ConfigValueFactory.fromAnyRef(gaasJobExecutionIdHash))
             .withoutPath())
   ```
   ```
   it would have been needed to add the config like this, but
   In general, it is not recommended to keep two configuration keys with the same base name (e.g., a.key1 and a.key1.hash) because this can lead to ambiguity and conflicts within the configuration management system.
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "pratapaditya04 (via GitHub)" <gi...@apache.org>.
pratapaditya04 commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2048312182


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/spec/JobExecutionPlan.java:
##########
@@ -116,6 +117,10 @@ private static JobSpec buildJobSpec(FlowSpec flowSpec, Config jobConfig, Long fl
 
       String jobName = ConfigUtils.getString(jobConfig, ConfigurationKeys.JOB_NAME_KEY, "");
       String edgeId = ConfigUtils.getString(jobConfig, FlowGraphConfigurationKeys.FLOW_EDGE_ID_KEY, "");
+      final UUID gaasJobExecutionUUID = UUID.randomUUID();
+      final String gaasJobExecutionId = gaasJobExecutionUUID.toString(); // Creating a unique Identifier for JobExecution

Review Comment:
   addressed



##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java:
##########
@@ -1047,6 +1047,12 @@ public class ConfigurationKeys {
   public static final String AZKABAN_FLOW_ID = "azkaban.flow.flowid";
   public static final String AZKABAN_JOB_ID = "azkaban.job.id";
   public static final String AZKABAN_EXEC_ID = "azkaban.flow.execid";
+  // Configuration Key for setting a unique job execution identifier in GaaS, the value is a UUID
+  public static final String GAAS_JOB_EXEC_ID = "gaas.job.execid";
+
+  // Configuration Key for to store has of gaas.job.execid, to be used for jobExecutionId for backwards compatibility

Review Comment:
   addressed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain commented on code in PR #4108:
URL: https://github.com/apache/gobblin/pull/4108#discussion_r2052395789


##########
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java:
##########
@@ -385,7 +385,11 @@ private boolean isCurrentTimeInRange() {
    */
   private static List<? extends Tag<?>> addAdditionalMetadataTags(Properties jobProps) {
     List<Tag<?>> metadataTags = Lists.newArrayList();
-    String jobExecutionId = jobProps.getProperty(AZKABAN_FLOW_EXEC_ID, "");
+    String jobExecutionId = jobProps.getProperty(ConfigurationKeys.GAAS_JOB_EXEC_ID_HASH, "");

Review Comment:
   Are we planning to use this class if we want to use something other than Azkaban? If we create a new class, would we still require this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2200]Moving Away From Azkaban Execution Id [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain merged PR #4108:
URL: https://github.com/apache/gobblin/pull/4108


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org