You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/10/29 07:53:19 UTC

[GitHub] [gobblin] phet opened a new pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

phet opened a new pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421


   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1569
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   
   Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
   
   Present storage bindings for `JobCatalog` are in-memory or file system.  While the latter is durable, it requires an NFS-like capability for cross-instance sharing.
   
   Support alternative multi-instance backing through MySQL, precluding the need for NFS.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   
   New unit tests
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7175a4a) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `2.56%`.
   > The diff coverage is `58.02%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   49.09%   +2.56%     
   + Complexity    10246     8838    -1408     
   ============================================
     Files          2063     1689     -374     
     Lines         80475    65210   -15265     
     Branches       8989     7476    -1513     
   ============================================
   - Hits          37443    32012    -5431     
   + Misses        39567    30175    -9392     
   + Partials       3465     3023     -442     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | [...va/org/apache/gobblin/runtime/SourceDecorator.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU291cmNlRGVjb3JhdG9yLmphdmE=) | `41.02% <20.00%> (-0.16%)` | :arrow_down: |
   | [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | `56.16% <28.57%> (-3.26%)` | :arrow_down: |
   | ... and [410 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7175a4a](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r742355943



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740472936



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       This map function seems unnecessarily complicated, couldn't we just do:
   
   ```
       try {
         templateURI = Optional.ofNullable(new URI(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()));
       } catch (URISyntaxException e) {
          LOGGER.warn(String.format("error deserializing '%s' as a URI",
                 jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
         templateURI = Optional.empty();
       }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740665535



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       first off, I agree it's unsatisfying.  (I welcome any tips toward finessing java's checked exceptions!)
   
   the transformation you suggest doesn't protect against passing `null` to the `URI` ctor.  that's why I was working (via `map`) *within* the `Optional`.  still, you did helped spot a NPE bug from the parens closed in the wrong place.  we need:
   ```
   Optional.ofNullable(jsonObj.get(...)).map
   ```
   not the potentially explosive:
   ```
   Optional.ofNullable(jsonObj.get(...).getAsString()).map
   ```
   in addition, I'll use `Optional.flatMap` to eliminate the second level of exception handling.  I'll push those corrections, and you please tell me if you see any other potential improvement.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter commented on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **decrease** coverage by `3.33%`.
   > The diff coverage is `41.46%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   - Coverage     46.52%   43.18%   -3.34%     
   + Complexity    10246     2008    -8238     
   ============================================
     Files          2063      401    -1662     
     Lines         80475    17275   -63200     
     Branches       8989     2121    -6868     
   ============================================
   - Hits          37443     7461   -29982     
   + Misses        39567     8977   -30590     
   + Partials       3465      837    -2628     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
   | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
   | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
   | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | | |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | | |
   | [...ache/gobblin/metrics/kafka/KafkaEventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyLmphdmE=) | | |
   | ... and [1660 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r742355943



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7175a4a) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **decrease** coverage by `3.33%`.
   > The diff coverage is `41.46%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   - Coverage     46.52%   43.19%   -3.34%     
   + Complexity    10246     2008    -8238     
   ============================================
     Files          2063      401    -1662     
     Lines         80475    17275   -63200     
     Branches       8989     2121    -6868     
   ============================================
   - Hits          37443     7462   -29981     
   + Misses        39567     8977   -30590     
   + Partials       3465      836    -2629     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
   | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
   | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | | |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | | |
   | [...ache/gobblin/metrics/kafka/KafkaEventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyLmphdmE=) | | |
   | [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | | |
   | ... and [1659 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7175a4a](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";

Review comment:
       it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
   ```
       Config config = ConfigBuilder.create()
           .addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
           .addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
   ...
   ```
   what do you think?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.

Review comment:
       sure, will rephrase to:
   ```
   MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
   ```

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);
+          this.listeners.onUpdateJob(jobSpec);
+        } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+          throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+        }
+      } else {
+        jobSpecStore.addSpec(jobSpec);
+        this.mutableMetrics.updatePutJobTime(startTime);
+        this.listeners.onAddJob(jobSpec);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+    }
+  }
+
+  /**
+   * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   */
+  @Override
+  public synchronized void remove(URI jobURI) {
+    remove(jobURI, false);
+  }
+
+  /**
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   *
+   * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+   */
+  @Override
+  public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobURI);
+    try {
+      long startTime = System.currentTimeMillis();
+      JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+      jobSpecStore.deleteSpec(jobURI);
+      this.mutableMetrics.updateRemoveJobTime(startTime);
+      this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());

Review comment:
       I lack the history on this one.  I'm also not following what additional spec store calls arise as a result.
   (overall I'm not opposed to streamlining the functionality on offer...)
   please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+  private GsonSerDe<JobSpec> gsonSerDe;

Review comment:
       yes, true

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {

Review comment:
       good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
   
   since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer.  handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);

Review comment:
       I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `2.55%`.
   > The diff coverage is `57.44%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   49.08%   +2.55%     
   + Complexity    10246     8839    -1407     
   ============================================
     Files          2063     1688     -375     
     Lines         80475    65217   -15258     
     Branches       8989     7476    -1513     
   ============================================
   - Hits          37443    32013    -5430     
   + Misses        39567    30179    -9388     
   + Partials       3465     3025     -440     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | [...va/org/apache/gobblin/runtime/SourceDecorator.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU291cmNlRGVjb3JhdG9yLmphdmE=) | `41.02% <20.00%> (-0.16%)` | :arrow_down: |
   | [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | `56.16% <28.57%> (-3.26%)` | :arrow_down: |
   | ... and [407 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741274525



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";

Review comment:
       This name does not match the value it holds, right? how about MYSQL_JOB_CATALOG or similar?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.

Review comment:
       {@link JobSpec}
   nit *supports

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+  private GsonSerDe<JobSpec> gsonSerDe;

Review comment:
       could be `final`

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);

Review comment:
       This line could be out of if else ?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);
+          this.listeners.onUpdateJob(jobSpec);
+        } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+          throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+        }
+      } else {
+        jobSpecStore.addSpec(jobSpec);
+        this.mutableMetrics.updatePutJobTime(startTime);
+        this.listeners.onAddJob(jobSpec);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+    }
+  }
+
+  /**
+   * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   */
+  @Override
+  public synchronized void remove(URI jobURI) {
+    remove(jobURI, false);
+  }
+
+  /**
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   *
+   * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+   */
+  @Override
+  public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobURI);
+    try {
+      long startTime = System.currentTimeMillis();
+      JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+      jobSpecStore.deleteSpec(jobURI);
+      this.mutableMetrics.updateRemoveJobTime(startTime);
+      this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());

Review comment:
       So it seems supporting a feature (spec versioning) which may have bugs, and is not being used is costing us more spec store calls. Should we remove this in future, or at least provide a version of APIs without the version (that passes to the existing API with a default version)

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {

Review comment:
       There must be a way to reuse `GsonFlowSpecSerDe` , can we use generics in the interface?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aff832) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **decrease** coverage by `3.35%`.
   > The diff coverage is `41.46%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   - Coverage     46.52%   43.17%   -3.36%     
   + Complexity    10246     2007    -8239     
   ============================================
     Files          2063      401    -1662     
     Lines         80475    17275   -63200     
     Branches       8989     2121    -6868     
   ============================================
   - Hits          37443     7458   -29985     
   + Misses        39567     8981   -30586     
   + Partials       3465      836    -2629     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
   | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
   | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
   | [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | | |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | | |
   | [...ache/gobblin/metrics/kafka/KafkaEventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyLmphdmE=) | | |
   | [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | | |
   | ... and [1659 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7aff832](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] ZihanLi58 merged pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7175a4a) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.02%`.
   > The diff coverage is `57.43%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   46.55%   +0.02%     
   - Complexity    10246    10302      +56     
   ============================================
     Files          2063     2070       +7     
     Lines         80475    80751     +276     
     Branches       8989     9009      +20     
   ============================================
   + Hits          37443    37591     +148     
   - Misses        39567    39684     +117     
   - Partials       3465     3476      +11     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
   | [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | ... and [33 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7175a4a](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";

Review comment:
       it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
   ```
       Config config = ConfigBuilder.create()
           .addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
           .addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
   ...
   ```
   what do you think?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.

Review comment:
       sure, will rephrase to:
   ```
   MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
   ```

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);
+          this.listeners.onUpdateJob(jobSpec);
+        } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+          throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+        }
+      } else {
+        jobSpecStore.addSpec(jobSpec);
+        this.mutableMetrics.updatePutJobTime(startTime);
+        this.listeners.onAddJob(jobSpec);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+    }
+  }
+
+  /**
+   * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   */
+  @Override
+  public synchronized void remove(URI jobURI) {
+    remove(jobURI, false);
+  }
+
+  /**
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   *
+   * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+   */
+  @Override
+  public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobURI);
+    try {
+      long startTime = System.currentTimeMillis();
+      JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+      jobSpecStore.deleteSpec(jobURI);
+      this.mutableMetrics.updateRemoveJobTime(startTime);
+      this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());

Review comment:
       I lack the history on this one.  I'm also not following what additional spec store calls arise as a result.
   (overall I'm not opposed to streamlining the functionality on offer...)
   please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+  private GsonSerDe<JobSpec> gsonSerDe;

Review comment:
       yes, true

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {

Review comment:
       good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
   
   since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer.  handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);

Review comment:
       I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aff832) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.01%`.
   > The diff coverage is `57.11%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   46.54%   +0.01%     
   - Complexity    10246    10301      +55     
   ============================================
     Files          2063     2069       +6     
     Lines         80475    80754     +279     
     Branches       8989     9009      +20     
   ============================================
   + Hits          37443    37586     +143     
   - Misses        39567    39690     +123     
   - Partials       3465     3478      +13     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
   | [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | ... and [30 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7aff832](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aff832) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `2.55%`.
   > The diff coverage is `57.69%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   49.08%   +2.55%     
   + Complexity    10246     8837    -1409     
   ============================================
     Files          2063     1688     -375     
     Lines         80475    65213   -15262     
     Branches       8989     7476    -1513     
   ============================================
   - Hits          37443    32007    -5436     
   + Misses        39567    30181    -9386     
   + Partials       3465     3025     -440     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | [...va/org/apache/gobblin/runtime/SourceDecorator.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU291cmNlRGVjb3JhdG9yLmphdmE=) | `41.02% <20.00%> (-0.16%)` | :arrow_down: |
   | [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | `56.16% <28.57%> (-3.26%)` | :arrow_down: |
   | ... and [407 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7aff832](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.02%`.
   > The diff coverage is `56.87%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   46.54%   +0.02%     
   - Complexity    10246    10303      +57     
   ============================================
     Files          2063     2069       +6     
     Lines         80475    80758     +283     
     Branches       8989     9009      +20     
   ============================================
   + Hits          37443    37592     +149     
   - Misses        39567    39688     +121     
   - Partials       3465     3478      +13     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
   | [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | ... and [30 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.14%`.
   > The diff coverage is `38.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   46.67%   +0.14%     
   + Complexity    10246     3173    -7073     
   ============================================
     Files          2063      654    -1409     
     Lines         80475    25781   -54694     
     Branches       8989     3061    -5928     
   ============================================
   - Hits          37443    12033   -25410     
   + Misses        39567    12452   -27115     
   + Partials       3465     1296    -2169     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...lin/service/modules/flow/MultiHopFlowCompiler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L011bHRpSG9wRmxvd0NvbXBpbGVyLmphdmE=) | `69.87% <42.85%> (-0.83%)` | :arrow_down: |
   | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
   | [...n/service/modules/dataset/FSDatasetDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0ZTRGF0YXNldERlc2NyaXB0b3IuamF2YQ==) | `78.00% <100.00%> (+0.91%)` | :arrow_up: |
   | [...dules/flowgraph/pathfinder/AbstractPathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvcGF0aGZpbmRlci9BYnN0cmFjdFBhdGhGaW5kZXIuamF2YQ==) | `87.06% <100.00%> (ø)` | |
   | [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
   | [...gobblin/service/modules/core/GitConfigMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdENvbmZpZ01vbml0b3IuamF2YQ==) | `81.35% <0.00%> (-1.70%)` | :arrow_down: |
   | ... and [1416 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r739012044



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }
+    Properties properties;
+
+    try {
+      properties = context.deserialize(jsonObject.get(JobSpecSerializer.JOB_SPEC_CONFIG_AS_PROPERTIES_KEY), Properties.class);
+    } catch (JsonParseException e) {
+      // for backward compatibility... (is this needed for `JobSpec`, or only for (inspiration) `FlowSpec`???)
+      properties = new Properties();
+      try {
+        properties.load(new StringReader(jsonObject.get(JobSpecSerializer.JOB_SPEC_CONFIG_AS_PROPERTIES_KEY).getAsString()));

Review comment:
       this originated (with the comment "for backward compatibility") in `FlowSpecDeserializer`... is it necessary here too or a historical quirk specific to the former?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";

Review comment:
       it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
   ```
       Config config = ConfigBuilder.create()
           .addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
           .addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
   ...
   ```
   what do you think?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.

Review comment:
       sure, will rephrase to:
   ```
   MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
   ```

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);
+          this.listeners.onUpdateJob(jobSpec);
+        } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+          throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+        }
+      } else {
+        jobSpecStore.addSpec(jobSpec);
+        this.mutableMetrics.updatePutJobTime(startTime);
+        this.listeners.onAddJob(jobSpec);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+    }
+  }
+
+  /**
+   * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   */
+  @Override
+  public synchronized void remove(URI jobURI) {
+    remove(jobURI, false);
+  }
+
+  /**
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   *
+   * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+   */
+  @Override
+  public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobURI);
+    try {
+      long startTime = System.currentTimeMillis();
+      JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+      jobSpecStore.deleteSpec(jobURI);
+      this.mutableMetrics.updateRemoveJobTime(startTime);
+      this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());

Review comment:
       I lack the history on this one.  I'm also not following what additional spec store calls arise as a result.
   (overall I'm not opposed to streamlining the functionality on offer...)
   please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+  private GsonSerDe<JobSpec> gsonSerDe;

Review comment:
       yes, true

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {

Review comment:
       good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
   
   since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer.  handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);

Review comment:
       I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740472936



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       This map function seems unnecessarily complicated, couldn't we just do:
   
   ```
       try {
         templateURI = Optional.ofNullable(new URI(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()));
       } catch (URISyntaxException e) {
          LOGGER.warn(String.format("error deserializing '%s' as a URI",
                 jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
         templateURI = Optional.empty();
       }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740665535



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       first off, I agree it's unsatisfying.  (I welcome any tips toward finessing java's checked exceptions!)
   
   the transformation you suggest doesn't protect against passing `null` to the `URI` ctor.  that's why I was working (via `map`) *within* the `Optional`.  still, you did helped spot a NPE bug from the parens closed in the wrong place.  we need:
   ```
   Optional.ofNullable(jsonObj.get(...)).map
   ```
   not the potentially explosive:
   ```
   Optional.ofNullable(jsonObj.get(...).getAsString()).map
   ```
   in addition, I'll use `Optional.flatMap` to eliminate the second level of exception handling.  I'll push those corrections, and you please tell me if you see any other potential improvement.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";

Review comment:
       it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
   ```
       Config config = ConfigBuilder.create()
           .addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
           .addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
   ...
   ```
   what do you think?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.

Review comment:
       sure, will rephrase to:
   ```
   MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
   ```

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);
+          this.listeners.onUpdateJob(jobSpec);
+        } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+          throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+        }
+      } else {
+        jobSpecStore.addSpec(jobSpec);
+        this.mutableMetrics.updatePutJobTime(startTime);
+        this.listeners.onAddJob(jobSpec);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+    }
+  }
+
+  /**
+   * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   */
+  @Override
+  public synchronized void remove(URI jobURI) {
+    remove(jobURI, false);
+  }
+
+  /**
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   *
+   * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+   */
+  @Override
+  public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobURI);
+    try {
+      long startTime = System.currentTimeMillis();
+      JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+      jobSpecStore.deleteSpec(jobURI);
+      this.mutableMetrics.updateRemoveJobTime(startTime);
+      this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());

Review comment:
       I lack the history on this one.  I'm also not following what additional spec store calls arise as a result.
   (overall I'm not opposed to streamlining the functionality on offer...)
   please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+  private GsonSerDe<JobSpec> gsonSerDe;

Review comment:
       yes, true

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {

Review comment:
       good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
   
   since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer.  handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);

Review comment:
       I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r742355943



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] ZihanLi58 merged pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] ZihanLi58 merged pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740665535



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+  @Override
+  public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+    JsonObject jsonObject = json.getAsJsonObject();
+
+    String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+    String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+    String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+    Optional<URI> templateURI;
+    try {
+      templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+        try {
+          return new URI(s);
+        } catch (URISyntaxException e) {
+          LOGGER.warn(String.format("error deserializing '%s' as a URI",
+              jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+          throw new RuntimeException(e);
+        }
+      });
+    } catch (RuntimeException e) {
+      templateURI = Optional.empty();
+    }

Review comment:
       first off, I agree it's unsatisfying.  (I welcome any tips toward finessing java's checked exceptions!)
   
   the transformation you suggest doesn't protect against passing `null` to the `URI` ctor.  that's why I was working (via `map`) *within* the `Optional`.  still, you did helped spot a NPE bug from the parens closed in the wrong place.  we need:
   ```
   Optional.ofNullable(jsonObj.get(...)).map
   ```
   not the potentially explosive:
   ```
   Optional.ofNullable(jsonObj.get(...).getAsString()).map
   ```
   in addition, I'll use `Optional.flatMap` to eliminate the second level of exception handling.  I'll push those corrections, and you please tell me if you see any other potential improvement.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741274525



##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";

Review comment:
       This name does not match the value it holds, right? how about MYSQL_JOB_CATALOG or similar?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.

Review comment:
       {@link JobSpec}
   nit *supports

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+  private GsonSerDe<JobSpec> gsonSerDe;

Review comment:
       could be `final`

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);

Review comment:
       This line could be out of if else ?

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information.  Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+  public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+  protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+  protected final MysqlBaseSpecStore jobSpecStore;
+
+  /**
+   * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+   */
+  public MysqlJobCatalog(Config sysConfig)
+      throws IOException {
+      this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+  }
+
+  public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+    super(env);
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+  }
+
+  public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+      boolean instrumentationEnabled) throws IOException {
+    super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+    this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+    this.jobSpecStore = createJobSpecStore(sysConfig);
+  }
+
+  @Override
+  protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+    log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+    return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+  }
+
+  protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+    try {
+      return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+        @Override
+        protected String getConfigPrefix() {
+          return MysqlJobCatalog.DB_CONFIG_PREFIX;
+        }
+      };
+    } catch (IOException e) {
+      throw new RuntimeException("unable to create `JobSpec` store", e);
+    }
+  }
+
+  /** @return all {@link JobSpec}s */
+  @Override
+  public List<JobSpec> getJobs() {
+    try {
+      return (List) jobSpecStore.getSpecs();
+    } catch (IOException e) {
+      throw new RuntimeException("error getting (all) job specs", e);
+    }
+  }
+
+  /**
+   * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+   * all job configs into memory in the very beginning.
+   * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+   *
+   * @return an iterator for (all) present {@link JobSpec}s
+   */
+  @Override
+  public Iterator<JobSpec> getJobSpecIterator() {
+    try {
+      return Iterators.<Optional<JobSpec>, JobSpec>transform(
+          Iterators.filter(
+              Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+                try {
+                  return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+                } catch (JobSpecNotFoundException e) {
+                  MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+                  return Optional.absent();
+                }}),
+              Optional::isPresent),
+          Optional::get);
+    } catch (IOException e) {
+      throw new RuntimeException("error iterating (all) job specs", e);
+    }
+  }
+
+  /**
+   * Fetch single {@link JobSpec} by URI.
+   * @return the `JobSpec`
+   * @throws {@link JobSpecNotFoundException}
+   */
+  @Override
+  public JobSpec getJobSpec(URI uri)
+      throws JobSpecNotFoundException {
+    Preconditions.checkNotNull(uri);
+    try {
+      return (JobSpec) jobSpecStore.getSpec(uri);
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+    } catch (SpecNotFoundException e) {
+      throw new JobSpecNotFoundException(uri);
+    }
+  }
+
+  /**
+   * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+   * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+   */
+  @Override
+  public synchronized void put(JobSpec jobSpec) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobSpec);
+    try {
+      long startTime = System.currentTimeMillis();
+      boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+      if (isUpdate) {
+        try {
+          jobSpecStore.updateSpec(jobSpec);
+          this.mutableMetrics.updatePutJobTime(startTime);
+          this.listeners.onUpdateJob(jobSpec);
+        } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+          throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+        }
+      } else {
+        jobSpecStore.addSpec(jobSpec);
+        this.mutableMetrics.updatePutJobTime(startTime);
+        this.listeners.onAddJob(jobSpec);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+    }
+  }
+
+  /**
+   * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+   *
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   */
+  @Override
+  public synchronized void remove(URI jobURI) {
+    remove(jobURI, false);
+  }
+
+  /**
+   * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+   *
+   * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+   */
+  @Override
+  public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+    Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+    Preconditions.checkNotNull(jobURI);
+    try {
+      long startTime = System.currentTimeMillis();
+      JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+      jobSpecStore.deleteSpec(jobURI);
+      this.mutableMetrics.updateRemoveJobTime(startTime);
+      this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());

Review comment:
       So it seems supporting a feature (spec versioning) which may have bugs, and is not being used is costing us more spec store calls. Should we remove this in future, or at least provide a version of APIs without the version (that passes to the existing API with a default version)

##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {

Review comment:
       There must be a way to reuse `GsonFlowSpecSerDe` , can we use generics in the interface?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855


   # [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7985803) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.01%`.
   > The diff coverage is `57.43%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/gobblin/pull/3421/graphs/tree.svg?width=650&height=150&src=pr&token=4MgURJ0bGc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3421      +/-   ##
   ============================================
   + Coverage     46.52%   46.54%   +0.01%     
   - Complexity    10246    10300      +54     
   ============================================
     Files          2063     2070       +7     
     Lines         80475    80751     +276     
     Branches       8989     9009      +20     
   ============================================
   + Hits          37443    37584     +141     
   - Misses        39567    39690     +123     
   - Partials       3465     3477      +12     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
   | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
   | [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
   | [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
   | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
   | [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
   | [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
   | [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
   | ... and [31 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7985803](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org