You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/10/29 07:53:19 UTC
[GitHub] [gobblin] phet opened a new pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
phet opened a new pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1569
### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):
Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Present storage bindings for `JobCatalog` are in-memory or file system. While the latter is durable, it requires an NFS-like capability for cross-instance sharing.
Support alternative multi-instance backing through MySQL, precluding the need for NFS.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
New unit tests
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7175a4a) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `2.56%`.
> The diff coverage is `58.02%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 49.09% +2.56%
+ Complexity 10246 8838 -1408
============================================
Files 2063 1689 -374
Lines 80475 65210 -15265
Branches 8989 7476 -1513
============================================
- Hits 37443 32012 -5431
+ Misses 39567 30175 -9392
+ Partials 3465 3023 -442
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| [...va/org/apache/gobblin/runtime/SourceDecorator.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU291cmNlRGVjb3JhdG9yLmphdmE=) | `41.02% <20.00%> (-0.16%)` | :arrow_down: |
| [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | `56.16% <28.57%> (-3.26%)` | :arrow_down: |
| ... and [410 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7175a4a](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r742355943
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740472936
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
This map function seems unnecessarily complicated, couldn't we just do:
```
try {
templateURI = Optional.ofNullable(new URI(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()));
} catch (URISyntaxException e) {
LOGGER.warn(String.format("error deserializing '%s' as a URI",
jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
templateURI = Optional.empty();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740665535
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
first off, I agree it's unsatisfying. (I welcome any tips toward finessing java's checked exceptions!)
the transformation you suggest doesn't protect against passing `null` to the `URI` ctor. that's why I was working (via `map`) *within* the `Optional`. still, you did helped spot a NPE bug from the parens closed in the wrong place. we need:
```
Optional.ofNullable(jsonObj.get(...)).map
```
not the potentially explosive:
```
Optional.ofNullable(jsonObj.get(...).getAsString()).map
```
in addition, I'll use `Optional.flatMap` to eliminate the second level of exception handling. I'll push those corrections, and you please tell me if you see any other potential improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter commented on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **decrease** coverage by `3.33%`.
> The diff coverage is `41.46%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
- Coverage 46.52% 43.18% -3.34%
+ Complexity 10246 2008 -8238
============================================
Files 2063 401 -1662
Lines 80475 17275 -63200
Branches 8989 2121 -6868
============================================
- Hits 37443 7461 -29982
+ Misses 39567 8977 -30590
+ Partials 3465 837 -2628
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | | |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | | |
| [...ache/gobblin/metrics/kafka/KafkaEventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyLmphdmE=) | | |
| ... and [1660 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r742355943
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7175a4a) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **decrease** coverage by `3.33%`.
> The diff coverage is `41.46%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
- Coverage 46.52% 43.19% -3.34%
+ Complexity 10246 2008 -8238
============================================
Files 2063 401 -1662
Lines 80475 17275 -63200
Branches 8989 2121 -6868
============================================
- Hits 37443 7462 -29981
+ Misses 39567 8977 -30590
+ Partials 3465 836 -2629
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | | |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | | |
| [...ache/gobblin/metrics/kafka/KafkaEventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyLmphdmE=) | | |
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | | |
| ... and [1659 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7175a4a](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
Review comment:
it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
```
Config config = ConfigBuilder.create()
.addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
.addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
...
```
what do you think?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
Review comment:
sure, will rephrase to:
```
MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
```
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onUpdateJob(jobSpec);
+ } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+ throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+ }
+ } else {
+ jobSpecStore.addSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onAddJob(jobSpec);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+ }
+ }
+
+ /**
+ * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ */
+ @Override
+ public synchronized void remove(URI jobURI) {
+ remove(jobURI, false);
+ }
+
+ /**
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ *
+ * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+ */
+ @Override
+ public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobURI);
+ try {
+ long startTime = System.currentTimeMillis();
+ JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+ jobSpecStore.deleteSpec(jobURI);
+ this.mutableMetrics.updateRemoveJobTime(startTime);
+ this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());
Review comment:
I lack the history on this one. I'm also not following what additional spec store calls arise as a result.
(overall I'm not opposed to streamlining the functionality on offer...)
please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+ private GsonSerDe<JobSpec> gsonSerDe;
Review comment:
yes, true
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
Review comment:
good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer. handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
Review comment:
I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `2.55%`.
> The diff coverage is `57.44%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 49.08% +2.55%
+ Complexity 10246 8839 -1407
============================================
Files 2063 1688 -375
Lines 80475 65217 -15258
Branches 8989 7476 -1513
============================================
- Hits 37443 32013 -5430
+ Misses 39567 30179 -9388
+ Partials 3465 3025 -440
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| [...va/org/apache/gobblin/runtime/SourceDecorator.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU291cmNlRGVjb3JhdG9yLmphdmE=) | `41.02% <20.00%> (-0.16%)` | :arrow_down: |
| [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | `56.16% <28.57%> (-3.26%)` | :arrow_down: |
| ... and [407 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741274525
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
Review comment:
This name does not match the value it holds, right? how about MYSQL_JOB_CATALOG or similar?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
Review comment:
{@link JobSpec}
nit *supports
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+ private GsonSerDe<JobSpec> gsonSerDe;
Review comment:
could be `final`
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
Review comment:
This line could be out of if else ?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onUpdateJob(jobSpec);
+ } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+ throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+ }
+ } else {
+ jobSpecStore.addSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onAddJob(jobSpec);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+ }
+ }
+
+ /**
+ * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ */
+ @Override
+ public synchronized void remove(URI jobURI) {
+ remove(jobURI, false);
+ }
+
+ /**
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ *
+ * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+ */
+ @Override
+ public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobURI);
+ try {
+ long startTime = System.currentTimeMillis();
+ JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+ jobSpecStore.deleteSpec(jobURI);
+ this.mutableMetrics.updateRemoveJobTime(startTime);
+ this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());
Review comment:
So it seems supporting a feature (spec versioning) which may have bugs, and is not being used is costing us more spec store calls. Should we remove this in future, or at least provide a version of APIs without the version (that passes to the existing API with a default version)
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
Review comment:
There must be a way to reuse `GsonFlowSpecSerDe` , can we use generics in the interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aff832) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **decrease** coverage by `3.35%`.
> The diff coverage is `41.46%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
- Coverage 46.52% 43.17% -3.36%
+ Complexity 10246 2007 -8239
============================================
Files 2063 401 -1662
Lines 80475 17275 -63200
Branches 8989 2121 -6868
============================================
- Hits 37443 7458 -29985
+ Misses 39567 8981 -30586
+ Partials 3465 836 -2629
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...apache/gobblin/salesforce/SalesforceExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUV4dHJhY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | | |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | | |
| [...ache/gobblin/metrics/kafka/KafkaEventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FFdmVudFJlcG9ydGVyLmphdmE=) | | |
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | | |
| ... and [1659 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7aff832](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 merged pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7175a4a) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.02%`.
> The diff coverage is `57.43%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 46.55% +0.02%
- Complexity 10246 10302 +56
============================================
Files 2063 2070 +7
Lines 80475 80751 +276
Branches 8989 9009 +20
============================================
+ Hits 37443 37591 +148
- Misses 39567 39684 +117
- Partials 3465 3476 +11
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7175a4a](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
Review comment:
it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
```
Config config = ConfigBuilder.create()
.addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
.addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
...
```
what do you think?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
Review comment:
sure, will rephrase to:
```
MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
```
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onUpdateJob(jobSpec);
+ } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+ throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+ }
+ } else {
+ jobSpecStore.addSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onAddJob(jobSpec);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+ }
+ }
+
+ /**
+ * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ */
+ @Override
+ public synchronized void remove(URI jobURI) {
+ remove(jobURI, false);
+ }
+
+ /**
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ *
+ * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+ */
+ @Override
+ public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobURI);
+ try {
+ long startTime = System.currentTimeMillis();
+ JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+ jobSpecStore.deleteSpec(jobURI);
+ this.mutableMetrics.updateRemoveJobTime(startTime);
+ this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());
Review comment:
I lack the history on this one. I'm also not following what additional spec store calls arise as a result.
(overall I'm not opposed to streamlining the functionality on offer...)
please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+ private GsonSerDe<JobSpec> gsonSerDe;
Review comment:
yes, true
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
Review comment:
good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer. handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
Review comment:
I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aff832) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.01%`.
> The diff coverage is `57.11%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 46.54% +0.01%
- Complexity 10246 10301 +55
============================================
Files 2063 2069 +6
Lines 80475 80754 +279
Branches 8989 9009 +20
============================================
+ Hits 37443 37586 +143
- Misses 39567 39690 +123
- Partials 3465 3478 +13
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| ... and [30 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7aff832](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7aff832) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `2.55%`.
> The diff coverage is `57.69%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 49.08% +2.55%
+ Complexity 10246 8837 -1409
============================================
Files 2063 1688 -375
Lines 80475 65213 -15262
Branches 8989 7476 -1513
============================================
- Hits 37443 32007 -5436
+ Misses 39567 30181 -9386
+ Partials 3465 3025 -440
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| [...va/org/apache/gobblin/runtime/SourceDecorator.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU291cmNlRGVjb3JhdG9yLmphdmE=) | `41.02% <20.00%> (-0.16%)` | :arrow_down: |
| [...apache/gobblin/metrics/reporter/EventReporter.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9yZXBvcnRlci9FdmVudFJlcG9ydGVyLmphdmE=) | `56.16% <28.57%> (-3.26%)` | :arrow_down: |
| ... and [407 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7aff832](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.02%`.
> The diff coverage is `56.87%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 46.54% +0.02%
- Complexity 10246 10303 +57
============================================
Files 2063 2069 +6
Lines 80475 80758 +283
Branches 8989 9009 +20
============================================
+ Hits 37443 37592 +149
- Misses 39567 39688 +121
- Partials 3465 3478 +13
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| ... and [30 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e6b7fc0) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.14%`.
> The diff coverage is `38.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 46.67% +0.14%
+ Complexity 10246 3173 -7073
============================================
Files 2063 654 -1409
Lines 80475 25781 -54694
Branches 8989 3061 -5928
============================================
- Hits 37443 12033 -25410
+ Misses 39567 12452 -27115
+ Partials 3465 1296 -2169
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `33.33% <0.00%> (+1.10%)` | :arrow_up: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...lin/service/modules/flow/MultiHopFlowCompiler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L011bHRpSG9wRmxvd0NvbXBpbGVyLmphdmE=) | `69.87% <42.85%> (-0.83%)` | :arrow_down: |
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `67.04% <49.51%> (-14.33%)` | :arrow_down: |
| [...n/service/modules/dataset/FSDatasetDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0ZTRGF0YXNldERlc2NyaXB0b3IuamF2YQ==) | `78.00% <100.00%> (+0.91%)` | :arrow_up: |
| [...dules/flowgraph/pathfinder/AbstractPathFinder.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvcGF0aGZpbmRlci9BYnN0cmFjdFBhdGhGaW5kZXIuamF2YQ==) | `87.06% <100.00%> (ø)` | |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [...gobblin/service/modules/core/GitConfigMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdENvbmZpZ01vbml0b3IuamF2YQ==) | `81.35% <0.00%> (-1.70%)` | :arrow_down: |
| ... and [1416 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...e6b7fc0](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r739012044
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
+ Properties properties;
+
+ try {
+ properties = context.deserialize(jsonObject.get(JobSpecSerializer.JOB_SPEC_CONFIG_AS_PROPERTIES_KEY), Properties.class);
+ } catch (JsonParseException e) {
+ // for backward compatibility... (is this needed for `JobSpec`, or only for (inspiration) `FlowSpec`???)
+ properties = new Properties();
+ try {
+ properties.load(new StringReader(jsonObject.get(JobSpecSerializer.JOB_SPEC_CONFIG_AS_PROPERTIES_KEY).getAsString()));
Review comment:
this originated (with the comment "for backward compatibility") in `FlowSpecDeserializer`... is it necessary here too or a historical quirk specific to the former?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
Review comment:
it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
```
Config config = ConfigBuilder.create()
.addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
.addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
...
```
what do you think?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
Review comment:
sure, will rephrase to:
```
MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
```
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onUpdateJob(jobSpec);
+ } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+ throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+ }
+ } else {
+ jobSpecStore.addSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onAddJob(jobSpec);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+ }
+ }
+
+ /**
+ * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ */
+ @Override
+ public synchronized void remove(URI jobURI) {
+ remove(jobURI, false);
+ }
+
+ /**
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ *
+ * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+ */
+ @Override
+ public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobURI);
+ try {
+ long startTime = System.currentTimeMillis();
+ JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+ jobSpecStore.deleteSpec(jobURI);
+ this.mutableMetrics.updateRemoveJobTime(startTime);
+ this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());
Review comment:
I lack the history on this one. I'm also not following what additional spec store calls arise as a result.
(overall I'm not opposed to streamlining the functionality on offer...)
please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+ private GsonSerDe<JobSpec> gsonSerDe;
Review comment:
yes, true
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
Review comment:
good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer. handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
Review comment:
I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740472936
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
This map function seems unnecessarily complicated, couldn't we just do:
```
try {
templateURI = Optional.ofNullable(new URI(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()));
} catch (URISyntaxException e) {
LOGGER.warn(String.format("error deserializing '%s' as a URI",
jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
templateURI = Optional.empty();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740665535
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
first off, I agree it's unsatisfying. (I welcome any tips toward finessing java's checked exceptions!)
the transformation you suggest doesn't protect against passing `null` to the `URI` ctor. that's why I was working (via `map`) *within* the `Optional`. still, you did helped spot a NPE bug from the parens closed in the wrong place. we need:
```
Optional.ofNullable(jsonObj.get(...)).map
```
not the potentially explosive:
```
Optional.ofNullable(jsonObj.get(...).getAsString()).map
```
in addition, I'll use `Optional.flatMap` to eliminate the second level of exception handling. I'll push those corrections, and you please tell me if you see any other potential improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741592138
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
Review comment:
it's named after its purpose/function, to read clearly when used; e.g. from the unit test:
```
Config config = ConfigBuilder.create()
.addPrimitive(ConfigurationKeys.METRICS_ENABLED_KEY, "true")
.addPrimitive(MysqlJobCatalog.DB_CONFIG_PREFIX + "." + ConfigurationKeys.STATE_STORE_DB_URL_KEY, testDb.getJdbcUrl())
...
```
what do you think?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
Review comment:
sure, will rephrase to:
```
MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information, which fully supports (mutation) listeners and metrics.
```
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onUpdateJob(jobSpec);
+ } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+ throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+ }
+ } else {
+ jobSpecStore.addSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onAddJob(jobSpec);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+ }
+ }
+
+ /**
+ * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ */
+ @Override
+ public synchronized void remove(URI jobURI) {
+ remove(jobURI, false);
+ }
+
+ /**
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ *
+ * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+ */
+ @Override
+ public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobURI);
+ try {
+ long startTime = System.currentTimeMillis();
+ JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+ jobSpecStore.deleteSpec(jobURI);
+ this.mutableMetrics.updateRemoveJobTime(startTime);
+ this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());
Review comment:
I lack the history on this one. I'm also not following what additional spec store calls arise as a result.
(overall I'm not opposed to streamlining the functionality on offer...)
please either provide more background context, or, if that effort would be a separate follow-on, we could save the discussion for later.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+ private GsonSerDe<JobSpec> gsonSerDe;
Review comment:
yes, true
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
Review comment:
good suggestion... I guess I took the prior impl at face value, but I was able to re-work w/ generics into an abstract base class doing the heavy lifting.
since java lacks a type class resolution capability, we still need to create a specific, named derivation to supply the serializer and the deserializer. handily, that non-generic class name also imparts the ability to refer to the type within HOCON or even simple properties-based config.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
Review comment:
I wanted to conclude the timing prior to listener dispatching, especially since that's a `synchronized` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] jack-moseley commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r742355943
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
Okay I see, yeah I think the current way looks fine, it is good to not have the two levels of exception handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 merged pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 merged pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
phet commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r740665535
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/JobSpecDeserializer.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.gson.JsonDeserializationContext;
+import com.google.gson.JsonDeserializer;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParseException;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.reflect.Type;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Optional;
+import java.util.Properties;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class JobSpecDeserializer implements JsonDeserializer<JobSpec> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(JobSpecDeserializer.class);
+
+ @Override
+ public JobSpec deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) {
+ JsonObject jsonObject = json.getAsJsonObject();
+
+ String uri = jsonObject.get(JobSpecSerializer.JOB_SPEC_URI_KEY).getAsString();
+ String version = jsonObject.get(JobSpecSerializer.JOB_SPEC_VERSION_KEY).getAsString();
+ String description = jsonObject.get(JobSpecSerializer.JOB_SPEC_DESCRIPTION_KEY).getAsString();
+ Optional<URI> templateURI;
+ try {
+ templateURI = Optional.ofNullable(jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()).map(s -> {
+ try {
+ return new URI(s);
+ } catch (URISyntaxException e) {
+ LOGGER.warn(String.format("error deserializing '%s' as a URI",
+ jsonObject.get(JobSpecSerializer.JOB_SPEC_TEMPLATE_URI_KEY).getAsString()), e);
+ throw new RuntimeException(e);
+ }
+ });
+ } catch (RuntimeException e) {
+ templateURI = Optional.empty();
+ }
Review comment:
first off, I agree it's unsatisfying. (I welcome any tips toward finessing java's checked exceptions!)
the transformation you suggest doesn't protect against passing `null` to the `URI` ctor. that's why I was working (via `map`) *within* the `Optional`. still, you did helped spot a NPE bug from the parens closed in the wrong place. we need:
```
Optional.ofNullable(jsonObj.get(...)).map
```
not the potentially explosive:
```
Optional.ofNullable(jsonObj.get(...).getAsString()).map
```
in addition, I'll use `Optional.flatMap` to eliminate the second level of exception handling. I'll push those corrections, and you please tell me if you see any other potential improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#discussion_r741274525
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
Review comment:
This name does not match the value it holds, right? how about MYSQL_JOB_CATALOG or similar?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
Review comment:
{@link JobSpec}
nit *supports
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
+ private GsonSerDe<JobSpec> gsonSerDe;
Review comment:
could be `final`
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
Review comment:
This line could be out of if else ?
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/job_catalog/MysqlJobCatalog.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.runtime.job_catalog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Iterators;
+import com.typesafe.config.Config;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.gobblin.metrics.GobblinMetrics;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
+import org.apache.gobblin.runtime.api.JobCatalog;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.JobSpecNotFoundException;
+import org.apache.gobblin.runtime.api.MutableJobCatalog;
+import org.apache.gobblin.runtime.api.SpecNotFoundException;
+import org.apache.gobblin.runtime.spec_serde.GsonJobSpecSerDe;
+import org.apache.gobblin.runtime.spec_store.MysqlBaseSpecStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * MySQL-backed Job Catalog for persisting (`JobSpec`) job configuration information. Fully support (mutation) listeners and metrics.
+ */
+public class MysqlJobCatalog extends JobCatalogBase implements MutableJobCatalog {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(MysqlJobCatalog.class);
+ public static final String DB_CONFIG_PREFIX = "mysqlJobCatalog";
+
+ protected final MutableJobCatalog.MutableStandardMetrics mutableMetrics;
+ protected final MysqlBaseSpecStore jobSpecStore;
+
+ /**
+ * Initialize, with DB config contextualized by `DB_CONFIG_PREFIX`.
+ */
+ public MysqlJobCatalog(Config sysConfig)
+ throws IOException {
+ this(sysConfig, Optional.<MetricContext>absent(), GobblinMetrics.isEnabled(sysConfig));
+ }
+
+ public MysqlJobCatalog(GobblinInstanceEnvironment env) throws IOException {
+ super(env);
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(env.getSysConfig().getConfig());
+ }
+
+ public MysqlJobCatalog(Config sysConfig, Optional<MetricContext> parentMetricContext,
+ boolean instrumentationEnabled) throws IOException {
+ super(Optional.of(LOGGER), parentMetricContext, instrumentationEnabled, Optional.of(sysConfig));
+ this.mutableMetrics = (MutableJobCatalog.MutableStandardMetrics) metrics;
+ this.jobSpecStore = createJobSpecStore(sysConfig);
+ }
+
+ @Override
+ protected JobCatalog.StandardMetrics createStandardMetrics(Optional<Config> sysConfig) {
+ log.info("create standard metrics {} for {}", MutableJobCatalog.MutableStandardMetrics.class.getName(), this.getClass().getName());
+ return new MutableJobCatalog.MutableStandardMetrics(this, sysConfig);
+ }
+
+ protected MysqlBaseSpecStore createJobSpecStore(Config sysConfig) {
+ try {
+ return new MysqlBaseSpecStore(sysConfig, new GsonJobSpecSerDe()) {
+ @Override
+ protected String getConfigPrefix() {
+ return MysqlJobCatalog.DB_CONFIG_PREFIX;
+ }
+ };
+ } catch (IOException e) {
+ throw new RuntimeException("unable to create `JobSpec` store", e);
+ }
+ }
+
+ /** @return all {@link JobSpec}s */
+ @Override
+ public List<JobSpec> getJobs() {
+ try {
+ return (List) jobSpecStore.getSpecs();
+ } catch (IOException e) {
+ throw new RuntimeException("error getting (all) job specs", e);
+ }
+ }
+
+ /**
+ * Obtain an iterator to fetch all job specifications. Unlike {@link #getJobs()}, this method avoids loading
+ * all job configs into memory in the very beginning.
+ * Interleaving notes: jobs added/modified/deleted between `Iterator` creation and exhaustion MAY or MAY NOT be reflected.
+ *
+ * @return an iterator for (all) present {@link JobSpec}s
+ */
+ @Override
+ public Iterator<JobSpec> getJobSpecIterator() {
+ try {
+ return Iterators.<Optional<JobSpec>, JobSpec>transform(
+ Iterators.filter(
+ Iterators.transform(jobSpecStore.getSpecURIs(), uri -> {
+ try {
+ return Optional.of(MysqlJobCatalog.this.getJobSpec(uri));
+ } catch (JobSpecNotFoundException e) {
+ MysqlJobCatalog.this.log.info("unable to retrieve previously identified JobSpec by URI '{}'", uri);
+ return Optional.absent();
+ }}),
+ Optional::isPresent),
+ Optional::get);
+ } catch (IOException e) {
+ throw new RuntimeException("error iterating (all) job specs", e);
+ }
+ }
+
+ /**
+ * Fetch single {@link JobSpec} by URI.
+ * @return the `JobSpec`
+ * @throws {@link JobSpecNotFoundException}
+ */
+ @Override
+ public JobSpec getJobSpec(URI uri)
+ throws JobSpecNotFoundException {
+ Preconditions.checkNotNull(uri);
+ try {
+ return (JobSpec) jobSpecStore.getSpec(uri);
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error accessing job spec '%s'", uri), e);
+ } catch (SpecNotFoundException e) {
+ throw new JobSpecNotFoundException(uri);
+ }
+ }
+
+ /**
+ * Add or update (when an existing) {@link JobSpec}, triggering the appropriate
+ * {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` (w/ `remove()`) for integrity of (existence) check-then-update.
+ */
+ @Override
+ public synchronized void put(JobSpec jobSpec) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobSpec);
+ try {
+ long startTime = System.currentTimeMillis();
+ boolean isUpdate = jobSpecStore.exists(jobSpec.getUri());
+ if (isUpdate) {
+ try {
+ jobSpecStore.updateSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onUpdateJob(jobSpec);
+ } catch (SpecNotFoundException e) { // should never happen (since `synchronized`)
+ throw new RuntimeException(String.format("error finding spec to update '%s'", jobSpec.getUri()), e);
+ }
+ } else {
+ jobSpecStore.addSpec(jobSpec);
+ this.mutableMetrics.updatePutJobTime(startTime);
+ this.listeners.onAddJob(jobSpec);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(String.format("error updating or adding JobSpec '%s'", jobSpec.getUri()), e);
+ }
+ }
+
+ /**
+ * Delete (an existing) {@link JobSpec}, triggering the appropriate {@link org.apache.gobblin.runtime.api.JobCatalogListener} callback.
+ *
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ */
+ @Override
+ public synchronized void remove(URI jobURI) {
+ remove(jobURI, false);
+ }
+
+ /**
+ * NOTE: `synchronized` w/ `put()` to protect its check-then-update.
+ *
+ * @param alwaysTriggerListeners whether invariably to trigger {@link org.apache.gobblin.runtime.api.JobCatalogListener#onCancelJob(URI)}
+ */
+ @Override
+ public synchronized void remove(URI jobURI, boolean alwaysTriggerListeners) {
+ Preconditions.checkState(state() == State.RUNNING, String.format("%s is not running.", this.getClass().getName()));
+ Preconditions.checkNotNull(jobURI);
+ try {
+ long startTime = System.currentTimeMillis();
+ JobSpec jobSpec = (JobSpec) jobSpecStore.getSpec(jobURI);
+ jobSpecStore.deleteSpec(jobURI);
+ this.mutableMetrics.updateRemoveJobTime(startTime);
+ this.listeners.onDeleteJob(jobURI, jobSpec.getVersion());
Review comment:
So it seems supporting a feature (spec versioning) which may have bugs, and is not being used is costing us more spec store calls. Should we remove this in future, or at least provide a version of APIs without the version (that passes to the existing API with a default version)
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_serde/GsonJobSpecSerDe.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.runtime.spec_serde;
+
+import com.google.common.base.Charsets;
+import com.google.gson.Gson;
+import com.google.gson.JsonParseException;
+import com.google.gson.reflect.TypeToken;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecSerDe;
+import org.apache.gobblin.runtime.api.SpecSerDeException;
+
+
+/**
+ * {@link SpecSerDe} that serializes as Json using {@link Gson}. Note that currently only {@link JobSpec}s are supported.
+ */
+public class GsonJobSpecSerDe implements SpecSerDe {
Review comment:
There must be a way to reuse `GsonFlowSpecSerDe` , can we use generics in the interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3421: [GOBBLIN-1569] Add RDBMS-backed `MysqlJobCatalog`, as alternative to file system storage
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3421:
URL: https://github.com/apache/gobblin/pull/3421#issuecomment-954526855
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3421](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7985803) into [master](https://codecov.io/gh/apache/gobblin/commit/384dc096da8b7f05447be68fa001fcba39d64276?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (384dc09) will **increase** coverage by `0.01%`.
> The diff coverage is `57.43%`.
[](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3421 +/- ##
============================================
+ Coverage 46.52% 46.54% +0.01%
- Complexity 10246 10300 +54
============================================
Files 2063 2070 +7
Lines 80475 80751 +276
Branches 8989 9009 +20
============================================
+ Hits 37443 37584 +141
- Misses 39567 39690 +123
- Partials 3465 3477 +12
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...org/apache/gobblin/stream/WorkUnitChangeEvent.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1dvcmtVbml0Q2hhbmdlRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...che/gobblin/cluster/FsJobConfigurationManager.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvRnNKb2JDb25maWd1cmF0aW9uTWFuYWdlci5qYXZh) | `66.66% <0.00%> (-4.45%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `28.26% <0.00%> (-3.98%)` | :arrow_down: |
| [...ava/org/apache/gobblin/metastore/ZkStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZS5qYXZh) | `63.06% <ø> (ø)` | |
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.11% <0.00%> (-2.00%)` | :arrow_down: |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | `75.22% <ø> (ø)` | |
| [...apache/gobblin/runtime/spec\_store/FSSpecStore.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19zdG9yZS9GU1NwZWNTdG9yZS5qYXZh) | `51.75% <ø> (ø)` | |
| [.../java/org/apache/gobblin/runtime/api/FlowSpec.java](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0Zsb3dTcGVjLmphdmE=) | `40.93% <7.69%> (-2.72%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/apache/gobblin/pull/3421/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [384dc09...7985803](https://codecov.io/gh/apache/gobblin/pull/3421?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org