You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2020/11/25 20:23:27 UTC
[GitHub] [incubator-gobblin] Will-Lo opened a new pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Will-Lo opened a new pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io edited a comment on pull request #3158: [GOBBLIN-1377] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#issuecomment-733942602
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=h1) Report
> Merging [#3158](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=desc) (cd94d5b) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f6e063245df709c6981226a98129087cef0f9d4f?el=desc) (f6e0632) will **decrease** coverage by `37.02%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3158 +/- ##
============================================
- Coverage 46.18% 9.16% -37.03%
+ Complexity 9771 1737 -8034
============================================
Files 2019 2021 +2
Lines 77529 77607 +78
Branches 8608 8617 +9
============================================
- Hits 35809 7109 -28700
- Misses 38386 69804 +31418
+ Partials 3334 694 -2640
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `0.00% <0.00%> (-55.56%)` | `0.00 <0.00> (-36.00)` | |
| [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.00% <0.00%> (-0.94%)` | `0.00 <0.00> (-1.00)` | |
| [...ement/copy/hive/NoopHiveTargetDirectoryClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9Ob29wSGl2ZVRhcmdldERpcmVjdG9yeUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...org/apache/gobblin/ack/BasicAckableForTesting.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Jhc2ljQWNrYWJsZUZvclRlc3RpbmcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...n/java/org/apache/gobblin/salesforce/SfConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2ZDb25maWcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| ... and [1062 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=footer). Last update [f6e0632...cd94d5b](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] Will-Lo closed pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
Will-Lo closed pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io edited a comment on pull request #3158: [GOBBLIN-1377] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#issuecomment-733942602
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=h1) Report
> Merging [#3158](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=desc) (b23740c) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f6e063245df709c6981226a98129087cef0f9d4f?el=desc) (f6e0632) will **decrease** coverage by `37.02%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3158 +/- ##
============================================
- Coverage 46.18% 9.16% -37.03%
+ Complexity 9771 1737 -8034
============================================
Files 2019 2020 +1
Lines 77529 77551 +22
Branches 8608 8606 -2
============================================
- Hits 35809 7108 -28701
- Misses 38386 69748 +31362
+ Partials 3334 695 -2639
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `0.00% <0.00%> (-55.56%)` | `0.00 <0.00> (-36.00)` | |
| [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.00% <0.00%> (-0.94%)` | `0.00 <0.00> (-1.00)` | |
| [...ement/copy/hive/NoopHiveTargetDirectoryClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9Ob29wSGl2ZVRhcmdldERpcmVjdG9yeUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...org/apache/gobblin/ack/BasicAckableForTesting.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Jhc2ljQWNrYWJsZUZvclRlc3RpbmcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...n/java/org/apache/gobblin/salesforce/SfConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2ZDb25maWcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| ... and [1060 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=footer). Last update [f6e0632...b23740c](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] aplex commented on a change in pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#discussion_r561210799
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java
##########
@@ -573,6 +631,8 @@ protected int addTableDeregisterSteps(List<CopyEntity> copyEntities, String file
TableDeregisterStep deregister =
new TableDeregisterStep(table.getTTable(), this.getTargetURI(), this.getHiveRegProps());
copyEntities.add(new PostPublishStep(fileSet, Maps.<String, String> newHashMap(), deregister, stepPriority++));
+ targetDirectoryClient.deletePath(table.getDataLocation());
Review comment:
A couple of things here:
- Looks like a table can have multiple paths (see above)
- The surrounding code does not actually delete files or folders, it creates steps that will be executed later on. I guess the motivation for this is to run those deletions in parallel, have retry logic, or something like that. We should follow the same pattern, and either add a step, or update existing one. Same in other places.
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java
##########
@@ -389,6 +340,106 @@
}
}
+ /**
+ * Checks {@value COPY_PARTITION_FILTER_GENERATOR} in configuration to determine which class to use for hive filtering
+ * Default is to filter based on {@value COPY_PARTITIONS_FILTER_CONSTANT}, a constant regex
+ * @throws ReflectiveOperationException if the generator class in the configuration is not found
+ */
+ private Optional<String> initializePartitionFilter() throws ReflectiveOperationException {
+ if (this.dataset.getProperties().containsKey(COPY_PARTITION_FILTER_GENERATOR)) {
+ PartitionFilterGenerator generator = GobblinConstructorUtils.invokeFirstConstructor(
+ (Class<PartitionFilterGenerator>) Class.forName(
+ this.dataset.getProperties().getProperty(COPY_PARTITION_FILTER_GENERATOR)),
+ Lists.<Object>newArrayList(this.dataset.getProperties()), Lists.newArrayList());
+ Optional<String> partitionFilter = Optional.of(generator.getFilter(this.dataset));
+ log.info(String.format("Dynamic partition filter for table %s: %s.", this.dataset.table.getCompleteName(),
+ partitionFilter.get()));
+ return partitionFilter;
+ } else {
+ return Optional.fromNullable(this.dataset.getProperties().getProperty(COPY_PARTITIONS_FILTER_CONSTANT));
+ }
+ }
+
+ /**
+ * Checks {@value HIVE_PARTITION_EXTENDED_FILTER_TYPE} in configuration to initialize more granular filtering class
+ * Default is to use none
+ * @throws ReflectiveOperationException if the filter class in the configuration is not found
+ */
+ private Optional<HivePartitionExtendedFilter> initializeExtendedPartitionFilter() throws IOException, ReflectiveOperationException {
+ // Initialize extended partition filter
Review comment:
Don't need that comment, method name covers the meaning
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveTargetDirectoryClient.java
##########
@@ -0,0 +1,36 @@
+package org.apache.gobblin.data.management.copy.hive;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.fs.Path;
+
+
+/**
+ * Determines and manages target paths for {@link HiveCopyEntityHelper}
+ * An example implementation would create target directories on cloud environments before initiating the copy
+ * By default, returns the input path (do nothing)
+ */
+public class HiveTargetDirectoryClient {
Review comment:
This can be an interface and a default implementation class.
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveTargetDirectoryClient.java
##########
@@ -0,0 +1,36 @@
+package org.apache.gobblin.data.management.copy.hive;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.fs.Path;
+
+
+/**
+ * Determines and manages target paths for {@link HiveCopyEntityHelper}
+ * An example implementation would create target directories on cloud environments before initiating the copy
+ * By default, returns the input path (do nothing)
+ */
+public class HiveTargetDirectoryClient {
+
+ public HiveTargetDirectoryClient(Properties properties){}
+
+ /**
+ * Creates a target directory path if it does not exist, otherwise fetch its path
+ * @param path initial target path
+ * @return the path after creating the target directory
+ * @throws IOException if creating or getting the path fails
+ */
+ public Path getOrCreateTargetPath(Path path) throws IOException {
+ return path;
+ }
+
+ /**
+ * Deletes references to the path from the directory client
+ * @param path path on target directory
+ * @throws IOException if deleting the path fails
+ */
+ public void deletePath(Path path) throws IOException {
Review comment:
This is a surprising behavior - we call delete and nothing happens by default. I would expect it to delete a thing.
Let's think about how this interface can be made more obvious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] aplex commented on a change in pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#discussion_r561210799
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java
##########
@@ -573,6 +631,8 @@ protected int addTableDeregisterSteps(List<CopyEntity> copyEntities, String file
TableDeregisterStep deregister =
new TableDeregisterStep(table.getTTable(), this.getTargetURI(), this.getHiveRegProps());
copyEntities.add(new PostPublishStep(fileSet, Maps.<String, String> newHashMap(), deregister, stepPriority++));
+ targetDirectoryClient.deletePath(table.getDataLocation());
Review comment:
A couple of things here:
- Looks like a table can have multiple paths (see above)
- The surrounding code does not actually delete files or folders, it creates steps that will be executed later on. I guess the motivation for this is to run those deletions in parallel, have retry logic, or something like that. We should follow the same pattern, and either add a step, or update existing one. Same in other places.
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java
##########
@@ -389,6 +340,106 @@
}
}
+ /**
+ * Checks {@value COPY_PARTITION_FILTER_GENERATOR} in configuration to determine which class to use for hive filtering
+ * Default is to filter based on {@value COPY_PARTITIONS_FILTER_CONSTANT}, a constant regex
+ * @throws ReflectiveOperationException if the generator class in the configuration is not found
+ */
+ private Optional<String> initializePartitionFilter() throws ReflectiveOperationException {
+ if (this.dataset.getProperties().containsKey(COPY_PARTITION_FILTER_GENERATOR)) {
+ PartitionFilterGenerator generator = GobblinConstructorUtils.invokeFirstConstructor(
+ (Class<PartitionFilterGenerator>) Class.forName(
+ this.dataset.getProperties().getProperty(COPY_PARTITION_FILTER_GENERATOR)),
+ Lists.<Object>newArrayList(this.dataset.getProperties()), Lists.newArrayList());
+ Optional<String> partitionFilter = Optional.of(generator.getFilter(this.dataset));
+ log.info(String.format("Dynamic partition filter for table %s: %s.", this.dataset.table.getCompleteName(),
+ partitionFilter.get()));
+ return partitionFilter;
+ } else {
+ return Optional.fromNullable(this.dataset.getProperties().getProperty(COPY_PARTITIONS_FILTER_CONSTANT));
+ }
+ }
+
+ /**
+ * Checks {@value HIVE_PARTITION_EXTENDED_FILTER_TYPE} in configuration to initialize more granular filtering class
+ * Default is to use none
+ * @throws ReflectiveOperationException if the filter class in the configuration is not found
+ */
+ private Optional<HivePartitionExtendedFilter> initializeExtendedPartitionFilter() throws IOException, ReflectiveOperationException {
+ // Initialize extended partition filter
Review comment:
Don't need that comment, method name covers the meaning
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveTargetDirectoryClient.java
##########
@@ -0,0 +1,36 @@
+package org.apache.gobblin.data.management.copy.hive;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.fs.Path;
+
+
+/**
+ * Determines and manages target paths for {@link HiveCopyEntityHelper}
+ * An example implementation would create target directories on cloud environments before initiating the copy
+ * By default, returns the input path (do nothing)
+ */
+public class HiveTargetDirectoryClient {
Review comment:
This can be an interface and a default implementation class.
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveTargetDirectoryClient.java
##########
@@ -0,0 +1,36 @@
+package org.apache.gobblin.data.management.copy.hive;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.fs.Path;
+
+
+/**
+ * Determines and manages target paths for {@link HiveCopyEntityHelper}
+ * An example implementation would create target directories on cloud environments before initiating the copy
+ * By default, returns the input path (do nothing)
+ */
+public class HiveTargetDirectoryClient {
+
+ public HiveTargetDirectoryClient(Properties properties){}
+
+ /**
+ * Creates a target directory path if it does not exist, otherwise fetch its path
+ * @param path initial target path
+ * @return the path after creating the target directory
+ * @throws IOException if creating or getting the path fails
+ */
+ public Path getOrCreateTargetPath(Path path) throws IOException {
+ return path;
+ }
+
+ /**
+ * Deletes references to the path from the directory client
+ * @param path path on target directory
+ * @throws IOException if deleting the path fails
+ */
+ public void deletePath(Path path) throws IOException {
Review comment:
This is a surprising behavior - we call delete and nothing happens by default. I would expect it to delete a thing.
Let's think about how this interface can be made more obvious.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io edited a comment on pull request #3158: [GOBBLIN-1377] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#issuecomment-733942602
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=h1) Report
> Merging [#3158](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=desc) (cd94d5b) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f6e063245df709c6981226a98129087cef0f9d4f?el=desc) (f6e0632) will **decrease** coverage by `37.02%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3158 +/- ##
============================================
- Coverage 46.18% 9.16% -37.03%
+ Complexity 9771 1737 -8034
============================================
Files 2019 2021 +2
Lines 77529 77607 +78
Branches 8608 8617 +9
============================================
- Hits 35809 7109 -28700
- Misses 38386 69804 +31418
+ Partials 3334 694 -2640
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `0.00% <0.00%> (-55.56%)` | `0.00 <0.00> (-36.00)` | |
| [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.00% <0.00%> (-0.94%)` | `0.00 <0.00> (-1.00)` | |
| [...ement/copy/hive/NoopHiveTargetDirectoryClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9Ob29wSGl2ZVRhcmdldERpcmVjdG9yeUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...org/apache/gobblin/ack/BasicAckableForTesting.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Jhc2ljQWNrYWJsZUZvclRlc3RpbmcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...n/java/org/apache/gobblin/salesforce/SfConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2ZDb25maWcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| ... and [1062 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=footer). Last update [f6e0632...cd94d5b](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] aplex commented on a change in pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#discussion_r530649551
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveTargetDirectoryClient.java
##########
@@ -0,0 +1,16 @@
+package org.apache.gobblin.data.management.copy.hive;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.fs.Path;
+
+
+public class HiveTargetDirectoryClient {
Review comment:
You can add some docs all around, explaining how this can be used and extended
##########
File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveCopyEntityHelper.java
##########
@@ -351,11 +358,21 @@
} else {
this.existingTargetTable = Optional.absent();
}
-
Review comment:
This constructor is too complex, and it's hard to follow what's going on here. You can look if it can be refactored and split into smaller functions. If the refactoring is big and touches a lot of code, you can send it as a separate PR, before you main change.
You can also install this IntelliJ plugin that will tell you if the function is too complex, right in the IDE: https://plugins.jetbrains.com/plugin/12159-codemetrics
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] Will-Lo closed pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
Will-Lo closed pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io edited a comment on pull request #3158: [GOBBLIN-1377] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#issuecomment-733942602
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=h1) Report
> Merging [#3158](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=desc) (b23740c) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f6e063245df709c6981226a98129087cef0f9d4f?el=desc) (f6e0632) will **decrease** coverage by `37.02%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3158 +/- ##
============================================
- Coverage 46.18% 9.16% -37.03%
+ Complexity 9771 1737 -8034
============================================
Files 2019 2020 +1
Lines 77529 77551 +22
Branches 8608 8606 -2
============================================
- Hits 35809 7108 -28701
- Misses 38386 69748 +31362
+ Partials 3334 695 -2639
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `0.00% <0.00%> (-55.56%)` | `0.00 <0.00> (-36.00)` | |
| [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.00% <0.00%> (-0.94%)` | `0.00 <0.00> (-1.00)` | |
| [...ement/copy/hive/NoopHiveTargetDirectoryClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9Ob29wSGl2ZVRhcmdldERpcmVjdG9yeUNsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...org/apache/gobblin/ack/BasicAckableForTesting.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Jhc2ljQWNrYWJsZUZvclRlc3RpbmcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...n/java/org/apache/gobblin/salesforce/SfConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2ZDb25maWcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| ... and [1060 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=footer). Last update [f6e0632...b23740c](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io commented on pull request #3158: [WIP] Adds functionality to create shards for target directories in hive distcp
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #3158:
URL: https://github.com/apache/incubator-gobblin/pull/3158#issuecomment-733942602
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=h1) Report
> Merging [#3158](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=desc) (28289a9) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/7352cad8ae3a1d9be10d3b6fb78383ccbada9b19?el=desc) (7352cad) will **decrease** coverage by `0.03%`.
> The diff coverage is `84.61%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3158 +/- ##
============================================
- Coverage 41.69% 41.66% -0.04%
- Complexity 8743 8751 +8
============================================
Files 1993 1998 +5
Lines 75990 76108 +118
Branches 8462 8470 +8
============================================
+ Hits 31685 31710 +25
- Misses 41297 41391 +94
+ Partials 3008 3007 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ata/management/copy/hive/HiveCopyEntityHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlQ29weUVudGl0eUhlbHBlci5qYXZh) | `55.71% <81.81%> (+0.56%)` | `36.00 <0.00> (ø)` | |
| [...anagement/copy/hive/HiveTargetDirectoryClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlVGFyZ2V0RGlyZWN0b3J5Q2xpZW50LmphdmE=) | `100.00% <100.00%> (ø)` | `2.00 <2.00> (?)` | |
| [...apache/gobblin/converter/parquet/ParquetGroup.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb252ZXJ0ZXIvcGFycXVldC9QYXJxdWV0R3JvdXAuamF2YQ==) | `45.05% <0.00%> (-2.62%)` | `16.00% <0.00%> (ø%)` | |
| [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `61.42% <0.00%> (-1.43%)` | `4.00% <0.00%> (ø%)` | |
| [...c/main/java/org/apache/gobblin/util/AvroUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvQXZyb1V0aWxzLmphdmE=) | `55.80% <0.00%> (-1.25%)` | `80.00% <0.00%> (-1.00%)` | |
| [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...ce/extractor/extract/restapi/RestApiExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9yZXN0YXBpL1Jlc3RBcGlFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `1.33% <0.00%> (ø)` | `1.00% <0.00%> (ø%)` | |
| [.../apache/gobblin/service/GroupOwnershipService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9Hcm91cE93bmVyc2hpcFNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...ava/org/apache/gobblin/util/DummyTrustManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRHVtbXlUcnVzdE1hbmFnZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| ... and [7 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3158/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=footer). Last update [7352cad...28289a9](https://codecov.io/gh/apache/incubator-gobblin/pull/3158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org