You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2020/06/09 20:13:16 UTC
[GitHub] [incubator-gobblin] arekusuri opened a new pull request #3034: GOBBLIN-1186: explicitly set source.querybased.salesforce.is.soft.del…
arekusuri opened a new pull request #3034:
URL: https://github.com/apache/incubator-gobblin/pull/3034
…etes.pull.disabled for simple mode
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- https://issues.apache.org/jira/browse/GOBBLIN-1186
### Description
Problem statement
source.querybased.salesforce.is.soft.deletes.pull.disabled
doesn’t work for simple mode, it works only for dynamic mode.
the reason is - we explicitly set up the key-value for the dynamic mode
https://github.com/hanghangliu/gobblin/blob/9029a89b85ef373f78d603b14d6aaa75998f3356/gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java#L327
Root cause
The extract state is blank(please see code)
What we set up in job file is not able to see in extractor state.
https://github.com/hashdoop/hashdoop-incubator-gobblin/blob/a871e5c5d6f539bcfbcc4e2850685c58dd72dd1a/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java#L234
Solution:
explicitly set up the soft.deletes.pull.disabled for simple mode, as we did for dynamic mode
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] asfgit closed pull request #3034: GOBBLIN-1186: explicitly set source.querybased.salesforce.is.soft.del…
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3034:
URL: https://github.com/apache/incubator-gobblin/pull/3034
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter edited a comment on pull request #3034: GOBBLIN-1186: explicitly set source.querybased.salesforce.is.soft.del…
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3034:
URL: https://github.com/apache/incubator-gobblin/pull/3034#issuecomment-641618520
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=h1) Report
> Merging [#3034](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/31f2ef1478c3b5d4887cfc772f0e85057cde1c56&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3034 +/- ##
============================================
+ Coverage 45.66% 45.68% +0.01%
- Complexity 9296 9301 +5
============================================
Files 1956 1956
Lines 74397 74400 +3
Branches 8247 8247
============================================
+ Hits 33977 33986 +9
+ Misses 37242 37238 -4
+ Partials 3178 3176 -2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh) | `20.83% <100.00%> (+1.01%)` | `13.00 <2.00> (+1.00)` | |
| [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `47.57% <0.00%> (-1.95%)` | `12.00% <0.00%> (ø%)` | |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0.00%> (+0.93%)` | `24.00% <0.00%> (ø%)` | |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `40.47% <0.00%> (+3.17%)` | `16.00% <0.00%> (+1.00%)` | |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `100.00% <0.00%> (+7.14%)` | `4.00% <0.00%> (+1.00%)` | |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `60.00% <0.00%> (+20.00%)` | `3.00% <0.00%> (+1.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=footer). Last update [31f2ef1...cc99036](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter commented on pull request #3034: GOBBLIN-1186: explicitly set source.querybased.salesforce.is.soft.del…
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3034:
URL: https://github.com/apache/incubator-gobblin/pull/3034#issuecomment-641618520
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=h1) Report
> Merging [#3034](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/31f2ef1478c3b5d4887cfc772f0e85057cde1c56&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3034 +/- ##
============================================
+ Coverage 45.66% 45.68% +0.01%
- Complexity 9296 9301 +5
============================================
Files 1956 1956
Lines 74397 74400 +3
Branches 8247 8247
============================================
+ Hits 33977 33986 +9
+ Misses 37242 37238 -4
+ Partials 3178 3176 -2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...rg/apache/gobblin/salesforce/SalesforceSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZVNvdXJjZS5qYXZh) | `20.83% <100.00%> (+1.01%)` | `13.00 <2.00> (+1.00)` | |
| [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `47.57% <0.00%> (-1.95%)` | `12.00% <0.00%> (ø%)` | |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0.00%> (+0.93%)` | `24.00% <0.00%> (ø%)` | |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `40.47% <0.00%> (+3.17%)` | `16.00% <0.00%> (+1.00%)` | |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `100.00% <0.00%> (+7.14%)` | `4.00% <0.00%> (+1.00%)` | |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3034/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `60.00% <0.00%> (+20.00%)` | `3.00% <0.00%> (+1.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=footer). Last update [31f2ef1...cc99036](https://codecov.io/gh/apache/incubator-gobblin/pull/3034?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org