You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/08/05 15:38:00 UTC
[jira] [Work logged] (GOBBLIN-2118) Reduce no of network calls while fetching kafka offsets during startup
[ https://issues.apache.org/jira/browse/GOBBLIN-2118?focusedWorklogId=928739&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-928739 ]
ASF GitHub Bot logged work on GOBBLIN-2118:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 05/Aug/24 15:37
Start Date: 05/Aug/24 15:37
Worklog Time Spent: 10m
Work Description: codecov-commenter commented on PR #4009:
URL: https://github.com/apache/gobblin/pull/4009#issuecomment-2269366851
## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4009?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 41.10%. Comparing base [(`b44a91e`)](https://app.codecov.io/gh/apache/gobblin/commit/b44a91ea69dded11fc2229237026a575ec94603a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`370b5fb`)](https://app.codecov.io/gh/apache/gobblin/commit/370b5fbd01516cdc59dc79a46e5abd03afce8b29?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 20 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4009 +/- ##
============================================
+ Coverage 38.72% 41.10% +2.37%
- Complexity 1596 2197 +601
============================================
Files 388 479 +91
Lines 15957 20353 +4396
Branches 1578 2354 +776
============================================
+ Hits 6180 8366 +2186
- Misses 9283 11097 +1814
- Partials 494 890 +396
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/4009?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
Issue Time Tracking
-------------------
Worklog Id: (was: 928739)
Remaining Estimate: 0h
Time Spent: 10m
> Reduce no of network calls while fetching kafka offsets during startup
> ----------------------------------------------------------------------
>
> Key: GOBBLIN-2118
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2118
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arpit Varshney
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> During starting while creating work unit, in Kafkasource there are network calls that tries to fetch the kafka offsets (both earliest and latest) to find out the watermark (to find the offsets where the gobblin job will start consuming from)
> These calls are fetched for each topic and each partition in the topic. For each partition, there is a separate call that goes to kafka client, which increases the no of network calls. If there are cross colo calls (calls to different datacenters in different regions) this increase the time to fetch and results in timeout which leads to skipping of the topic partition to fetch leading to starvation.
> This ticket targets to reduce the no of network calls, rather than doing a call for each partition. Utilize kafka source to fetch the offsets for all the paritions at once from kafka.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)