You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/05/09 01:10:00 UTC
[jira] [Work logged] (GOBBLIN-1830) Improving Container Transition Tracking in Streaming Data Ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1830?focusedWorklogId=861096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-861096 ]
ASF GitHub Bot logged work on GOBBLIN-1830:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 09/May/23 01:09
Start Date: 09/May/23 01:09
Worklog Time Spent: 10m
Work Description: ZihanLi58 opened a new pull request, #3693:
URL: https://github.com/apache/gobblin/pull/3693
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1830
### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):
Currently, we rely on the flushing event to indicate when a container transition occurs. However, if the ingestion process fails to successfully flush the data, we won't know which container is responsible. This becomes problematic when the pipeline restarts, as we won't be able to identify the root cause without manually reviewing the logs of thousands of containers. To avoid this issue, we need to develop a more reliable way of tracking container transitions that doesn't rely solely on the flushing event.
The change for this PR is to emit the event when the extractor is firstly initialized
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
Unit test, and print out the event information:
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: jobName
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: KafkaHdfsStreamingTrackingOrcTest
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: helixInstance
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: GobblinYarnTaskRunner_1
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: taskAttemptId
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: GobblinYarnTaskRunner_1
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: kafkaTopic
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: PremiumInsightsNotableAlumniImpressionEvent
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: dataset.urn
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value:
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: clusterIdentifier
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: holdem
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: construct
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: Extractor
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: metricContextName
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: org.apache.gobblin.prototype.kafka.KafkaAvroBinaryStreamingExtractor.510352353
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: jobId
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: job_KafkaHdfsStreamingTrackingOrcTest_1683594035964
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: helixTaskId
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: b5ef4c8b-7b3d-47fb-9380-d92c11509050
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: partition
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: 0
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: metricContextID
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: 7355f230-dad9-450f-b8e4-4e4c6a0e0b87
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: helixJobId
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: job_KafkaHdfsStreamingTrackingOrcTest_1683594035964_job_KafkaHdfsStreamingTrackingOrcTest_1683594035964
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: topic
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: PremiumInsightsNotableAlumniImpressionEvent
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: containerNode
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: xxxx.xxx.xx.xx (remove this info as it's internal specific)
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: containerId
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: container_e32_1683322902150_274389_01_000003
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: class
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: org.apache.gobblin.prototype.kafka.KafkaAvroBinaryStreamingExtractor
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: taskId
2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: task_KafkaHdfsStreamingTrackingOrcTest_1683594035964_0
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
Issue Time Tracking
-------------------
Worklog Id: (was: 861096)
Remaining Estimate: 0h
Time Spent: 10m
> Improving Container Transition Tracking in Streaming Data Ingestion
> -------------------------------------------------------------------
>
> Key: GOBBLIN-1830
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1830
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, we rely on the flushing event to indicate when a container transition occurs. However, if the ingestion process fails to successfully flush the data, we won't know which container is responsible. This becomes problematic when the pipeline restarts, as we won't be able to identify the root cause without manually reviewing the logs of thousands of containers. To avoid this issue, we need to develop a more reliable way of tracking container transitions that doesn't rely solely on the flushing event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)