[jira] [Created] (GOBBLIN-1830) Improving Container Transition Tracking in Streaming Data Ingestion

["Zihan Li (Jira)" <ji...@apache.org>]

Zihan Li created GOBBLIN-1830:
---------------------------------

             Summary: Improving Container Transition Tracking in Streaming Data Ingestion
                 Key: GOBBLIN-1830
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1830
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Zihan Li


Currently, we rely on the flushing event to indicate when a container transition occurs. However, if the ingestion process fails to successfully flush the data, we won't know which container is responsible. This becomes problematic when the pipeline restarts, as we won't be able to identify the root cause without manually reviewing the logs of thousands of containers. To avoid this issue, we need to develop a more reliable way of tracking container transitions that doesn't rely solely on the flushing event.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)