[GitHub] [gobblin] Will-Lo commented on pull request #3667: [GOBBLIN-1806] Submit dataset summary event post commit and integrate them into GaaSObservabilityEvent

["Will-Lo (via GitHub)" <gi...@apache.org>]

Will-Lo commented on PR #3667:
URL: https://github.com/apache/gobblin/pull/3667#issuecomment-1500676439

   Responding to the top level comment: `would we wish to pre-aggregate that within the event itself`, I think aggregation is needed because there can be thousands of tasks for large pipelines, which makes serializing all the states into an event lead to large Kafka events which we want to avoid. Also, I think most clients wouldn't care too much about the inner details of every individual task/mapper, it's mainly the concern of the Gobblin framework to deal with correctly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org