[GitHub] [gobblin] phet commented on pull request #3667: [GOBBLIN-1806] Submit dataset summary event post commit and integrate them into GaaSObservabilityEvent

["phet (via GitHub)" <gi...@apache.org>]

phet commented on PR #3667:
URL: https://github.com/apache/gobblin/pull/3667#issuecomment-1503758833

   > Responding to the top level comment: `would we wish to pre-aggregate that within the event itself`, I think aggregation is needed because there can be thousands of tasks for large pipelines, which makes serializing all the states into an event lead to large Kafka events which we want to avoid. Also, I think most clients wouldn't care too much about the inner details of every individual task/mapper, it's mainly the concern of the Gobblin framework to deal with correctly.
   
   does this mean you'll add a summary field to aggregate measurement across all datasets?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org