You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Lukasz Gajowy (JIRA)" <ji...@apache.org> on 2019/04/02 11:11:00 UTC
[jira] [Created] (BEAM-6969) Provide way to collect start/end
read/write time inside the IOs
Lukasz Gajowy created BEAM-6969:
-----------------------------------
Summary: Provide way to collect start/end read/write time inside the IOs
Key: BEAM-6969
URL: https://issues.apache.org/jira/browse/BEAM-6969
Project: Beam
Issue Type: Wish
Components: io-ideas, testing
Reporter: Lukasz Gajowy
Currently, IO tests measure time using Metrics API but collect start/end time from ParDo transforms that are adjacent to the IO. It's fine for some tests but maybe could be done better. The drawback of the current solution is that we cannot collect time before PBegin and after PDone. Other than that the time we collect now is still not the exact time of read/write start/end but only the time at which first/last record appeared in the DoFn.
See: [TimeMonitor.java|https://github.com/apache/beam/blob/957b7cc7746aa626d2eb4dea341f668ec19d5d39/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/TimeMonitor.java] as an example of such DoFn.
Possible solution: save metrics in startBundle / finishBundle method in IOs whenever a dedicated pipelineOption is set to true.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)