You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/12 16:28:44 UTC
[GitHub] [beam] lukecwik commented on a change in pull request #11060:
[BEAM-9454] Create Deduplication transform based on user timer/state
lukecwik commented on a change in pull request #11060: [BEAM-9454] Create Deduplication transform based on user timer/state
URL: https://github.com/apache/beam/pull/11060#discussion_r391728602
##########
File path: sdks/python/apache_beam/runners/sdf_utils.py
##########
@@ -244,3 +251,63 @@ def get_estimator_state(self):
return None
return _NoOpWatermarkEstimator()
+
+
+class DeduplictaionWithinDuration(ptransform.PTransform):
Review comment:
It would be useful to expose a keyed deduplication transform as the common implementation that all use internally so in the future we can turn into a well known URN and then runners could provide optimized deduplication transform implementations.
We want pipeline authors to use this transform and I think it should go into sdks/python/apache_beam/transforms/util.py or into a dedicated file such as sdks/python/apache_beam/transforms/ such as deduplicate.py.
CC: @udim What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services