You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/09 23:04:32 UTC
[GitHub] [beam] TheNeuralBit commented on a change in pull request #13066: [BEAM-11052] Memoize to_pcollection
TheNeuralBit commented on a change in pull request #13066:
URL: https://github.com/apache/beam/pull/13066#discussion_r502706227
##########
File path: sdks/python/apache_beam/dataframe/convert.py
##########
@@ -67,6 +68,9 @@ def to_dataframe(
expressions.PlaceholderExpression(proxy, pcoll))
+TO_PCOLLECTION_CACHE = {}
Review comment:
Note this uses a global cache which can be problematic in testing. I don't think we need to worry about pcollections being shared across pipelines though since the dataframe expressions should have a reference to the pipeline through the to_dataframe roots.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org