[GitHub] [beam] KevinGG commented on a change in pull request #15146: [BEAM-12388] Add caching to deferred dataframes

[GitBox <gi...@apache.org>]

KevinGG commented on a change in pull request #15146:
URL: https://github.com/apache/beam/pull/15146#discussion_r669148330



##########
File path: sdks/python/apache_beam/runners/interactive/utils.py
##########
@@ -267,3 +270,17 @@ def return_as_json(*args, **kwargs):
       return str(return_value)
 
   return return_as_json
+
+
+def deferred_df_to_pcollection(df):
+  assert isinstance(df, DeferredBase), '{} is not a DeferredBase'.format(df)
+
+  # The proxy is used to output a DataFrame with the correct columns.
+  #
+  # TODO(BEAM-11064): Once type hints are implemented for pandas, use those
+  # instead of the proxy.
+  cache = ExpressionCache()
+  cache.replace_with_cached(df._expr)
+
+  proxy = df._expr.proxy()
+  return to_pcollection(df, yield_elements='pandas', label=str(df._expr)), proxy

Review comment:
       Is the proxy basically the `pcoll.element_type`? Will this be redundant?

##########
File path: sdks/python/apache_beam/runners/interactive/caching/expression_cache.py
##########
@@ -0,0 +1,90 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import *
+
+import apache_beam as beam
+from apache_beam.dataframe import convert
+from apache_beam.dataframe import expressions
+
+
+class ExpressionCache(object):

Review comment:
       Is the `ExpressionCache` to be used anywhere besides the `deferred_df_to_pcollection`? Can `replace_with_cached` be part of the constructor? If not, can we document how to use `ExpressionCache`? Like the lifecycle of a ExpressionCache. When to create a new `ExpressionCache`? When to invoke `replace_with_cached`? What are the side effects to the given expression and the interactive environment? How long does the side effects last?
   
   I'm wondering how the cache is persisted between multiple runs? Or is it just a temporary helper class to alter the given expression.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org