[GitHub] [beam] rohdesamuel commented on a change in pull request #15146: [BEAM-12388] Add caching to deferred dataframes

[GitBox <gi...@apache.org>]

rohdesamuel commented on a change in pull request #15146:
URL: https://github.com/apache/beam/pull/15146#discussion_r670016061



##########
File path: sdks/python/apache_beam/runners/interactive/utils.py
##########
@@ -267,3 +270,17 @@ def return_as_json(*args, **kwargs):
       return str(return_value)
 
   return return_as_json
+
+
+def deferred_df_to_pcollection(df):
+  assert isinstance(df, DeferredBase), '{} is not a DeferredBase'.format(df)
+
+  # The proxy is used to output a DataFrame with the correct columns.
+  #
+  # TODO(BEAM-11064): Once type hints are implemented for pandas, use those
+  # instead of the proxy.
+  cache = ExpressionCache()
+  cache.replace_with_cached(df._expr)
+
+  proxy = df._expr.proxy()
+  return to_pcollection(df, yield_elements='pandas', label=str(df._expr)), proxy

Review comment:
       No not yet, the proxy is different than the pcoll.element_type. The proxy is a tool used in the Beam DataFrames implementation to keep track of the column names and index. Once the TODO above is fixed, however, then it can be replaced with the pcoll.element_type.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org