Re: [I] Bug MERGE INTO PySpark with UDF causes: `Cannot generate code for expression` [iceberg]

["94Sip (via GitHub)" <gi...@apache.org>]

94Sip commented on issue #7656:
URL: https://github.com/apache/iceberg/issues/7656#issuecomment-1945247209

   I just had the same error with v1.4.3 (with AWS Glue) I tried the foreach recommendation and that created the following error:
   
   279; PicklingError: Could not serialize object: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063
   
   I tried to use collect() on the dataframe before the foreach method and that didn't resolve the problem either.
   
   In the end, I used the cache() method to resolve the problem.
   
   So I still think this is a bug with Icerberg v1.X


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org