[GitHub] [iceberg] jfz commented on issue #2796: Sporadic `Resetting to invalid mark` Error

[GitBox <gi...@apache.org>]

jfz commented on issue #2796:
URL: https://github.com/apache/iceberg/issues/2796#issuecomment-1183818236

   > 
   
   
   
   > For the record, I was able to resolve this by (much in line with @otayel's suggestion):
   > 
   > 1. Reducing the number of spark executors (to 1, in my case)
   > 2. configuring the aws sdk s3 stream buffer size:
   > 
   > ```
   > spark.driver.extraJavaOptions -Dcom.amazonaws.sdk.s3.defaultStreamBufferSize=512m
   > spark.executor.extraJavaOptions -Dcom.amazonaws.sdk.s3.defaultStreamBufferSize=512m
   > ```
   
   Wondering if this actually helped? These look like settings for AWS SDK V1, but the stack trace seems from AWS SDK V2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org