Re: [I] Avro reader memory leak [iceberg-python]

["andormarkus (via GitHub)" <gi...@apache.org>]

andormarkus commented on issue #2325:
URL: https://github.com/apache/iceberg-python/issues/2325#issuecomment-3625295829

   Hi @thomas-pfeiffer,
   
   Thanks for sharing your workaround! I'm curious about your specific scenario that led you to disable the cache completely.
   
   In my case, I implemented cache clearing at invocation boundaries:
   1. **Init step** - Clear cache at the beginning of Lambda execution
   2. **Post-execution step** - Clear cache after completing the operation
   
   This approach gives me:
   - **During execution**: Full benefit of the cache (faster performance)
   - **Between invocations**: Memory growth is limited/bounded
   
   With this strategy, my Lambda memory stabilizes at 1600-1800MB instead of continuously growing to OOM, while still maintaining cache performance benefits within each execution. I'm running with 2048MB memory allocation, so this 10-20% overhead is acceptable in our environment.
   
   Could you share more details about your scenario? 
   - What led you to disable the cache entirely rather than clearing at invocation boundaries?
   - Did you try clearing at execution start/end and find it insufficient?
   - Are your Lambda executions particularly long-running or processing very large manifest lists?
   - What memory allocation are you working with?
   
   Understanding your use case would help the community find the right balance between performance and memory stability.
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org