[PR] AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak [avro]

["iemejia (via GitHub)" <gi...@apache.org>]

iemejia opened a new pull request, #3963:
URL: https://github.com/apache/avro/pull/3963

   ## What changes were proposed in this pull request?
   
   `FastReaderBuilder`'s `RecordReader` cache was a weak-identity map keyed on the reader `Schema`:
   
   ```java
   private final Map<Schema, Map<Schema, RecordReader>> readerCache =
       Collections.synchronizedMap(new WeakIdentityHashMap<>());
   ```
   
   However, each cached `RecordReader` holds a **strong** reference to that same reader `Schema` (`RecordReader.schema`, needed by its `InstanceSupplier` at read time — `getNewRecordSupplier` returns `this::newRecord`, which takes the schema as a parameter). Because the cached **value** strongly references its own **weak key**, the weak entries can never be reclaimed. The cache therefore grows without bound whenever many distinct `Schema` instances are used — e.g. when a schema is re-parsed for every file or message in a long-running process — leading to unbounded memory growth (reports of caches reaching 100GB).
   
   This is the memory leak tracked by AVRO-4253 (and its duplicate AVRO-3524). Note the earlier commit referencing AVRO-4253 (#3764) actually addressed an unrelated logging concern; the leak itself was still present.
   
   ## How was this patch fixed?
   
   Replace the weak two-level map with a **bounded LRU cache** keyed on an identity-based `(reader, writer)` schema pair:
   
   - Bounds memory unconditionally, independent of the value-references-key cycle.
   - Identity-based key preserves the original cache semantics and avoids the cost of `Schema.equals`/`hashCode` on large schemas.
   - **Recursion-safe:** entries that are still `INITIALIZING` are never evicted, so recursive schema resolution still terminates by resolving back to the same in-flight instance instead of rebuilding endlessly.
   - Evicting a cached reader is safe for in-use readers: the caller holds the returned `DatumReader` directly, so eviction only means a future lookup rebuilds it.
   - The bound defaults to `2048` and is configurable via the `org.apache.avro.fastreader.recordReaderCacheSize` system property.
   
   ## How was this patch tested?
   
   New `TestFastReaderBuilderCacheBounded`:
   - `cacheStaysBoundedAcrossDistinctSchemaInstances` — 3048 freshly-parsed (distinct-identity) schemas leave the cache bounded at `<= 2048` (fails on the old unbounded code).
   - `reusedSchemaInstanceHitsCache` — a reused schema instance keeps a single cache entry (cache still effective).
   - `recursiveSchemaReadsCorrectly` — a self-referential (linked-list) schema round-trips correctly.
   
   Also verified no regressions across the read path: `TestResolvingIO` (816), `TestResolvingIOResolving` (192), `TestGenericDatumReader`, `TestGenericData`, `TestDataFile`, `TestResolver`, and the FastReaderBuilder tests all pass under the default, custom-coders, and without-fast-reader surefire profiles.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak [avro]

["ganeshashree (via GitHub)" <gi...@apache.org>]

ganeshashree commented on PR #3963:
URL: https://github.com/apache/avro/pull/3963#issuecomment-6010513758

   Spark 4.1.0, which uses Avro 1.12.1, is affected by this issue. Avro 1.12.1 enables the fast reader by default.  This affects long-running jobs, such as Spark Structured Streaming, that read many Avro files with distinct writer schemas, due to the accumulation of reader objects on the executor heap. 
   
   @RyanSkraba @iemejia Could we have this reviewed and merged soon?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] AVRO-4253: [java] Bound FastReaderBuilder record-reader cache to fix memory leak [avro]

["ganeshashree (via GitHub)" <gi...@apache.org>]

ganeshashree commented on PR #3963:
URL: https://github.com/apache/avro/pull/3963#issuecomment-6010556422

   @iemejia Please review and fix the CI failures. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org