You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Robert Yokota (Jira)" <ji...@apache.org> on 2026/10/16 00:35:00 UTC

[jira] [Created] (AVRO-4361) [Java] Collection size guard rejects valid data when the reader's element schema encodes larger than the writer's

Robert Yokota created AVRO-4361:
-----------------------------------

             Summary: [Java] Collection size guard rejects valid data when the reader's element schema encodes larger than the writer's
                 Key: AVRO-4361
                 URL: https://issues.apache.org/jira/browse/AVRO-4361
             Project: Apache Avro
          Issue Type: Bug
          Components: java
    Affects Versions: 1.12.2
            Reporter: Robert Yokota
            Assignee: Robert Yokota
             Fix For: 1.13.0, 1.12.3


The collection size guard added by AVRO-4241 (and extended to the fast reader                                                                                                     
  by AVRO-4300 and to ReflectDatumReader by AVRO-4324) computes the minimum bytes                                                                                                   
  per array element, or map value, from the reader's schema. But the elements                                                                                                       
  were encoded with the writer's schema. When the reader's element type has a                                                                                                       
  larger minimum encoded size than the writer's, the guard rejects valid data                                                                                                       
  with an EOFException.                                                                                                                                                             
                                                                                                                                                                                    
  Schema resolution allows this in several compatible cases:                                                                                                                        
                                                                                                                                                                                    
  - The reader's record adds a field with a default. Example: the writer has                                                                                                        
    Item {id: int} and the reader has Item {id: int, qty: int (default 0)}.                                                                                                         
  - Type promotion: int, long or float read as double (8 bytes in the guard),                                                                                                       
    or int or long read as float (4 bytes).                                                                                                                                         
  - A non-union writer's element read as a branch of a reader's union. Example:                                                                                                     
    array<null>, which is 0 bytes per element, read as array<["null","int"]>,                                                                                                       
    which the guard counts as at least 1 byte.                                                                                                                                      
  - The same cases for map values.                                                                                                                                                  
                                                                                                                                                                                    
  These are backward-compatible schema changes, and 1.12.1 decodes the same                                                                                                         
  bytes. The guard is only active when the decoder knows how many bytes remain                                                                                                      
  (e.g. BinaryDecoder over a byte array), which is the common case for                                                                                                              
  message-oriented use such as Kafka.

Repro (classic and fast reader both fail):                                                                                                                                        

```                                                                                                                                                                             
      String w = "{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"items\","                                                                                         
          + "\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"Item\","                                                                                        
          + "\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}}}]}";                                                                                                                 
      String r = w.replace("{\"name\":\"id\",\"type\":\"int\"}",                                                                                                                    
          "{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"qty\",\"type\":\"int\",\"default\":0}");                                                                                  
      Schema writer = new Schema.Parser().parse(w);                                                                                                                                 
      Schema reader = new Schema.Parser().parse(r);                                                                                                                                 
      // write an Order with 100 Items, each with id = 0 (1 byte per element)                                                                                                       
      ...                                                                                                                                                                           
      new GenericDatumReader<>(writer, reader).read(null, DecoderFactory.get().binaryDecoder(bytes, null));      
```                                                                   
                                                                                                                                                                                    
  1.12.1: decodes 100 items.                                                                                                                                                        
  1.12.2: java.io.EOFException: Collection claims 100 elements with at least 2 bytes                                                                                                
          each, but only 101 bytes are available                                                                                                                                    
                                                                                                                                                                                    
  Cause:                                                                                                                                                                            
  - GenericDatumReader.readArray/readMap and ReflectDatumReader.readArray pass the                                                                                                  
    reader's element schema (expected.getElementType()) to                                                                                                                          
    ensureAvailableCollectionBytes.                                                                                                                                                 
  - FastReaderBuilder.createArrayReader does the same with                                                                                                                          
    readerSchema.getElementType(), although the writer's schema is available                                                                                                        
    there as action.writer.                                                                                                                                                         
  - The zero-byte allocation cap (isZeroByteSchema) is chosen from the reader's                                                                                                     
    schema in the same way.  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)