[jira] [Commented] (AVRO-724) C implementation does not write datum values that are larger than the memory write buffer (currently 16K)

["Michael Cooper (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168131#comment-13168131 ] 

Michael Cooper commented on AVRO-724:
-------------------------------------

I have added a way to set the upper limit of the datum_buffer as part of AVRO-957.

It was implemented in memory with avro_malloc instead of it being a stack array.

The relevant commit: https://github.com/hitwise/avro/commit/859f83e09354aeb13ac394777adc449586c6d770
                
> C implementation does not write datum values that are larger than the memory write buffer (currently 16K)
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: AVRO-724
>                 URL: https://issues.apache.org/jira/browse/AVRO-724
>             Project: Avro
>          Issue Type: Bug
>          Components: c
>    Affects Versions: 1.4.1
>            Reporter: Jeremy Hinegardner
>
> The current C implementation does not allow for datum values greater than 16K.
> The {{avro_file_writer_append}} flushes blocks to disk over time, but does not deal with the single case of a single datum being larger than {{avro_file_writer_t.datum_buffer}}.  This is noted in the source code:
> {code:title=datafile.c:294-313}
> int avro_file_writer_append(avro_file_writer_t w, avro_datum_t datum)
> {
>     int rval;                  
>     if (!w || !datum) {        
>         return EINVAL;         
>     }
>     rval = avro_write_data(w->datum_writer, w->writers_schema, datum);
>     if (rval) {                
>         check(rval, file_write_block(w));
>         rval =
>             avro_write_data(w->datum_writer, w->writers_schema, datum);
>         if (rval) {            
>             /* TODO: if the datum encoder larger than our buffer,
>                just write a single large datum */
>             return rval;       
>         }
>     }
>     w->block_count++;          
>     return 0;                  
> }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira