You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/16 02:53:26 UTC

[GitHub] [iceberg] renshangtao opened a new pull request #4338: parquet_writer.java should close the writeStore when flushRowGroup's finished is false

renshangtao opened a new pull request #4338:
URL: https://github.com/apache/iceberg/pull/4338


   When parquetWriter.java call close it called the writeStore.close 
   
   public void close() throws IOException {
       if (!closed) {
         this.closed = true;
         flushRowGroup(true);
         writeStore.close();                       ------ here
         if (writer != null) {
           writer.end(metadata);
         }
       }
     }
   
   but in checksize function when call flushRowGroup(false); 
   
   private void flushRowGroup(boolean finished) {
       try {
         if (recordCount > 0) {
           ensureWriterInitialized();
           writer.startBlock(recordCount);
           writeStore.flush();
           flushPageStoreToWriter.invoke(writer);
           writer.endBlock();
           if (!finished) {
             startRowGroup();                ---- here new another one, but the old not closed
           }
         }
       } catch (IOException e) {
         throw new UncheckedIOException("Failed to flush row group", e);
       }
     }
   
   
   it doesn't close the writeStore and then new another one


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on pull request #4338: Parquet: parquet_writer.java should close the writeStore when flushRowGroup's finished is false

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #4338:
URL: https://github.com/apache/iceberg/pull/4338#issuecomment-1069291663


   @renshangtao, was this causing a problem in your environment? Won't the store get cleaned up by garbage collection?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on pull request #4338: Parquet: parquet_writer.java should close the writeStore when flushRowGroup's finished is false

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #4338:
URL: https://github.com/apache/iceberg/pull/4338#issuecomment-1069294185


   Ah, looks like this eventually calls `CapacityByteArrayOutputStream.close`, which releases buffers back to the allocator. Looks like a good change to reduce memory consumption.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue merged pull request #4338: Parquet: parquet_writer.java should close the writeStore when flushRowGroup's finished is false

Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #4338:
URL: https://github.com/apache/iceberg/pull/4338


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] rdblue commented on pull request #4338: Parquet: parquet_writer.java should close the writeStore when flushRowGroup's finished is false

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #4338:
URL: https://github.com/apache/iceberg/pull/4338#issuecomment-1069541813


   Thanks, @renshangtao!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org