You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/11 12:25:38 UTC

[GitHub] [hudi] nsivabalan commented on pull request #5129: [HUDI-3709] Fixing `ParquetWriter` impls not respecting Parquet Max File Size limit

nsivabalan commented on PR #5129:
URL: https://github.com/apache/hudi/pull/5129#issuecomment-1123690948

   I went through the code again wrt this patch and https://github.com/apache/hudi/pull/5497. Probably we should bring back the WrapperFileSystem again so that we don't hit the parquetWriter to fetch the size. if we ensure we flush at regular intervals, wrapperFileSystem.getBytesWritten(Path file) should give us the right size of data that got written. this will also ensure we don't hit the disk or incur the cost due to column meta refresh within parquetWriter. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org