You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "GreatStone (via GitHub)" <gi...@apache.org> on 2024/04/15 06:37:08 UTC
[I] Iceberg and Avro File Writing: Inquiring About FlushOnEveryBlock Feature and Its Impact on Performance and Data Integrity [iceberg]
GreatStone opened a new issue, #10142:
URL: https://github.com/apache/iceberg/issues/10142
### Query engine
EMR
### Question
I’ve observed that Iceberg employs org.apache.avro.file.DataFileWriter for writing Avro files, relying on some default settings. One particular setting, FlushOnEveryBlock, triggers numerous flush operations during extensive single updates, potentially impacting system performance. I’m curious about two aspects:
- Does Iceberg offer a way to disable this feature?
- Is the FlushOnEveryBlock setting critical? Would turning it off compromise data integrity in any way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Does the FlushOnEveryBlock feature in Avro affect Iceberg data integrity? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10142: Does the FlushOnEveryBlock feature in Avro affect Iceberg data integrity?
URL: https://github.com/apache/iceberg/issues/10142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Does the FlushOnEveryBlock feature in Avro affect Iceberg data integrity? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10142:
URL: https://github.com/apache/iceberg/issues/10142#issuecomment-2448718918
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Does the FlushOnEveryBlock feature in Avro affect Iceberg data integrity? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10142:
URL: https://github.com/apache/iceberg/issues/10142#issuecomment-2477674912
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org