You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2023/01/13 14:21:16 UTC
[GitHub] [iceberg] qinghui-xu opened a new issue, #6579: Compaction: Abort the job smartly when partial commit starts to fail
qinghui-xu opened a new issue, #6579:
URL: https://github.com/apache/iceberg/issues/6579
### Feature Request / Improvement
We have a streaming pipeline serving (upsert) data to a table, and a spark compaction job to rewrite files asynchronously.
Compaction job will fail to commit when streaming pipeline commits to the table with some deletion in existing data. To address this we enabled partial commit in compaction job.
What we observe after enabling partial commit (say, with `partial-progress.max-commit = 10`):
- First few partial commits succeeded
- Streaming job commits a snapshot with upsert
- All subsequent partial commits failed
In our case, the streaming pipeline is writing to all partitions at the same time constantly, which means when a first partial commit fails because of conflict all the subsequent partial commits would fail almost for sure. It would be nice to abort the job sooner to avoid wasting resources on doomed-to-fail processing.
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #6579:
URL: https://github.com/apache/iceberg/issues/6579#issuecomment-1384128689
Seems good to me, I would probably just stick to something like "number of allowed failures, or smart" as the two failure modes, although if we can figure out "smart" let's just do that and not even give a parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6579:
URL: https://github.com/apache/iceberg/issues/6579#issuecomment-1384084128
@qinghui-xu it seems reasonable to have different commit failure handling modes . Would you like to open a PR for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
URL: https://github.com/apache/iceberg/issues/6579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] qinghui-xu commented on issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
Posted by GitBox <gi...@apache.org>.
qinghui-xu commented on issue #6579:
URL: https://github.com/apache/iceberg/issues/6579#issuecomment-1384392111
Sure, I will be happy to contribute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6579:
URL: https://github.com/apache/iceberg/issues/6579#issuecomment-1657304620
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6579: Compaction: Abort the job smartly when partial commit starts to fail
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6579:
URL: https://github.com/apache/iceberg/issues/6579#issuecomment-1636925320
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org