You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/09/01 16:48:48 UTC
[GitHub] [druid] rohangarg opened a new pull request, #13015: Enable using SHARED lock for append ingestion jobs
rohangarg opened a new pull request, #13015:
URL: https://github.com/apache/druid/pull/13015
Currently, all append ingestion jobs to a datasource use EXCLUSIVE locks on time intervals or segments by default. #12041 added a config to allow specifying a flag in the ingestion job context which tries to take SHARED locks whenever possible. This allows for concurrent append jobs to co-exist.
This change enables that config by default so that all appends can run concurrently.
This PR has:
- [x] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg commented on pull request #13015: Enable using SHARED lock for append ingestion jobs
Posted by GitBox <gi...@apache.org>.
rohangarg commented on PR #13015:
URL: https://github.com/apache/druid/pull/13015#issuecomment-1254686270
Closing this since there are multiple test failures after enabling the context flag (not necessarily related to bugs, but to different configurations in tests which would need tweaking).
Since MSQ already uses SHARED locks for `INSERTS` and EXCLUSIVE locks for `REPLACE`, we should be leveraging that capability instead of spending time on this.
It can be re-opened if we think otherwise in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rohangarg closed pull request #13015: Enable using SHARED lock for append ingestion jobs
Posted by GitBox <gi...@apache.org>.
rohangarg closed pull request #13015: Enable using SHARED lock for append ingestion jobs
URL: https://github.com/apache/druid/pull/13015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13015: Enable using SHARED lock for append ingestion jobs
Posted by GitBox <gi...@apache.org>.
kfaraz commented on PR #13015:
URL: https://github.com/apache/druid/pull/13015#issuecomment-1235165611
The changes look good, @rohangarg .
I had a couple of questions.
1. How would this affect compaction jobs? I suppose compaction would want an exclusive lock and would thus wait (or fail?) for an interval on which an append job has a shared lock.
2. Do we already have ITs/UTs that test using a shared lock for append jobs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org