You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2024/11/22 18:31:12 UTC
[I] Seeding compaction commit and split fate transactions is slow [accumulo]
keith-turner opened a new issue, #5097:
URL: https://github.com/apache/accumulo/issues/5097
**Describe the bug**
While testing splitting lots of tablets it was observed that seeding the split fate transactions was the bottleneck and not running the actual fate operation. Looking at [the code](https://github.com/apache/accumulo/blob/bbfd250d8694c210faf31116d199b56570c46f38/core/src/main/java/org/apache/accumulo/core/fate/Fate.java#L421-L455) there is a lot of back and forth to the seed a fate transaction
1. Do conditional mutation reserve
2. Do a scan to see if there is a top on the stack
3. Do a conditional mutation to set the transaction info
4. Do a conditional mutation to set the transaction info
5. Do a conditional mutation to set the status
All of this could likely be done in a single conditional mutation. This would avoid the scan and doing multiple conditional mutations. Observed this while looking into splits, but improving this would speed up compaction commit and split.
**Expected behavior**
If possible a single conditional mutation is run to seed a fate transaction.
**Additional context**
There may be other places in the code that are submitting multiple conditional mutation when one would suffice. This is caused by the code evolving from initially having a zookeeper persistent store where data was stored in multiple zk nodes for a single fate operation. Multiple ZK nodes require multiple write. Implementing #4905 could allow zookeeper and accumuo table impls to have less writes for fate updates by improving the code for both to support this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Seeding compaction commit and split fate transactions is slow [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on issue #5097:
URL: https://github.com/apache/accumulo/issues/5097#issuecomment-2503894972
Sounds good, I can take a look at the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Seeding compaction commit and split fate transactions is slow [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner closed issue #5097: Seeding compaction commit and split fate transactions is slow
URL: https://github.com/apache/accumulo/issues/5097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Seeding compaction commit and split fate transactions is slow [accumulo]
Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on issue #5097:
URL: https://github.com/apache/accumulo/issues/5097#issuecomment-2498225531
Interesting, yeah the number of conditional mutations could definitely be decreased here and maybe elsewhere too...
I can take a look at this when I get the chance (unless someone else wants to pick this up before then, which is also fine).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Seeding compaction commit and split fate transactions is slow [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #5097:
URL: https://github.com/apache/accumulo/issues/5097#issuecomment-2502468971
> Interesting, yeah the number of conditional mutations could definitely be decreased here and maybe elsewhere too...
I can take a look at this when I get the chance (unless someone else wants to pick this up before then, which is also fine).
I just saw this comment and I had started working on this earlier today. I hope to have a PR up tomorrow. Also I have realized there are follow on changes that could further speed some things up, plan to open follow on issues up after posting that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Seeding compaction commit and split fate transactions is slow [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner closed issue #5097: Seeding compaction commit and split fate transactions is slow
URL: https://github.com/apache/accumulo/issues/5097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Seeding compaction commit and split fate transactions is slow [accumulo]
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #5097:
URL: https://github.com/apache/accumulo/issues/5097#issuecomment-2494534259
When experimenting with this was seeing that a single thread could seed around 1.3 fate operations per second. Making this a single conditional mutation should increase that rate significantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org