You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2023/08/17 22:08:01 UTC
[GitHub] [accumulo] keith-turner opened a new issue, #3709: Avoid storing split points in Zookeeper
keith-turner opened a new issue, #3709:
URL: https://github.com/apache/accumulo/issues/3709
In the elasticity branch split code was moved from the tablet server to the manager. The Manager execture a FATE operation to perform split. This FATE operation will store the split points in ZK memory. As part of a #3705 a new test was added the creates 1 million tablets. This test adds 10K splits at a time. The test was manually modified to do 100K splits at a time and it caused ZK and/or the manager to fall over.
It would be better if the split points were stored in a tmp file. The create table fate operation already does this when it has splits. The new split fate code could probably share code with create table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2059750284
@jmark99 - are you working this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "jmark99 (via GitHub)" <gi...@apache.org>.
jmark99 commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2069301773
@dlmarion yes, I'm getting back to it this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "jmark99 (via GitHub)" <gi...@apache.org>.
jmark99 commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2069403146
@ok, thanks for the update. I'll close the issue as no longer relevant based upon the existence of the new Accumulo table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2069317297
> From the comments above is there no longer a need to work this?
@jmark99 - Based on what @cshannon said, I think that is the case. I think we could probably close this issue. @cshannon modified FATE such that user FATE operations are stored in a new Accumulo table (+fate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2059553175
I believe so, the information is stored in SplitInfo which is persisted as part of the Repos in the Accumulo table now. So at least for the user tables it's no longer in ZK and I don't think we need to worry about it being in ZK for metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "jmark99 (via GitHub)" <gi...@apache.org>.
jmark99 commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2069304479
@dlmarion I've been away for two weeks. From the comments above is there no longer a need to work this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-2059475183
@cshannon - Is this issue OBE now that we have user FATE operations being stored in an Accumulo table instead of ZooKeeper?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid storing split points in Zookeeper [accumulo]
Posted by "jmark99 (via GitHub)" <gi...@apache.org>.
jmark99 closed issue #3709: Avoid storing split points in Zookeeper
URL: https://github.com/apache/accumulo/issues/3709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on issue #3709: Avoid storing split points in Zookeeper
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on issue #3709:
URL: https://github.com/apache/accumulo/issues/3709#issuecomment-1702731996
In general, I was thinking as we go to Elasticity it might be nice to see if we can get more things out of Zookeeper and store in Accumulo itself. Zookeeper can be a bottleneck and other projects are dropping it for performance reasons as clusters are getting larger (such as Kafka). In theory the changes in Elasticity will allow us to scale up with many more tablets since they are offline/ondemand which would require a lot more metadata tracking in ZK and I can see that could cause issues, like in this case.
For example, for the split points here, I'm wondering if in the future it would be make sense to store the split points in Accumulo in another table. Anyways, just a thought, in this case the tmp file probably makes sense since it's already doing that elsewhere and code could be shared. There also might be reasons that isn't possible but again this is just more of a general comment that could be applied to other situations too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org