You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Konstantin Pelykh <kp...@gmail.com> on 2015/06/25 20:31:29 UTC
Any limits on Accumulo namespaces/tables?
Hi everyone,
Are there any limits on a number of namespaces and tables in Accumulo?
For my current application I'm planning to use multiple namespaces to
separate data for different clients. Each namespace will have 8 tables.
Would it be posible to have 100,500, 1000 namespaces in Accumulo? How will
such namespace layout impact performance of accumulo scanner? As far as I
rememebr every time when scanner is created it gets the whole list of
namespaces/tables from zookeeper and it slows down performace as number of
namespaces increases.
Thanks!
Konstantin
--------
Big Data / Search Consultant
LinkedIn: linkedin.com/in/kpelykh <http://www.linkedin.com/in/kpelykh>
Website: www.kpelykh.com
Re: Any limits on Accumulo namespaces/tables?
Posted by Konstantin Pelykh <kp...@gmail.com>.
When I start working on this issue I'll look in detail if it's possible to
modify Scanner to avoid having Zookeeper watches for all tables availbale
in Accumulo.
I'll share my results if I find any bottlenecks or will be able to optimize
scanner to work with many namespaces.
--------
Big Data / Search Consultant
LinkedIn: linkedin.com/in/kpelykh <http://www.linkedin.com/in/kpelykh>
Website: www.kpelykh.com
On Thu, Jun 25, 2015 at 11:38 AM, Christopher <ct...@apache.org> wrote:
> In theory, no, there aren't limits. In practice, you'd be limited by
> the amount of memory you have available in ZooKeeper, and you may have
> trouble (currently) with lots of ZooKeeper watchers which monitor
> configuration changes for tables.
>
> There may be data structures in the API and throughout the code which
> contain a complete list of tables, also, and that could hurt in terms
> of memory.
>
> In short, we don't define a limit, but there may be bottlenecks you
> might encounter. If you do find these, we may be able to make
> improvements, so let us know. If this because a big problem, we may
> impose some upper bound in the future, just so people can reason about
> their environment better, and we could offer more concrete numbers on
> what is feasible.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Thu, Jun 25, 2015 at 2:31 PM, Konstantin Pelykh <kp...@gmail.com>
> wrote:
> > Hi everyone,
> >
> > Are there any limits on a number of namespaces and tables in Accumulo?
> > For my current application I'm planning to use multiple namespaces to
> > separate data for different clients. Each namespace will have 8 tables.
> > Would it be posible to have 100,500, 1000 namespaces in Accumulo? How
> will
> > such namespace layout impact performance of accumulo scanner? As far as I
> > rememebr every time when scanner is created it gets the whole list of
> > namespaces/tables from zookeeper and it slows down performace as number
> of
> > namespaces increases.
> >
> > Thanks!
> > Konstantin
> > --------
> > Big Data / Search Consultant
> > LinkedIn: linkedin.com/in/kpelykh
> > Website: www.kpelykh.com
>
Re: Any limits on Accumulo namespaces/tables?
Posted by Christopher <ct...@apache.org>.
Correction: s/this because a big problem/this causes a big problem/ :)
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Thu, Jun 25, 2015 at 2:38 PM, Christopher <ct...@apache.org> wrote:
> In theory, no, there aren't limits. In practice, you'd be limited by
> the amount of memory you have available in ZooKeeper, and you may have
> trouble (currently) with lots of ZooKeeper watchers which monitor
> configuration changes for tables.
>
> There may be data structures in the API and throughout the code which
> contain a complete list of tables, also, and that could hurt in terms
> of memory.
>
> In short, we don't define a limit, but there may be bottlenecks you
> might encounter. If you do find these, we may be able to make
> improvements, so let us know. If this because a big problem, we may
> impose some upper bound in the future, just so people can reason about
> their environment better, and we could offer more concrete numbers on
> what is feasible.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Thu, Jun 25, 2015 at 2:31 PM, Konstantin Pelykh <kp...@gmail.com> wrote:
>> Hi everyone,
>>
>> Are there any limits on a number of namespaces and tables in Accumulo?
>> For my current application I'm planning to use multiple namespaces to
>> separate data for different clients. Each namespace will have 8 tables.
>> Would it be posible to have 100,500, 1000 namespaces in Accumulo? How will
>> such namespace layout impact performance of accumulo scanner? As far as I
>> rememebr every time when scanner is created it gets the whole list of
>> namespaces/tables from zookeeper and it slows down performace as number of
>> namespaces increases.
>>
>> Thanks!
>> Konstantin
>> --------
>> Big Data / Search Consultant
>> LinkedIn: linkedin.com/in/kpelykh
>> Website: www.kpelykh.com
Re: Any limits on Accumulo namespaces/tables?
Posted by Christopher <ct...@apache.org>.
In theory, no, there aren't limits. In practice, you'd be limited by
the amount of memory you have available in ZooKeeper, and you may have
trouble (currently) with lots of ZooKeeper watchers which monitor
configuration changes for tables.
There may be data structures in the API and throughout the code which
contain a complete list of tables, also, and that could hurt in terms
of memory.
In short, we don't define a limit, but there may be bottlenecks you
might encounter. If you do find these, we may be able to make
improvements, so let us know. If this because a big problem, we may
impose some upper bound in the future, just so people can reason about
their environment better, and we could offer more concrete numbers on
what is feasible.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Thu, Jun 25, 2015 at 2:31 PM, Konstantin Pelykh <kp...@gmail.com> wrote:
> Hi everyone,
>
> Are there any limits on a number of namespaces and tables in Accumulo?
> For my current application I'm planning to use multiple namespaces to
> separate data for different clients. Each namespace will have 8 tables.
> Would it be posible to have 100,500, 1000 namespaces in Accumulo? How will
> such namespace layout impact performance of accumulo scanner? As far as I
> rememebr every time when scanner is created it gets the whole list of
> namespaces/tables from zookeeper and it slows down performace as number of
> namespaces increases.
>
> Thanks!
> Konstantin
> --------
> Big Data / Search Consultant
> LinkedIn: linkedin.com/in/kpelykh
> Website: www.kpelykh.com