You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Mac Noland <mc...@gmail.com> on 2018/11/29 17:58:26 UTC
Tablets Per Tablet Server
Good day,
We have a number of customers using Kudu for a large number of tables from
db/warehouse offload use cases and commonly run into situations where our
tablets are over 2000 per tablet server. Many of the tables are actually
small, but with the minimum of 2 partitions and default of 3 replicas, we
end up chewing up 6 tablets minimally for each table. Being we have
customers way over this and are running fine, we understand this is not a
hard guideline, but overall does cause some concern with some customers
knowing we are outside of the written guidelines.
*The maximum number of tablets per tablet server is 2000, post-replication,
but we recommend 1000 tablets or fewer per tablet server.*
https://kudu.apache.org/docs/known_issues.html#_scale
Do we know how this number was picked and if there is any work being done
to look at what it would take to increase the number? Or things we could
contribute to help increase that number? Not 100% sure what we'd be
signing up for to do that, but always looking for opportunity to help.
Thanks in advance.
Mac
Re: Tablets Per Tablet Server
Posted by Boris Tyukin <bo...@boristyukin.com>.
Mac, I asked the same question some time ago if you want to check out the
comments from Kudu team..There is also umbrella Jira below to support high
density nodes.
http://mail-archives.apache.org/mod_mbox/kudu-user/201711.mbox/%3CCANRT7T1aktogW1-1p7Lz9atZRyTMuB1auBYkCns3_-FYU1DEOg@mail.gmail.com%3E
https://issues.apache.org/jira/browse/KUDU-1967
On Thu, Nov 29, 2018 at 12:58 PM Mac Noland <mc...@gmail.com>
wrote:
> Good day,
>
> We have a number of customers using Kudu for a large number of tables from
> db/warehouse offload use cases and commonly run into situations where our
> tablets are over 2000 per tablet server. Many of the tables are actually
> small, but with the minimum of 2 partitions and default of 3 replicas, we
> end up chewing up 6 tablets minimally for each table. Being we have
> customers way over this and are running fine, we understand this is not a
> hard guideline, but overall does cause some concern with some customers
> knowing we are outside of the written guidelines.
>
> *The maximum number of tablets per tablet server is 2000,
> post-replication, but we recommend 1000 tablets or fewer per tablet server.*
> https://kudu.apache.org/docs/known_issues.html#_scale
>
> Do we know how this number was picked and if there is any work being done
> to look at what it would take to increase the number? Or things we could
> contribute to help increase that number? Not 100% sure what we'd be
> signing up for to do that, but always looking for opportunity to help.
>
> Thanks in advance.
>
> Mac
>
>
>
Re: Tablets Per Tablet Server
Posted by Mac Noland <mc...@gmail.com>.
Thanks for the feedback. We are going to cut our small table partitions by
half with that feature to not require a partition.
On Thu, Nov 29, 2018 at 7:03 PM helifu <hz...@corp.netease.com> wrote:
> Hi Mac,
>
>
>
> To many of the small tables which have the minimum of 2 partitions, there
> is another way to help you to save half the tablets.(Thanks to Andrew, who
> shared this info to me on the slack)
>
> You can find the detail in both link:
>
> https://jira.apache.org/jira/browse/IMPALA-5546 IMPALA-2.10.0
>
>
> https://www.cloudera.com/documentation/enterprise/latest/topics/kudu_impala.html#concept_g51_5vk_ft
>
>
>
> *Note:* In Impala included in CDH 5.13 and higher, the PARTITION BY clause
> is optional for Kudu tables. If the clause is omitted, Impala automatically
> constructs a single partition that is not connected to any column. Because
> such a table cannot take advantage of Kudu features for parallelized
> queries and query optimizations, omitting the PARTITION BY clause is only
> appropriate for small lookup tables.
>
>
>
>
>
> 何李夫
>
> 2018-11-30 09:02:21
>
>
>
> *发件人:* user-return-1546-hzhelifu=corp.netease.com@kudu.apache.org
> <us...@kudu.apache.org> *代表 *Mac
> Noland
> *发送时间:* 2018年11月30日 1:58
> *收件人:* user@kudu.apache.org
> *主题:* Tablets Per Tablet Server
>
>
>
> Good day,
>
>
>
> We have a number of customers using Kudu for a large number of tables from
> db/warehouse offload use cases and commonly run into situations where our
> tablets are over 2000 per tablet server. Many of the tables are actually
> small, but with the minimum of 2 partitions and default of 3 replicas, we
> end up chewing up 6 tablets minimally for each table. Being we have
> customers way over this and are running fine, we understand this is not a
> hard guideline, but overall does cause some concern with some customers
> knowing we are outside of the written guidelines.
>
>
>
> *The maximum number of tablets per tablet server is 2000,
> post-replication, but we recommend 1000 tablets or fewer per tablet server.*
> https://kudu.apache.org/docs/known_issues.html#_scale
>
>
>
> Do we know how this number was picked and if there is any work being done
> to look at what it would take to increase the number? Or things we could
> contribute to help increase that number? Not 100% sure what we'd be
> signing up for to do that, but always looking for opportunity to help.
>
>
>
> Thanks in advance.
>
>
>
> Mac
>
>
>
>
>
答复: Tablets Per Tablet Server
Posted by helifu <hz...@corp.netease.com>.
Hi Mac,
To many of the small tables which have the minimum of 2 partitions, there is another way to help you to save half the tablets.(Thanks to Andrew, who shared this info to me on the slack)
You can find the detail in both link:
https://jira.apache.org/jira/browse/IMPALA-5546 IMPALA-2.10.0
https://www.cloudera.com/documentation/enterprise/latest/topics/kudu_impala.html#concept_g51_5vk_ft
Note: In Impala included in CDH 5.13 and higher, the PARTITION BY clause is optional for Kudu tables. If the clause is omitted, Impala automatically constructs a single partition that is not connected to any column. Because such a table cannot take advantage of Kudu features for parallelized queries and query optimizations, omitting the PARTITION BY clause is only appropriate for small lookup tables.
何李夫
2018-11-30 09:02:21
发件人: user-return-1546-hzhelifu=corp.netease.com@kudu.apache.org <us...@kudu.apache.org> 代表 Mac Noland
发送时间: 2018年11月30日 1:58
收件人: user@kudu.apache.org
主题: Tablets Per Tablet Server
Good day,
We have a number of customers using Kudu for a large number of tables from db/warehouse offload use cases and commonly run into situations where our tablets are over 2000 per tablet server. Many of the tables are actually small, but with the minimum of 2 partitions and default of 3 replicas, we end up chewing up 6 tablets minimally for each table. Being we have customers way over this and are running fine, we understand this is not a hard guideline, but overall does cause some concern with some customers knowing we are outside of the written guidelines.
The maximum number of tablets per tablet server is 2000, post-replication, but we recommend 1000 tablets or fewer per tablet server. https://kudu.apache.org/docs/known_issues.html#_scale
Do we know how this number was picked and if there is any work being done to look at what it would take to increase the number? Or things we could contribute to help increase that number? Not 100% sure what we'd be signing up for to do that, but always looking for opportunity to help.
Thanks in advance.
Mac