You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2023/01/18 10:06:39 UTC
[GitHub] [pinot] ankitsultana opened a new issue, #10137: Determine Number of Rows for a Realtime Segment Based on a Config
ankitsultana opened a new issue, #10137:
URL: https://github.com/apache/pinot/issues/10137
At present we have two types of configs:
1. Set threshold based on size
2. Set threshold based on number of rows
Regarding the second type, while the value set in those configs is proportional to the number of rows in a segment, it's not the exact value. It seems to divide the number of rows by the maximum number of kafka partitions in a server.
Feature: It would be great to have a config that determines the number of rows in a segment exactly. If the config says 1M, then the segments should be flushed after 1M rows.
cc: @wirybeaver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1399029119
We have some tables that use number of rows as flush threshold, but vast majority of the them use auto-tuning and have defined desired segment size. I think we should still have support for num rows config for backward compatibility, but we should strive for using desired segment size which is:
1. consistent (remains the same) if we uplift or downsize the cluster
2. segment size is an important factor in different aspects like memory management servers, segment build duration in commit protocol, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1396323369
Most users are not using the recommendation engine though. What I'm trying to convey is that currently there is not a straight forward way to configure the rows per segment. Fixing rows per segment is the easiest way to guarantee good performance, and we should provide an easy way to configure it without knowing how things are managed internally. I agree using the more complex mechanism can give better stability, but we also want a simple configuration for non-expert users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kishoreg commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by "kishoreg (via GitHub)" <gi...@apache.org>.
kishoreg commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1399190414
Sorry for being late to the party.. I have to admit.. I really thought the num rows threshold was per segment.. this is first time I am learning that we actually take that number and divide by the number of partitions on a server. I find that very confusing and hard to reason about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1396237583
@mcvsubbu The proposed way is very useful when we want to limit the CPU usage for all the consumers on a given server, but for certain use cases, CPU might not be the bottleneck (e.g. ingestion rate is low, query throughput is low, no strict latency requirement), and we just want an easy way to configure the segment size so that segment is not too small or too big.
It will be similar to the size (in bytes) based config, but we fix the number of rows per segment. It will be easier to config and get desired performance than the size based because for wide or narrow table, size based will give very different row count per segment. For columnar store, row count is the major factor of the query performance. Also, the size based config can only be properly set after the first several segments are consumed, which is not as straight-forward as the row based.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1396307559
> @mcvsubbu The proposed way is very useful when we want to limit the CPU usage for all the consumers on a given server, but for certain use cases, CPU might not be the bottleneck (e.g. ingestion rate is low, query throughput is low, no strict latency requirement), and we just want an easy way to configure the segment size so that segment is not too small or too big. It will be similar to the size (in bytes) based config, but we fix the number of rows per segment. It will be easier to config and get desired performance than the size based because for wide or narrow table, size based will give very different row count per segment. For columnar store, row count is the major factor of the query performance. Also, the size based config can only be properly set after the first several segments are consumed, which is not as straight-forward as the row based.
I don't agree with that last part. The size is recommended by the recommendation engine, and can be configured in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1398806514
>
I can see how they may get confused. The right way to tackle that problem is to use the segment size based configuration. The `ReealtimeProvisioningHelper` tool can be used to set the right segment size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1396336852
We should really think about configs that add no value. IMO this config adds no value. I think the non-expert users will come back asking us to tune this number because they rebalanced, or the number of partitions changed etc. It is one less config to document.
This is my vote -- but sure, majority wins :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ankitsultana commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1398771142
New config since we wouldn't want to break the existing behavior. The existing behavior is by design as @mcvsubbu had mentioned.
A note on the motivation for this issue: I agree with the point about being careful about adding new configs, but what we have seen internally is that almost every time what we are interested in is setting a threshold on the exact number of rows per segment. We have come close to incidents a few times when dealing with high ingestion throughput tables because someone set a flush threshold thinking it will directly control the number of rows, but it ended up dividing the number of rows by maxPartitionsPerInstance, leading to very frequent flushes which put pressure on controller/deep-store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1399076616
Agree with the advantages of segment size based threshold, but it also has some drawbacks:
1. User sometimes wants to fix the row count per segment. It is not straight forward to translate segment size into row count, and it also differs with different data distribution.
2. When moving to a new server, we need to start with a small segment, then slowly increase the row count to get the desired segment size.
Even for segment size based threshold, we translate the desired segment size into row count. IMO we should give user the option to directly configure the segment row count. For a lot of use cases, memory is not really a concern, and I myself find it difficult to tune the segment size, not mention the average user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1396217784
@ankitsultana anda @Jackie-Jiang Pinot serves queries while consuming rows. In order to have enough CPU left over to process queries, it is best to limit the consumption rate to some maximum number. That is the reason we have the math as described -- the number of rows consumed by any instance within a period of time is limited to a max number. Further, if we rebalance the table with more instances per replica, the number automatically adjusts itself.
Granted, that all this does not help much when there are multiple tables in a tenant.
But then neither does the new config as proposed.
Instead of coming up with yet another config that the administrator cannot do much with in the long run, there are a few things we may be able to do:
- Allow the user to specify a max for the average ingestion rate (bytes may make more sense) for each instance in the cluster.
- Have a continuous measurement of the ingestion rate (again, bytes may be better than rows)
- Run an algorithm on the controller to come up with an optimal number of rows for a segment given the tables and partitions hosted (either in the tenant or in the cluster, as the case maybe)
- If any host in the cluster is in danger of exceeding the max, then raise a metric.
I agree this is more complex than adding another config, but it is also more useful, and hopefully more interesting problem to solve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1396181623
cc @mcvsubbu @siddharthteotia @sajjad-moradi
+1 on this. Dividing the number of kafka partitions on a server always confuses me, which makes the stream very hard to config. In most cases, we just want the segment to be proper sized, instead of worrying about the total records in memory. This config is also only table level, so it cannot limit the total records in memory for the whole server.
To keep it backward compatible, we can introduce a new config as the exact segment size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #10137: Determine Number of Rows for a Realtime Segment Based on a Config
Posted by "mcvsubbu (via GitHub)" <gi...@apache.org>.
mcvsubbu commented on issue #10137:
URL: https://github.com/apache/pinot/issues/10137#issuecomment-1399174396
> Agree with the advantages of segment size based threshold, but it also has some drawbacks:
>
> 1. User sometimes wants to fix the row count per segment. It is not straight forward to translate segment size into row count, and it also differs with different data distribution.
> 2. When moving to a new server, we need to start with a small segment, then slowly increase the row count to get the desired segment size.
that is not true. Only on a new table we start with a small segment and increase. New servers/rebalance is not a problem.
>
> Even for segment size based threshold, we translate the desired segment size into row count. IMO we should give user the option to directly configure the segment row count. For a lot of use cases, memory is not really a concern, and I myself find it difficult to tune the segment size, not mention the average user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org