You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/01/18 17:16:37 UTC
[GitHub] [incubator-pinot] npawar opened a new issue #6453: Handle creation of empty segment
npawar opened a new issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453
Currently, is segment creation is started with no rows, we error out. We stumbled upon this issue while working on the Kinesis connector. In the Kinesis connector, we are going to trigger segment completion on criteria of reaching end-of-shard (in addition to rows and time thresholds). It is possible that the CONSUMING segment associated with the shard has no rows, and then the segment completion is reached.
It would be good if we could make Pinot gracefully handle segment creation with 0 rows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-763787916
do you mean we introduce a broker side pruner, something like EmptySegmentPruner, that is always in effect? Or did you mean prune it in some other way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-763238225
Please make sure the metadata can be correctly generated without any record. Also, we should prune out empty segments as the first step to prevent the query engine (both Broker and Server side) throwing exception while processing them. Currently we assume the segment cannot be empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
jackjlli commented on issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-791675203
Hi, what's the pros and cons between creating an empty segment v.s. not creating a segment, e.g. in Kinesis connector?
Failing the segment creation could be helpful to early detect the issue from the upstream. If we release this constraint, issues may appear underneath the creation job without any notice.
In realtime side, there could be no streaming data for newly created tables, thus it'd lead to empty segments flushed in the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-762444076
I am not sure creating empty segments is a good idea all the time. Certainly, for offline use cases, it is better to flag this early and fail the job, so at the minimum, we should implement this as an option.
Even in the realtime use cases, generating many empty segments may cause unnecessary overhead during query processing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-763189258
Regarding `Even in the realtime use cases, generating many empty segments may cause unnecessary overhead during query processing.` - In realtime case especially, this will be a very very rare occurrence. It will never even happen in streams which only rely on time/rows thresholds.
And whatever empty segments we do end up with, they should be handled in the segment merge/replace.
Also, lets keep the behavior consistent across offline/realtime. Especially considering we have a feature to filter rows during ingestion, it is possible that in offline, we cross the check for empty segment, but then fail later anyway because everything got filtered out.
FYI @kishoreg @Jackie-Jiang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-763851817
Yes, and we need to make sure all the pruners on the broker side can handle the empty segment metadata properly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar closed issue #6453: Handle creation of empty segment
Posted by GitBox <gi...@apache.org>.
npawar closed issue #6453:
URL: https://github.com/apache/incubator-pinot/issues/6453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org