You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/16 23:43:34 UTC
[GitHub] [pinot] PrachiKhobragade opened a new pull request, #8911: add freshness lag when there are no consuming segments
PrachiKhobragade opened a new pull request, #8911:
URL: https://github.com/apache/pinot/pull/8911
The current freshness lag metric doesn't emit a metric, when there are no
consuming segments. This won't emit lag when there are errors while consuming or
a consuming segment is not created, so calculate the lag from the endtime using
the last segment that was online as freshness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r903239715
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
@Jackie-Jiang The original logic that we try to compute was the following
```
if (_numConsumingSegmentsProcessed > 0 || (_numConsumingSegmentsProcessed == 0 && _minConsumingFreshnessTimeMs != Long.MAX_VALUE)) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r899649735
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
Review Comment:
I think that we can probably even remove this `else if` block
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
Review Comment:
Can you double-check on `NUM_CONSuMING_SEGMENTS_PROCESSED` and see if we emit any metric based on that? Do we need to put `0` here?
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
+ // case: no immutable segments, no metric emitted
+ LOGGER.warn("Request {} queried no consuming or online segments, with minConsumingFreshnessTimeMs: {}", requestId,
Review Comment:
Let's use `debug` level as well. Otherwise, we will log this line for each query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r907789158
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -195,19 +198,35 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
// Gather stats for realtime consuming segments
int numConsumingSegments = 0;
- long minIndexTimeMs = Long.MAX_VALUE;
- long minIngestionTimeMs = Long.MAX_VALUE;
- for (IndexSegment indexSegment : indexSegments) {
- if (indexSegment instanceof MutableSegment) {
- numConsumingSegments += 1;
- SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
- long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
- if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
- minIndexTimeMs = indexTimeMs;
- }
- long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
- if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
- minIngestionTimeMs = ingestionTimeMs;
+ int numOnlineSegments = 0;
+ long minIndexTimeMs = 0;
+ long minIngestionTimeMs = 0;
+ long maxEndTimeMs = 0;
+ if (tableDataManager instanceof RealtimeTableDataManager) {
+ numConsumingSegments = 0;
+ numOnlineSegments = 0;
+ minIndexTimeMs = Long.MAX_VALUE;
+ minIngestionTimeMs = Long.MAX_VALUE;
+ maxEndTimeMs = Long.MIN_VALUE;
+ for (IndexSegment indexSegment : indexSegments) {
+ if (indexSegment instanceof MutableSegment) {
+ numConsumingSegments += 1;
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
+ if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
+ minIndexTimeMs = indexTimeMs;
+ }
+ long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
+ if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
+ minIngestionTimeMs = ingestionTimeMs;
+ }
+ } else if (indexSegment instanceof ImmutableSegment) {
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ numOnlineSegments++;
+ if (timeInterval != null) {
+ maxEndTimeMs = Math.max(maxEndTimeMs, timeInterval.getEndMillis());
Review Comment:
segmentMetadata.getTimeInterval() gives you the time boundary on the values of the time column, not when the segment was completed. The values of time column is user defined and we shouldn't assume that it aligns with the actual time of rows being ingested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee merged pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee merged PR #8911:
URL: https://github.com/apache/pinot/pull/8911
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r903239715
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
@Jackie-Jiang The original logic that we try to compute was the following. Do you think that it's better to keep the following to show what we are attempting to do instead of reducing the condition check?
```
if (_numConsumingSegmentsProcessed > 0 || (_numConsumingSegmentsProcessed == 0 && _minConsumingFreshnessTimeMs != Long.MAX_VALUE)) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r899658714
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
+ // case: no immutable segments, no metric emitted
+ LOGGER.warn("Request {} queried no consuming or online segments, with minConsumingFreshnessTimeMs: {}", requestId,
Review Comment:
My reasoning was that we already faced not-happy-path when `numConsumingSegments == 0` so having a lot of lines of the same log every time the query runs when this condition is met probably would not help too much. (Current code only logs `debug` when numConsumingSegments ==0 but that's already not-happy situation.)
Instead, we should find the issue by the alert when `numConsumingSegments == 0` and this is being addressed by the metric that will be added by @sajjad-moradi . How do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r921692258
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -195,19 +198,35 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
// Gather stats for realtime consuming segments
int numConsumingSegments = 0;
- long minIndexTimeMs = Long.MAX_VALUE;
- long minIngestionTimeMs = Long.MAX_VALUE;
- for (IndexSegment indexSegment : indexSegments) {
- if (indexSegment instanceof MutableSegment) {
- numConsumingSegments += 1;
- SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
- long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
- if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
- minIndexTimeMs = indexTimeMs;
- }
- long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
- if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
- minIngestionTimeMs = ingestionTimeMs;
+ int numOnlineSegments = 0;
+ long minIndexTimeMs = 0;
+ long minIngestionTimeMs = 0;
+ long maxEndTimeMs = 0;
+ if (tableDataManager instanceof RealtimeTableDataManager) {
+ numConsumingSegments = 0;
+ numOnlineSegments = 0;
+ minIndexTimeMs = Long.MAX_VALUE;
+ minIngestionTimeMs = Long.MAX_VALUE;
+ maxEndTimeMs = Long.MIN_VALUE;
+ for (IndexSegment indexSegment : indexSegments) {
+ if (indexSegment instanceof MutableSegment) {
+ numConsumingSegments += 1;
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
+ if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
+ minIndexTimeMs = indexTimeMs;
+ }
+ long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
+ if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
+ minIngestionTimeMs = ingestionTimeMs;
+ }
+ } else if (indexSegment instanceof ImmutableSegment) {
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ numOnlineSegments++;
+ if (timeInterval != null) {
+ maxEndTimeMs = Math.max(maxEndTimeMs, timeInterval.getEndMillis());
Review Comment:
@sajjad-moradi Your statement applies to `MutableSegment`. In that case, we call `getLastIndexedTimestamp() / getLatestIngestionTimestamp()`. We call `getTimeInverval()` for `ImmutableSegment`. In this case, it indeed returns the correct start/end time of the segment.
```
SegmentMetadataImple:setTimeInfo()
_segmentStartTime = Long.parseLong(startTimeString);
_segmentEndTime = Long.parseLong(endTimeString);
_timeInterval =
new Interval(_timeUnit.toMillis(_segmentStartTime), _timeUnit.toMillis(_segmentEndTime), DateTimeZone.UTC);
```
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -195,19 +198,35 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
// Gather stats for realtime consuming segments
int numConsumingSegments = 0;
- long minIndexTimeMs = Long.MAX_VALUE;
- long minIngestionTimeMs = Long.MAX_VALUE;
- for (IndexSegment indexSegment : indexSegments) {
- if (indexSegment instanceof MutableSegment) {
- numConsumingSegments += 1;
- SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
- long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
- if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
- minIndexTimeMs = indexTimeMs;
- }
- long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
- if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
- minIngestionTimeMs = ingestionTimeMs;
+ int numOnlineSegments = 0;
+ long minIndexTimeMs = 0;
+ long minIngestionTimeMs = 0;
+ long maxEndTimeMs = 0;
+ if (tableDataManager instanceof RealtimeTableDataManager) {
+ numConsumingSegments = 0;
+ numOnlineSegments = 0;
+ minIndexTimeMs = Long.MAX_VALUE;
+ minIngestionTimeMs = Long.MAX_VALUE;
+ maxEndTimeMs = Long.MIN_VALUE;
+ for (IndexSegment indexSegment : indexSegments) {
+ if (indexSegment instanceof MutableSegment) {
+ numConsumingSegments += 1;
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
+ if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
+ minIndexTimeMs = indexTimeMs;
+ }
+ long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
+ if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
+ minIngestionTimeMs = ingestionTimeMs;
+ }
+ } else if (indexSegment instanceof ImmutableSegment) {
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ numOnlineSegments++;
+ if (timeInterval != null) {
+ maxEndTimeMs = Math.max(maxEndTimeMs, timeInterval.getEndMillis());
Review Comment:
@sajjad-moradi Your statement applies to `MutableSegment`. In that case, we call `getLastIndexedTimestamp() / getLatestIngestionTimestamp()`. We call `getTimeInverval()` for `ImmutableSegment`. In this case, it indeed returns the correct start/end time of the segment.
```
SegmentMetadataImpl:setTimeInfo()
_segmentStartTime = Long.parseLong(startTimeString);
_segmentEndTime = Long.parseLong(endTimeString);
_timeInterval =
new Interval(_timeUnit.toMillis(_segmentStartTime), _timeUnit.toMillis(_segmentEndTime), DateTimeZone.UTC);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r900625973
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
Review Comment:
it was checking before on`NUM_CONSuMING_SEGMENTS_PROCESSED > 0 `, now changed to check on `_numConsumingSegmentsProcessed != Long.MAX_VALUE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r903239715
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
@PrachiKhobragade We can probably put it as the following to exactly show what we would like to achieve?
```
if (_numConsumingSegmentsProcessed > 0 || (_numConsumingSegmentsProcessed == 0 && _minConsumingFreshnessTimeMs != Long.MAX_VALUE)) {
...
}
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r899651231
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
+ // case: no immutable segments, no metric emitted
+ LOGGER.warn("Request {} queried no consuming or online segments, with minConsumingFreshnessTimeMs: {}", requestId,
Review Comment:
shouldn't it be warn because this block shouldn't reach in happy-path, something is wrong here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r922888568
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -195,19 +198,35 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
// Gather stats for realtime consuming segments
int numConsumingSegments = 0;
- long minIndexTimeMs = Long.MAX_VALUE;
- long minIngestionTimeMs = Long.MAX_VALUE;
- for (IndexSegment indexSegment : indexSegments) {
- if (indexSegment instanceof MutableSegment) {
- numConsumingSegments += 1;
- SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
- long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
- if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
- minIndexTimeMs = indexTimeMs;
- }
- long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
- if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
- minIngestionTimeMs = ingestionTimeMs;
+ int numOnlineSegments = 0;
+ long minIndexTimeMs = 0;
+ long minIngestionTimeMs = 0;
+ long maxEndTimeMs = 0;
+ if (tableDataManager instanceof RealtimeTableDataManager) {
+ numConsumingSegments = 0;
+ numOnlineSegments = 0;
+ minIndexTimeMs = Long.MAX_VALUE;
+ minIngestionTimeMs = Long.MAX_VALUE;
+ maxEndTimeMs = Long.MIN_VALUE;
+ for (IndexSegment indexSegment : indexSegments) {
+ if (indexSegment instanceof MutableSegment) {
+ numConsumingSegments += 1;
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
+ if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
+ minIndexTimeMs = indexTimeMs;
+ }
+ long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
+ if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
+ minIngestionTimeMs = ingestionTimeMs;
+ }
+ } else if (indexSegment instanceof ImmutableSegment) {
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ numOnlineSegments++;
+ if (timeInterval != null) {
+ maxEndTimeMs = Math.max(maxEndTimeMs, timeInterval.getEndMillis());
Review Comment:
I was actually talking about immutable segments, not mutable ones!
The values for `startTimeString` and `endTimeString` variables are taken from the min and max value of the time column:
```java
SegmentColumnarIndexCreator.writeMetadata() {
...
if (_config.getStartTime() != null) {
startTime = Long.parseLong(_config.getStartTime());
endTime = Long.parseLong(_config.getEndTime());
timeUnit = Preconditions.checkNotNull(_config.getSegmentTimeUnit());
} else {
if (_totalDocs > 0) {
HERE-> String startTimeStr = timeColumnIndexCreationInfo.getMin().toString();
HERE-> String endTimeStr = timeColumnIndexCreationInfo.getMax().toString();
if (_config.getTimeColumnType() == SegmentGeneratorConfig.TimeColumnType.SIMPLE_DATE) {
// For TimeColumnType.SIMPLE_DATE_FORMAT, convert time value into millis since epoch
// Use DateTimeFormatter from DateTimeFormatSpec to handle default time zone consistently.
DateTimeFormatSpec formatSpec = _config.getDateTimeFormatSpec();
Preconditions.checkNotNull(formatSpec, "DateTimeFormatSpec must exist for SimpleDate");
DateTimeFormatter dateTimeFormatter = formatSpec.getDateTimeFormatter();
startTime = dateTimeFormatter.parseMillis(startTimeStr);
endTime = dateTimeFormatter.parseMillis(endTimeStr);
timeUnit = TimeUnit.MILLISECONDS;
} else {
// by default, time column type is TimeColumnType.EPOCH
startTime = Long.parseLong(startTimeStr);
endTime = Long.parseLong(endTimeStr);
timeUnit = Preconditions.checkNotNull(_config.getSegmentTimeUnit());
}
} else {
// No records in segment. Use current time as start/end
long now = System.currentTimeMillis();
if (_config.getTimeColumnType() == SegmentGeneratorConfig.TimeColumnType.SIMPLE_DATE) {
startTime = now;
endTime = now;
timeUnit = TimeUnit.MILLISECONDS;
} else {
timeUnit = Preconditions.checkNotNull(_config.getSegmentTimeUnit());
startTime = timeUnit.convert(now, TimeUnit.MILLISECONDS);
endTime = timeUnit.convert(now, TimeUnit.MILLISECONDS);
}
}
}
if (!_config.isSkipTimeValueCheck()) {
Interval timeInterval =
new Interval(timeUnit.toMillis(startTime), timeUnit.toMillis(endTime), DateTimeZone.UTC);
Preconditions.checkState(TimeUtils.isValidTimeInterval(timeInterval),
"Invalid segment start/end time: %s (in millis: %s/%s) for time column: %s, must be between: %s",
timeInterval, timeInterval.getStartMillis(), timeInterval.getEndMillis(), timeColumnName,
TimeUtils.VALID_TIME_INTERVAL);
}
HERE-> properties.setProperty(SEGMENT_START_TIME, startTime);
HERE-> properties.setProperty(SEGMENT_END_TIME, endTime);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on PR #8911:
URL: https://github.com/apache/pinot/pull/8911#issuecomment-1158278458
@snleee Please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r905578583
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
I see, the current code is okay. Is it possible that server returns a non-positive `minConsumingFreshnessTimeMs` in the current code? If so, this change might be backward incompatible; if not, then the change is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #8911:
URL: https://github.com/apache/pinot/pull/8911#issuecomment-1165754151
> +1 on maintaining the freshness lag for each table at ingestion time, which is independent of the segments queried. We might also consider maintaining it for each partition
There are two things here:
- Consumption lag (I think this is what you are referring to when you +1). https://github.com/apache/pinot/issues/5948. This changes each time we consume an event or if an event is produced to the stream.
- Freshness (or Staleness) lag: This is answers the question : "How old is the newest data we considered while answering a query?" This is computed each time a query is answered.
It is this second one that we are trying to address here. The first one is still open: https://github.com/apache/pinot/issues/5948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r901983227
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
Why removing the first part of the check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r943625219
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -197,19 +200,41 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
// Gather stats for realtime consuming segments
int numConsumingSegmentsQueried = 0;
- long minIndexTimeMs = Long.MAX_VALUE;
- long minIngestionTimeMs = Long.MAX_VALUE;
- for (IndexSegment indexSegment : indexSegments) {
- if (indexSegment instanceof MutableSegment) {
- numConsumingSegmentsQueried += 1;
- SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
- long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
- if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
- minIndexTimeMs = indexTimeMs;
- }
- long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
- if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
- minIngestionTimeMs = ingestionTimeMs;
+ int numOnlineSegments = 0;
+ long minIndexTimeMs = 0;
+ long minIngestionTimeMs = 0;
+ long maxEndTimeMs = 0;
+ if (tableDataManager instanceof RealtimeTableDataManager) {
+ numConsumingSegmentsQueried = 0;
+ numOnlineSegments = 0;
+ minIndexTimeMs = Long.MAX_VALUE;
+ minIngestionTimeMs = Long.MAX_VALUE;
+ maxEndTimeMs = Long.MIN_VALUE;
+ for (IndexSegment indexSegment : indexSegments) {
+ if (indexSegment instanceof MutableSegment) {
+ numConsumingSegmentsQueried += 1;
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
+ if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
+ minIndexTimeMs = indexTimeMs;
+ }
+ long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
+ if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
+ minIngestionTimeMs = ingestionTimeMs;
+ }
+ } else if (indexSegment instanceof ImmutableSegment) {
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexCreationTime = segmentMetadata.getIndexCreationTime();
+ numOnlineSegments++;
+ if (indexCreationTime != Long.MIN_VALUE) {
+ maxEndTimeMs = Math.max(maxEndTimeMs, indexCreationTime);
Review Comment:
instead of `maxEndTimeMs`, let's rename this to `maxIndexCreationTime`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r943625219
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -197,19 +200,41 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
// Gather stats for realtime consuming segments
int numConsumingSegmentsQueried = 0;
- long minIndexTimeMs = Long.MAX_VALUE;
- long minIngestionTimeMs = Long.MAX_VALUE;
- for (IndexSegment indexSegment : indexSegments) {
- if (indexSegment instanceof MutableSegment) {
- numConsumingSegmentsQueried += 1;
- SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
- long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
- if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
- minIndexTimeMs = indexTimeMs;
- }
- long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
- if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
- minIngestionTimeMs = ingestionTimeMs;
+ int numOnlineSegments = 0;
+ long minIndexTimeMs = 0;
+ long minIngestionTimeMs = 0;
+ long maxEndTimeMs = 0;
+ if (tableDataManager instanceof RealtimeTableDataManager) {
+ numConsumingSegmentsQueried = 0;
+ numOnlineSegments = 0;
+ minIndexTimeMs = Long.MAX_VALUE;
+ minIngestionTimeMs = Long.MAX_VALUE;
+ maxEndTimeMs = Long.MIN_VALUE;
+ for (IndexSegment indexSegment : indexSegments) {
+ if (indexSegment instanceof MutableSegment) {
+ numConsumingSegmentsQueried += 1;
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexTimeMs = segmentMetadata.getLastIndexedTimestamp();
+ if (indexTimeMs != Long.MIN_VALUE && indexTimeMs < minIndexTimeMs) {
+ minIndexTimeMs = indexTimeMs;
+ }
+ long ingestionTimeMs = segmentMetadata.getLatestIngestionTimestamp();
+ if (ingestionTimeMs != Long.MIN_VALUE && ingestionTimeMs < minIngestionTimeMs) {
+ minIngestionTimeMs = ingestionTimeMs;
+ }
+ } else if (indexSegment instanceof ImmutableSegment) {
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ long indexCreationTime = segmentMetadata.getIndexCreationTime();
+ numOnlineSegments++;
+ if (indexCreationTime != Long.MIN_VALUE) {
+ maxEndTimeMs = Math.max(maxEndTimeMs, indexCreationTime);
Review Comment:
instead of `maxEndTimeMs`, let's rename this to `maxIndexCreationTime`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on PR #8911:
URL: https://github.com/apache/pinot/pull/8911#issuecomment-1163303596
@mcvsubbu I discussed this with @PrachiKhobragade yesterday.
1. The server-side time pruner is covered if that's your concern. We compute the segment's min/max timestamp before we run the segment pruner. I think that this is covering most Pinot cases. (100% of use cases at LNKD fall into this bucket)
2. If the broker-side time pruner is used. This can be a concern. We have 2 ways to solve this:
```
1. We check the query object and check if there's a time predicate. If there is, we don't emit a metric.
2. We check query metadata for all-consuming segments even if the broker prunes those.
3. Create a shared atomic long value on the server and cache the most recent freshness metrics and use that number when reporting.
```
All approaches that I can think of will add more CPU cycles to the performance-critical query path. I don't think that it's worth making the above changes to achieve the complete correctness of the freshness metric. IMO, we probably need to re-think our freshness metrics (computing the freshness metrics on the query time looks not the best approach). I would prefer that the metrics should be emitted periodically in the background, not interfering with the query execution path but this is more like the long-term change.
I would prefer to add the comment that the metric would not behave as expected if the broker-side time-based pruner is being used and address the issue in the long-term re-do for the freshness metric.
Please let me know if you have a better approach to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r902946032
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
_minConsumingFreshnessTimeMs may have value even when _numConsumingSegmentsProcessed equals 0, in some error scenarios. Previously _numConsumingSegmentsProcessed > 0 was used in a way to check if consuming segments are present and emit metrics only when the condition is reached. Now we want to emit metrics even if no consuming segments are present, so the first part becomes redundant and can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on PR #8911:
URL: https://github.com/apache/pinot/pull/8911#issuecomment-1162605655
what happens if the query has time filters that causes time based pruner to eliminate consuming segments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r900618682
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r901982868
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -288,7 +288,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerResponseNative.setNumSegmentsPrunedByServer(_numSegmentsPrunedByServer);
brokerResponseNative.setExplainPlanNumEmptyFilterSegments(_explainPlanNumEmptyFilterSegments);
brokerResponseNative.setExplainPlanNumMatchAllFilterSegments(_explainPlanNumMatchAllFilterSegments);
- if (_numConsumingSegmentsProcessed > 0) {
+ if (_numConsumingSegmentsProcessed != Long.MAX_VALUE) {
Review Comment:
Is this correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r902939512
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -288,7 +288,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerResponseNative.setNumSegmentsPrunedByServer(_numSegmentsPrunedByServer);
brokerResponseNative.setExplainPlanNumEmptyFilterSegments(_explainPlanNumEmptyFilterSegments);
brokerResponseNative.setExplainPlanNumMatchAllFilterSegments(_explainPlanNumMatchAllFilterSegments);
- if (_numConsumingSegmentsProcessed > 0) {
+ if (_numConsumingSegmentsProcessed != Long.MAX_VALUE) {
Review Comment:
corrected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8911:
URL: https://github.com/apache/pinot/pull/8911#issuecomment-1162211207
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8911?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8911](https://codecov.io/gh/apache/pinot/pull/8911?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4ca33f2) into [master](https://codecov.io/gh/apache/pinot/commit/d437d3a80494828889be7b07d61ddce21a2d781e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d437d3a) will **decrease** coverage by `6.85%`.
> The diff coverage is `11.90%`.
```diff
@@ Coverage Diff @@
## master #8911 +/- ##
============================================
- Coverage 69.82% 62.96% -6.86%
- Complexity 4684 4869 +185
============================================
Files 1808 1770 -38
Lines 94276 92662 -1614
Branches 14057 13933 -124
============================================
- Hits 65824 58349 -7475
- Misses 23883 30079 +6196
+ Partials 4569 4234 -335
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `66.26% <11.90%> (-0.14%)` | :arrow_down: |
| unittests2 | `14.97% <0.00%> (+0.08%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8911?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/pinot/core/query/reduce/BaseReduceService.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvQmFzZVJlZHVjZVNlcnZpY2UuamF2YQ==) | `78.69% <0.00%> (-15.39%)` | :arrow_down: |
| [...core/query/executor/ServerQueryExecutorV1Impl.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=) | `65.92% <12.50%> (-17.02%)` | :arrow_down: |
| [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...apache/pinot/common/helix/ExtraInstanceConfig.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vaGVsaXgvRXh0cmFJbnN0YW5jZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/pinot/server/access/AccessControlFactory.java](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL2FjY2Vzcy9BY2Nlc3NDb250cm9sRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [490 more](https://codecov.io/gh/apache/pinot/pull/8911/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8911?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8911?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d437d3a...4ca33f2](https://codecov.io/gh/apache/pinot/pull/8911?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #8911:
URL: https://github.com/apache/pinot/pull/8911#issuecomment-1165733880
+1 on maintaining the freshness lag for each table at ingestion time, which is independent of the segments queried. We might also consider maintaining it for each partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r905598986
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BaseReduceService.java:
##########
@@ -318,7 +318,7 @@ protected void setStats(String rawTableName, BrokerResponseNative brokerResponse
brokerMetrics.addTimedTableValue(rawTableName, BrokerTimer.REALTIME_TOTAL_CPU_TIME_NS, _realtimeTotalCpuTimeNs,
TimeUnit.NANOSECONDS);
- if (_numConsumingSegmentsProcessed > 0 && _minConsumingFreshnessTimeMs > 0) {
+ if (_minConsumingFreshnessTimeMs != Long.MAX_VALUE) {
Review Comment:
The change in behavior of `minConsumingFreshnessTimeMs` is update it to maxEndTIme if `numConsumingSegments == 0` maxEndTIme is end of the segment timeInterval, so should never be negative.
we update the `minConsumingFreshnessTimeMs` only when `maxEndTIme != LONG.MIN` so `minConsumingFreshnessTimeMs` shouldn't get a -ve value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] PrachiKhobragade commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
PrachiKhobragade commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r900618313
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
+ // case: no immutable segments, no metric emitted
+ LOGGER.warn("Request {} queried no consuming or online segments, with minConsumingFreshnessTimeMs: {}", requestId,
Review Comment:
removed this block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a diff in pull request #8911: add freshness lag when there are no consuming segments
Posted by GitBox <gi...@apache.org>.
snleee commented on code in PR #8911:
URL: https://github.com/apache/pinot/pull/8911#discussion_r899658714
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
+ // case: no immutable segments, no metric emitted
+ LOGGER.warn("Request {} queried no consuming or online segments, with minConsumingFreshnessTimeMs: {}", requestId,
Review Comment:
My reasoning was that we already faced not-happy-path when `numConsumingSegments ==0 0` so having a lot of lines of the same log every time the query runs when this condition is met probably would not help too much. (Current code only logs `debug` when numConsumingSegments ==0 but that's already not-happy situation.
Instead, we should find the issue by the alert when `numConsumingSegments == 0` and this is being addressed by the metric that will be added by @sajjad-moradi . How do you think?
##########
pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java:
##########
@@ -261,12 +272,22 @@ private DataTable processQueryInternal(ServerQueryRequest queryRequest, Executor
_serverMetrics.addMeteredTableValue(tableNameWithType, ServerMeter.NUM_MISSING_SEGMENTS, numMissingSegments);
}
+ long minConsumingFreshnessTimeMs = Long.MAX_VALUE;
if (numConsumingSegments > 0) {
- long minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
- LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
- numConsumingSegments, minConsumingFreshnessTimeMs);
+ minConsumingFreshnessTimeMs = minIngestionTimeMs != Long.MAX_VALUE ? minIngestionTimeMs : minIndexTimeMs;
metadata.put(MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED.getName(), Integer.toString(numConsumingSegments));
metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(minConsumingFreshnessTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numConsumingSegments == 0 && maxEndTimeMs != Long.MIN_VALUE) {
+ minConsumingFreshnessTimeMs = maxEndTimeMs;
+ metadata.put(MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS.getName(), Long.toString(maxEndTimeMs));
+ LOGGER.debug("Request {} queried {} consuming segments with minConsumingFreshnessTimeMs: {}", requestId,
+ numConsumingSegments, minConsumingFreshnessTimeMs);
+ } else if (numOnlineSegments == 0) {
+ // case: no immutable segments, no metric emitted
+ LOGGER.warn("Request {} queried no consuming or online segments, with minConsumingFreshnessTimeMs: {}", requestId,
Review Comment:
My reasoning was that we already faced not-happy-path when `numConsumingSegments == 0` so having a lot of lines of the same log every time the query runs when this condition is met probably would not help too much. (Current code only logs `debug` when numConsumingSegments ==0 but that's already not-happy situation.
Instead, we should find the issue by the alert when `numConsumingSegments == 0` and this is being addressed by the metric that will be added by @sajjad-moradi . How do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org