You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2024/03/22 16:07:58 UTC
[PR] Multi stage stats [pinot]
gortiz opened a new pull request, #12704:
URL: https://github.com/apache/pinot/pull/12704
At this moment this PR is a draft where I'm trying to modify the way stats are collected and encoded in multi-stage engine.
The plan is to:
1. Use enums as stat keys
2. Collect stats per stage.
* Each OpChain will collect a partial view for its stage and the stages upstream.
* OpChains that receive data from more than one OpChain of the same stage will merge them.
* The root stage (on the broker), where there is only one OpChain, will collect the full view.
3. Change the way stats are encoded. Instead of JSON we will use a binary format equal or more effective than the one used in DataTable.
4. Create new stats for specific multi-stage operators where it makes sense
5. Export these stats to broker clients
Current state:
[*] Define the StatMap that will map from enums to values.
[*] Define the encoding
[*] Define the StageStats
[ ] Substitute OpChainStats with StageStats in OpChain
[ ] Merge StageStats in receiving mailboxes.
[ ] Return the StageStats to broker clients
[ ] Create new stats for specific multi-stage operators where it makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1580748812
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -19,107 +19,295 @@
package org.apache.pinot.query.runtime.operator;
import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import java.io.DataInput;
+import java.io.IOException;
import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.common.response.broker.BrokerResponseNativeV2;
import org.apache.pinot.core.common.Operator;
import org.apache.pinot.query.runtime.blocks.TransferableBlock;
import org.apache.pinot.query.runtime.blocks.TransferableBlockUtils;
+import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
import org.apache.pinot.query.runtime.plan.OpChainExecutionContext;
+import org.apache.pinot.query.runtime.plan.pipeline.PipelineBreakerOperator;
import org.apache.pinot.spi.exception.EarlyTerminationException;
import org.apache.pinot.spi.trace.InvocationScope;
import org.apache.pinot.spi.trace.Tracing;
import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-public abstract class MultiStageOperator implements Operator<TransferableBlock>, AutoCloseable {
- private static final Logger LOGGER = LoggerFactory.getLogger(MultiStageOperator.class);
+public abstract class MultiStageOperator<K extends Enum<K> & StatMap.Key>
+ implements Operator<TransferableBlock>, AutoCloseable {
protected final OpChainExecutionContext _context;
protected final String _operatorId;
- protected final OpChainStats _opChainStats;
+ protected final StatMap<K> _statMap;
protected boolean _isEarlyTerminated;
- public MultiStageOperator(OpChainExecutionContext context) {
+ public MultiStageOperator(OpChainExecutionContext context, Class<K> keyStatClass) {
_context = context;
_operatorId = Joiner.on("_").join(getClass().getSimpleName(), _context.getStageId(), _context.getServer());
- _opChainStats = _context.getStats();
_isEarlyTerminated = false;
+ _statMap = new StatMap<>(keyStatClass);
}
+ protected abstract Logger logger();
+
+ public abstract Type getOperatorType();
+
+ public abstract K getExecutionTimeKey();
+
+ public abstract K getEmittedRowsKey();
+
@Override
public TransferableBlock nextBlock() {
if (Tracing.ThreadAccountantOps.isInterrupted()) {
throw new EarlyTerminationException("Interrupted while processing next block");
}
+ if (logger().isDebugEnabled()) {
+ logger().debug("Operator {}: Reading next block", _operatorId);
+ }
try (InvocationScope ignored = Tracing.getTracer().createScope(getClass())) {
TransferableBlock nextBlock;
if (shouldCollectStats()) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, _operatorId);
- operatorStats.startTimer();
+ Stopwatch executeStopwatch = Stopwatch.createStarted();
Review Comment:
I've finally decided to always collect stats because this was difficult to explain and the cost is negligible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12704:
URL: https://github.com/apache/pinot/pull/12704#issuecomment-2078382275
Regarding the race condition, it might be related to the early termination. I'll also look more into it, but I remember seeing similar issues for other classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1584415369
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -135,14 +160,128 @@ protected TransferableBlock getNextBlock()
}
}
- private TransferableBlock constructMetadataBlock() {
- // All data blocks have been returned. Record the stats and return EOS.
- Map<String, String> executionStats = _executionStats;
+ private void mergeExecutionStats(@Nullable Map<String, String> executionStats) {
if (executionStats != null) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, getOperatorId());
- operatorStats.recordExecutionStats(executionStats);
+ for (Map.Entry<String, String> entry : executionStats.entrySet()) {
+ DataTable.MetadataKey key = DataTable.MetadataKey.getByName(entry.getKey());
+ if (key == null) {
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ continue;
+ }
+ switch (key) {
+ case UNKNOWN:
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ break;
+ case TABLE:
Review Comment:
I've removed OPERATOR_ID because there was no use for it.
Right now the TABLE stat is used in `MultiStageBrokerRequestHandler` to build the `stageStats` json object (the one that is shown in the description of this PR). There we use the Table stat to introduce the name of the table scanned when the json node of the leaf stage is created. In theory we don't need this here, we can infer the name of the table from the PlanNode, but there are cases where the PlanNode tree is quite different to the actual physical plan, so it is more difficult to do than expected.
There was another place where it was used, in `ResourceBasedQueriesTest.testQueryTestCasesWithMetadata`, but I ended up finding a way to do not need it. The previous version of the code was honestly simpler, but I guess the new version is better because ideally we would like to remove the TABLE stat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1586072327
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
I planned to move V1MetadataBlock to the test classes, but I found some issues and I think the correct thing is to keep them here.
It looks like we cannot always return `new Metadata(byteBuffer)` because errors are encoded in a different way. Specifically, the metadata type (whether it is a eos or an error) was encoded in the variable part in V1 and in the fixed part in V2.
The change is still forward compatible (V1 can read V2) and with these lines, V2 can also read V1. I've created more test to verify that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565236559
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
Review Comment:
Are we going to use different version for different block types (data block is still with v1)? I feel it will be safer to have all data blocks in the same version. Since we are not going to really maintain 2 versions and add flag to switch, should we just keep it as v1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565545708
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/MultiStageQueryStats.java:
##########
@@ -0,0 +1,668 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.plan;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.avro.util.ByteBufferInputStream;
+import org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.query.runtime.operator.BaseMailboxReceiveOperator;
+import org.apache.pinot.query.runtime.operator.LeafStageTransferableBlockOperator;
+import org.apache.pinot.query.runtime.operator.LiteralValueOperator;
+import org.apache.pinot.query.runtime.operator.MailboxSendOperator;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The stats of a given query.
+ * <p>
+ * For the same query, multiple instances of this class may exist. Each of them will have a partial view of the stats.
+ * Specifically, while the query is being executed, each operator will return its own partial view of the stats when
+ * EOS block is sent.
+ * <p>
+ * Simple operations with a single upstream, like filters or transforms, would just add their own information to the
+ * stats. More complex operations, like joins or receiving mailboxes, will merge the stats from all their upstreams and
+ * add their own stats.
+ * <p>
+ * The complete stats for the query are obtained in the execution root (usually the broker) by merging the partial
+ * views.
+ * <p>
+ * In order to reduce allocation, this class is mutable. Some operators may create their own stats, but most of them
+ * will receive a stats object from the upstream operator and modify it by adding their own stats and sometimes merging
+ * them with other upstream stats.
+ */
+public class MultiStageQueryStats {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MultiStageQueryStats.class);
+ private final int _currentStageId;
+ private final StageStats.Open _currentStats;
+ /**
+ * Known stats for stages whose id is higher than the current one.
+ * <p>
+ * A stage may not know all the stats whose id is higher than the current one, so this list may contain null values.
+ * It may also grow in size when different merge methods are called.
+ * <p>
+ * For example the stats of the left hand side of a join may know stats of stages 3 and 4 and the right side may know
+ * stats of stages 5. When merging the stats of the join, the stats of stages 5 will be added to this list.
+ *
+ * @see #mergeUpstream(List)
+ * @see #mergeUpstream(MultiStageQueryStats)
+ * @see #mergeInOrder(MultiStageQueryStats, MultiStageOperator.Type, StatMap)
+ */
+ private final ArrayList<StageStats.Closed> _closedStats;
+ private static final MultiStageOperator.Type[] ALL_TYPES = MultiStageOperator.Type.values();
+
+ private MultiStageQueryStats(int stageId) {
+ _currentStageId = stageId;
+ _currentStats = new StageStats.Open();
+ _closedStats = new ArrayList<>();
+ }
+
+ private static MultiStageQueryStats create(int stageId, MultiStageOperator.Type type, @Nullable StatMap<?> opStats) {
+ MultiStageQueryStats multiStageQueryStats = new MultiStageQueryStats(stageId);
+ multiStageQueryStats.getCurrentStats().addLastOperator(type, opStats);
+ return multiStageQueryStats;
+ }
+
+ public static MultiStageQueryStats emptyStats(int stageId) {
+ return new MultiStageQueryStats(stageId);
+ }
+
+ public static MultiStageQueryStats createLeaf(int stageId,
+ StatMap<LeafStageTransferableBlockOperator.StatKey> opStats) {
+ return create(stageId, MultiStageOperator.Type.LEAF, opStats);
+ }
+
+ public static MultiStageQueryStats createLiteral(int stageId, StatMap<LiteralValueOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.LITERAL, statMap);
+ }
+
+ public static MultiStageQueryStats createCancelledSend(int stageId,
+ StatMap<MailboxSendOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_SEND, statMap);
+ }
+
+ public static MultiStageQueryStats createReceive(int stageId, StatMap<BaseMailboxReceiveOperator.StatKey> stats) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_RECEIVE, stats);
+ }
+
+ public int getCurrentStageId() {
+ return _currentStageId;
+ }
+
+ /**
+ * Serialize the current stats in a way it is compatible with {@link #mergeUpstream(List)}.
+ * <p>
+ * The serialized stats are returned in a list where the index is the stage id. Stages downstream or not related to
+ * the current one will be null.
+ */
+ public List<ByteBuffer> serialize()
+ throws IOException {
+
+ ArrayList<ByteBuffer> serializedStats = new ArrayList<>(getMaxStageId());
+ for (int i = 0; i < _currentStageId; i++) {
+ serializedStats.add(null);
+ }
+
+ try (UnsynchronizedByteArrayOutputStream baos = new UnsynchronizedByteArrayOutputStream.Builder().get();
+ DataOutputStream output = new DataOutputStream(baos)) {
+
+ _currentStats.serialize(output);
+ ByteBuffer currentBuf = ByteBuffer.wrap(baos.toByteArray());
+
+ serializedStats.add(currentBuf);
+
+ for (StageStats.Closed closedStats : _closedStats) {
+ if (closedStats == null) {
+ serializedStats.add(null);
+ continue;
+ }
+ baos.reset();
+ closedStats.serialize(output);
+ ByteBuffer buf = ByteBuffer.wrap(baos.toByteArray());
+ serializedStats.add(buf);
+ }
+ }
+ Preconditions.checkState(serializedStats.size() == getMaxStageId() + 1,
+ "Serialized stats size is different from expected size. Expected %s, got %s",
+ getMaxStageId() + 1, serializedStats.size());
+ return serializedStats;
+ }
+
+ public StageStats.Open getCurrentStats() {
+ return _currentStats;
+ }
+
+ /**
+ * Returns the higher stage id known by this object.
+ */
+ public int getMaxStageId() {
+ return _currentStageId + _closedStats.size();
+ }
+
+ /**
+ * Get the stats of a stage whose id is higher than the current one.
+ * <p>
+ * This method returns null in case the stage id is unknown by this stage or no stats are stored for it.
+ */
+ @Nullable
+ public StageStats.Closed getUpstreamStageStats(int stageId) {
+ if (stageId <= _currentStageId) {
+ throw new IllegalArgumentException("Stage " + stageId + " cannot be upstream of current stage "
+ + _currentStageId);
+ }
+
+ int index = stageId - _currentStageId - 1;
+ if (index >= _closedStats.size()) {
+ return null;
+ }
+ return _closedStats.get(index);
+ }
+
+ public void mergeInOrder(MultiStageQueryStats otherStats, MultiStageOperator.Type type,
+ StatMap<?> statMap) {
+ Preconditions.checkArgument(_currentStageId == otherStats._currentStageId,
+ "Cannot merge stats from different stages (%s and %s)", _currentStageId, otherStats._currentStageId);
+ mergeUpstream(otherStats);
+ StageStats.Open currentStats = getCurrentStats();
+ currentStats.concat(otherStats.getCurrentStats());
+ currentStats.addLastOperator(type, statMap);
+ }
+
+ private void growUpToStage(int stageId) {
+ _closedStats.ensureCapacity(stageId - _currentStageId);
+ while (getMaxStageId() < stageId) {
+ _closedStats.add(null);
+ }
+ }
+
+ /**
+ * Merge upstream stats from another MultiStageQueryStats object into this one.
+ * <p>
+ * Only the stages whose id is higher than the current one are merged. The reason to do so is that upstream stats
+ * should be already closed while current stage may need some extra tuning.
+ * <p>
+ * For example set operations may need to merge the stats from all its upstreams before concatenating stats of the
+ * current stage.
+ */
+ public void mergeUpstream(MultiStageQueryStats otherStats) {
+ Preconditions.checkArgument(_currentStageId <= otherStats._currentStageId,
+ "Cannot merge stats from early stage %s into stats of later stage %s",
+ otherStats._currentStageId, _currentStageId);
+
+ growUpToStage(otherStats.getMaxStageId());
+
+ int currentDiff = otherStats._currentStageId - _currentStageId;
+ if (currentDiff > 0) {
+ StageStats.Closed close = otherStats._currentStats.close();
+ int selfIdx = currentDiff - 1;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ if (myStats == null) {
+ _closedStats.set(selfIdx, close);
+ } else {
+ myStats.merge(close);
+ }
+ }
+
+ for (int i = 0; i < otherStats._closedStats.size(); i++) {
+ StageStats.Closed otherStatsForStage = otherStats._closedStats.get(i);
+ if (otherStatsForStage == null) {
+ continue;
+ }
+ int selfIdx = i + currentDiff;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ try {
+ if (myStats == null) {
+ _closedStats.set(selfIdx, otherStatsForStage);
+ assert getUpstreamStageStats(i + otherStats._currentStageId + 1) == otherStatsForStage;
+ } else {
+ myStats.merge(otherStatsForStage);
+ }
+ } catch (IllegalArgumentException | IllegalStateException ex) {
+ LOGGER.warn("Error merging stats on stage " + i + ". Ignoring the new stats", ex);
+ }
+ }
+ }
+
+ public void mergeUpstream(List<ByteBuffer> otherStats) {
+ for (int i = 0; i <= _currentStageId && i < otherStats.size(); i++) {
+ if (otherStats.get(i) != null) {
+ throw new IllegalArgumentException("Cannot merge stats from early stage " + i + " into stats of "
+ + "later stage " + _currentStageId);
+ }
+ }
+ growUpToStage(otherStats.size() - 1);
+
+ for (int i = _currentStageId + 1; i < otherStats.size(); i++) {
+ ByteBuffer otherBuf = otherStats.get(i);
+ if (otherBuf != null) {
+ StageStats.Closed myStats = getUpstreamStageStats(i);
+ try (InputStream is = new ByteBufferInputStream(Collections.singletonList(otherBuf));
+ DataInputStream dis = new DataInputStream(is)) {
+ if (myStats == null) {
+ StageStats.Closed deserialized = StageStats.Closed.deserialize(dis);
+ _closedStats.set(i - _currentStageId - 1, deserialized);
+ assert getUpstreamStageStats(i) == deserialized;
+ } else {
+ myStats.merge(dis);
+ }
+ } catch (IOException ex) {
+ boolean assertOn = false;
+ // *assigns* true if assertions are on.
+ //CHECKSTYLE:OFF
+ assert assertOn = true;
+ if (assertOn) {
+ throw new UncheckedIOException("Error deserializing stats on stage " + i, ex);
+ }
+ //CHECKSTYLE:ON
Review Comment:
This is the _standard_ way to detect whether asserts are enabled or not. Maybe we should create a static class that does the same thing, specially given checkstyle doesn't like this code style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1584406372
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -419,4 +550,122 @@ public void send(BaseResultsBlock block)
addResultsBlock(block);
}
}
+
+ public enum StatKey implements StatMap.Key {
+ TABLE(StatMap.Type.STRING),
+ EXECUTION_TIME_MS(StatMap.Type.LONG, null, DataTable.MetadataKey.TIME_USED_MS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ EMITTED_ROWS(StatMap.Type.LONG, null, DataTable.MetadataKey.NUM_ROWS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ NUM_DOCS_SCANNED(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(StatMap.Type.LONG),
+ NUM_SEGMENTS_QUERIED(StatMap.Type.INT),
+ NUM_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(StatMap.Type.INT),
+ // the timestamp indicating the freshness of the data queried in consuming segments.
+ // This can be ingestion timestamp if provided by the stream, or the last index time
+ MIN_CONSUMING_FRESHNESS_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ TOTAL_DOCS(StatMap.Type.LONG),
+ NUM_GROUPS_LIMIT_REACHED(StatMap.Type.BOOLEAN),
+ //TRACE_INFO(StatMap.Type.STRING),
+ //REQUEST_ID(StatMap.Type.LONG),
+ NUM_RESIZES(StatMap.Type.INT),
+ RESIZE_TIME_MS(StatMap.Type.LONG),
+ THREAD_CPU_TIME_NS(StatMap.Type.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(StatMap.Type.LONG),
+ RESPONSE_SER_CPU_TIME_NS(StatMap.Type.LONG, "responseSerializationCpuTimeNs"),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_BLOCKS(StatMap.Type.INT),
+ OPERATOR_EXECUTION_TIME_MS(StatMap.Type.LONG),
+ OPERATOR_ID(StatMap.Type.STRING),
+ OPERATOR_EXEC_START_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ OPERATOR_EXEC_END_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return Math.max(value1, value2);
+ }
+ },;
+ private final StatMap.Type _type;
+ @Nullable
+ private final DataTable.MetadataKey _v1Key;
+ private final String _statName;
+
+ StatKey(StatMap.Type type) {
+ this(type, null);
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = DataTable.MetadataKey.getByName(getStatName());
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName, @Nullable DataTable.MetadataKey v1Key) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = v1Key == null ? DataTable.MetadataKey.getByName(getStatName()) : v1Key;
+ }
+
+ @Override
+ public String getStatName() {
+ return _statName;
+ }
+
+ @Override
+ public StatMap.Type getType() {
+ return _type;
+ }
+
+ public void updateV1Metadata(StatMap<DataTable.MetadataKey> oldMetadata, StatMap<StatKey> stats) {
Review Comment:
> Thus I would model StatMap.Key as the stats key for v2 engine, which is logically separated from v1 engine stats.
I think that is a difference we have in our models. In my model `StatMap.Key` is not related to V2 engine. It is just an efficient map from enums to primitives that we designed to be used in V2, but could be used in other places.
> From implementation perspective, based on my code reading (correct me if I missed some important steps), for multi-stage engine, on the broker side, we only get the stats for leaf stage operator.
I think the correct formulation would be *for multi-stage engine, on the broker side, we **mostly** get the stats for leaf stage operator*. There are some metrics, like `maxRowsInOperator`, `maxRowsInJoinReached` or `numGroupsLimitReached` that are collected from other multi-stage operators different than leaf stage. For example `BrokerNative.numGroupsLimitReached` is true if any aggregation in the tree has reached the group limits. In V1 that is the only info we have. In V2 we can iterate over `statsMap` and get which specific group by is the one that reached the limit.
Anyway, yesterday I modified the code so MetadataKey does not implement `StatMap.Key`. Instead I created a new enum with broker stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583030085
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
> Is this the only place where v1 metadata block is used?
Yes. It is used only here and in the tests.
> Does it work if we ignore the version and always return new MetadataBlock(byteBuffer);?
Yes.
> If the error handling is different, then sending v2 metadata block to old server will cause backward incompatibility right?
It is no different.
Remember that the main reason to have this class is to be able to run tests with it. The fact that it is useful here is just an extra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583030085
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
> Is this the only place where v1 metadata block is used?
Yes. It is used only here and in the tests.
> Does it work if we ignore the version and always return new MetadataBlock(byteBuffer);?
~Yes. ~ No, see the next comemnt
> If the error handling is different, then sending v2 metadata block to old server will cause backward incompatibility right?
It is no different.
Remember that the main reason to have this class is to be able to run tests with it. The fact that it is useful here is just an extra.
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
> Is this the only place where v1 metadata block is used?
Yes. It is used only here and in the tests.
> Does it work if we ignore the version and always return new MetadataBlock(byteBuffer);?
~Yes.~ No, see the next comemnt
> If the error handling is different, then sending v2 metadata block to old server will cause backward incompatibility right?
It is no different.
Remember that the main reason to have this class is to be able to run tests with it. The fact that it is useful here is just an extra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1546150898
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +36,86 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
@VisibleForTesting
static final int VERSION = 1;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
-
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
- }
-
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
-
- public void setType(String type) {
- _type = type;
- }
-
- public Map<String, String> getStats() {
- return _stats;
- }
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
- }
- }
-
- private final Contents _contents;
+ private final MetadataBlockType _type;
public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
+ this(type, Collections.emptyList());
}
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> stats) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, serializeStats(stats));
+ _type = type;
}
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ private static byte[] serializeStats(List<ByteBuffer> stats) {
+ try (UnsynchronizedByteArrayOutputStream baos = new UnsynchronizedByteArrayOutputStream(1024);
Review Comment:
A future PR should change BaseDataBlock to do not ask for `byte[]` but `ByteBuffer` so we won't need to do an extra heap copy of the content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565303435
##########
pinot-common/src/main/java/org/apache/pinot/common/datatable/DataTable.java:
##########
@@ -100,7 +110,7 @@ enum MetadataValueType {
* - NEVER decrease MAX_ID
* Otherwise, backward compatibility will be broken.
*/
- enum MetadataKey {
+ enum MetadataKey implements StatMap.Key {
Review Comment:
These are not he stats reported by Leaf operator. This operator uses its own `LeafStageTransferableBlockOperator.StatKey` class. But it is useful to implement `StatMap.Key` in `MetadataKey` in order to be able to store them in a new `StatMap`. Do we use that in V1? No, but in V2 we use it in order to:
* In the leaf operator, but only as an input. Leaf operator has its own stat class, but it receives MetadataKey stats fromt he V1 engine in a `Map<String, String>` and has to transform them into its own stats. See `LeafStageTransferableBlockOperator.mergeExecutionStats`.
* In brokers, when the response is being reduced. I've changed that code to reduce it using StatMap instead of a `HashMap<String, String>`. Specifically, I've removed a lot of attributes from `BrokerResponseNative` and transform them into a `StatMap<DataTable.MetadataKey> _serverStats`. This makes it easier to merge V2 stats into this object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565308960
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlock.java:
##########
@@ -80,8 +80,6 @@ byte[] toBytes()
@Nullable
RoaringBitmap getNullRowIds(int colId);
- DataBlock toMetadataOnlyDataTable();
-
DataBlock toDataOnlyDataTable();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565313954
##########
pinot-broker/src/test/java/org/apache/pinot/broker/querylog/QueryLoggerTest.java:
##########
@@ -259,11 +259,9 @@ private QueryLogger.QueryLogParams generateParams(boolean isGroupLimitHit, int n
response.setExceptions(
IntStream.range(0, numExceptions)
.mapToObj(i -> new ProcessingException()).collect(Collectors.toList()));
- response.setOfflineTotalCpuTimeNs(14);
Review Comment:
Yes, I've removed these `setOflineTotalCpuTimeNs` and `setRealtimeTotalCpuTimeNs` from `BrokerResponseNative`. In production code they are always set as `threadCpuTimeNs + systemActivitiesCpuTimeNs + responseSerializationCpuTimeNs`, so it looks error prone to have a setter (like the one we used here). Instead we never set the value of the total and always derive it from its parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583818070
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -419,4 +550,122 @@ public void send(BaseResultsBlock block)
addResultsBlock(block);
}
}
+
+ public enum StatKey implements StatMap.Key {
+ TABLE(StatMap.Type.STRING),
+ EXECUTION_TIME_MS(StatMap.Type.LONG, null, DataTable.MetadataKey.TIME_USED_MS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ EMITTED_ROWS(StatMap.Type.LONG, null, DataTable.MetadataKey.NUM_ROWS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ NUM_DOCS_SCANNED(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(StatMap.Type.LONG),
+ NUM_SEGMENTS_QUERIED(StatMap.Type.INT),
+ NUM_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(StatMap.Type.INT),
+ // the timestamp indicating the freshness of the data queried in consuming segments.
+ // This can be ingestion timestamp if provided by the stream, or the last index time
+ MIN_CONSUMING_FRESHNESS_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ TOTAL_DOCS(StatMap.Type.LONG),
+ NUM_GROUPS_LIMIT_REACHED(StatMap.Type.BOOLEAN),
+ //TRACE_INFO(StatMap.Type.STRING),
+ //REQUEST_ID(StatMap.Type.LONG),
+ NUM_RESIZES(StatMap.Type.INT),
+ RESIZE_TIME_MS(StatMap.Type.LONG),
+ THREAD_CPU_TIME_NS(StatMap.Type.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(StatMap.Type.LONG),
+ RESPONSE_SER_CPU_TIME_NS(StatMap.Type.LONG, "responseSerializationCpuTimeNs"),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_BLOCKS(StatMap.Type.INT),
+ OPERATOR_EXECUTION_TIME_MS(StatMap.Type.LONG),
+ OPERATOR_ID(StatMap.Type.STRING),
+ OPERATOR_EXEC_START_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ OPERATOR_EXEC_END_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return Math.max(value1, value2);
+ }
+ },;
+ private final StatMap.Type _type;
+ @Nullable
+ private final DataTable.MetadataKey _v1Key;
+ private final String _statName;
+
+ StatKey(StatMap.Type type) {
+ this(type, null);
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = DataTable.MetadataKey.getByName(getStatName());
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName, @Nullable DataTable.MetadataKey v1Key) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = v1Key == null ? DataTable.MetadataKey.getByName(getStatName()) : v1Key;
+ }
+
+ @Override
+ public String getStatName() {
+ return _statName;
+ }
+
+ @Override
+ public StatMap.Type getType() {
+ return _type;
+ }
+
+ public void updateV1Metadata(StatMap<DataTable.MetadataKey> oldMetadata, StatMap<StatKey> stats) {
Review Comment:
Let me try to explain why I don't like having `MetadataKey` implementing `StatMap.Key`. To me `MetadataKey` is the stats key for `v1` engine, and some stats don't apply to `v2` engine. Thus I would model `StatMap.Key` as the stats key for `v2` engine, which is logically separated from `v1` engine stats. Making `MetadataKey` implementing `StatMap.Key` is essentially declaring `v1` engine stats part of `v2` stats, which is a coupling of `v1` and `v2`.
From implementation perspective, based on my code reading (correct me if I missed some important steps), for multi-stage engine, on the broker side, we only get the stats for leaf stage operator. Then we convert the v2 stats into v1 stats (`MultiStageOperator.Type.LEAF.mergeInto()`), then finally call `BrokerResponseNativeV2.addServerStats()` to add v1 stats into v2 response. I don't see other usage of `MetadataKey` within v2 engine flow, and I don't really see the benefits of this back and forth conversion. If all the v1 stats are from v2 leaf stage stats, we should be able to directly use leaf stage stats to serve the stats within v2 response.
Let's bring this offline to ensure we are on the same page and I'm not missing important point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12704:
URL: https://github.com/apache/pinot/pull/12704#issuecomment-2082719975
> Is it possible to break this PR into smaller ones? Currently it covers too many topics and is hard to track all the changes
I can try, but I think most of the topics are related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582735688
##########
pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java:
##########
@@ -123,6 +127,17 @@ String toJsonString()
*/
long getMinConsumingFreshnessTimeMs();
+ /**
+ * Get the max number of rows seen by a single operator in the query processing chain.
+ * <p>
+ * In single-stage this value is equal to {@link #getNumEntriesScannedPostFilter()} because single-stage operators
+ * cannot generate more rows than the number of rows received. This is not the case in multi-stage operators, where
+ * operators like join or union can emit more rows than the ones received on each of its children operators.
+ */
+ default long getMaxRowsInOperator() {
Review Comment:
This is not a new method but a method that existed in `BrokerResponseNative` and was inherited by `BrokerResponseV2` . Given it is already there (even it doesn't make sense) and it was used to calculate `isPartialResponse` I think it is useful to have it in the base interface now that `BrokerResponseNativeV2` does not inherit `BrokerResponseNative` anymore.
The alternative would be to do not make `isPartialResponse` `default` and include the same code in both `BrokerResponseNative` and `BrokerResponseNativeV2`. If you think that is better I can change it, but it looks a bit repetitive. For sure currently how this stats are being declared doesn't make sense, but we can try to improve that in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565315967
##########
pinot-broker/src/test/java/org/apache/pinot/broker/querylog/QueryLoggerTest.java:
##########
@@ -259,11 +259,9 @@ private QueryLogger.QueryLogParams generateParams(boolean isGroupLimitHit, int n
response.setExceptions(
IntStream.range(0, numExceptions)
.mapToObj(i -> new ProcessingException()).collect(Collectors.toList()));
- response.setOfflineTotalCpuTimeNs(14);
Review Comment:
BTW, this is why in lines 110 an 111 I had to modify the result. Before we were returning the synthetic values 14 and 18 and now we return 48 and 60 which is the sum of 15, 16 and 17 and 19, 20 and 21 respectively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565385608
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotJoinToDynamicBroadcastRule.java:
##########
@@ -164,11 +165,15 @@ public void onMatch(RelOptRuleCall call) {
// 2. when hash key are the same but hash functions are different, it can be done via normal hash shuffle.
boolean isColocatedJoin = PinotHintStrategyTable.isHintOptionTrue(join.getHints(),
PinotHintOptions.JOIN_HINT_OPTIONS, PinotHintOptions.JoinHintOptions.IS_COLOCATED_BY_JOIN_KEYS);
- PinotLogicalExchange dynamicBroadcastExchange = isColocatedJoin
- ? PinotLogicalExchange.create(right.getInput(), RelDistributions.hash(join.analyzeCondition().rightKeys),
- PinotRelExchangeType.PIPELINE_BREAKER)
- : PinotLogicalExchange.create(right.getInput(), RelDistributions.BROADCAST_DISTRIBUTED,
- PinotRelExchangeType.PIPELINE_BREAKER);
+ PinotLogicalExchange dynamicBroadcastExchange;
+ RelNode rightInput = right.getInput();
+ if (isColocatedJoin) {
+ RelDistribution dist = RelDistributions.hash(join.analyzeCondition().rightKeys);
+ dynamicBroadcastExchange = PinotLogicalExchange.create(rightInput, dist, PinotRelExchangeType.PIPELINE_BREAKER);
+ } else {
+ RelDistribution dist = RelDistributions.BROADCAST_DISTRIBUTED;
+ dynamicBroadcastExchange = PinotLogicalExchange.create(rightInput, dist, PinotRelExchangeType.PIPELINE_BREAKER);
+ }
Review Comment:
This is the same code as before, but easier to read and to debug. The former is subjective, so I'm open to rollback this change if required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576205973
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/V1MetadataBlock.java:
##########
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.datablock;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.Map;
+
+
+/**
+ * The datablock used before the introduction of {@link org.apache.pinot.common.datatable.StatMap}.
+ * <p>
+ * This version stored the metadata in a {@code Map<String, String>} which was encoded as JSON and stored in the
+ * variable size data buffer.
+ * <p>
+ * Instances of this class are not actually seen by the operators. Instead, they are converted to {@link MetadataBlock}
+ * in {@link MetadataBlock#deserialize(ByteBuffer, int)}.
+ * <p>
+ * The reason to keep it here is mostly for backwards compatibility and testing. In order to simplify the code, the
+ * stats engine just ignores the metadata of these objects, but we need to be able to deserialize them anyway.
+ */
+public class V1MetadataBlock extends BaseDataBlock {
Review Comment:
This is not only for testing proposes. We use this class to read metadata blocks send by servers running previous versions. By doing that we can read the actual metadata and do whatever we want with that.
This metadata may be an error or an EOS with stats in the older format. The current implementation only cares about the error case. It is important to keep using that information (whose format wasn't changed in this PR) in order to correctly abort the execution if needed.
Right now we ignore the stats because it doesn't seem worth it to translate the stats given the idea is to run with different versions only while the new release is being deployed, so losing some stats during that time seems acceptable. We can change that if we think it is important, but I would suggest to do that in another PR if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576190502
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
+ brokerResponse.addStageStats(JsonUtils.newObjectNode());
+ } else {
+ stageStats.forEach((type, stats) -> type.mergeInto(brokerResponse, stats));
+
+ DispatchablePlanFragment dispatchablePlanFragment = dispatchableSubPlan.getQueryStageList().get(i);
+ MultiStageStatsTreeBuilder treeBuilder = new MultiStageStatsTreeBuilder(stageStats);
+ PlanNode fragmentRoot = dispatchablePlanFragment.getPlanFragment().getFragmentRoot();
+ JsonNode node = fragmentRoot.visit(treeBuilder, null);
+ brokerResponse.addStageStats(node);
+ }
+ }
+ }
+
+ public static class MultiStageStatsTreeBuilder implements PlanNodeVisitor<JsonNode, Void> {
+ private final MultiStageQueryStats.StageStats.Closed _stageStats;
+ private int _index;
+ private static final String CHILDREN_KEY = "children";
+
+ public MultiStageStatsTreeBuilder(MultiStageQueryStats.StageStats.Closed stageStats) {
+ _stageStats = stageStats;
+ _index = stageStats.getLastOperatorIndex();
+ }
+
+ private ObjectNode selfNode(MultiStageOperator.Type type) {
+ ObjectNode json = JsonUtils.newObjectNode();
+ json.put("type", type.toString());
+ Iterator<Map.Entry<String, JsonNode>> statsIt = _stageStats.getOperatorStats(_index).asJson().fields();
+ while (statsIt.hasNext()) {
+ Map.Entry<String, JsonNode> entry = statsIt.next();
+ json.set(entry.getKey(), entry.getValue());
+ }
+ return json;
+ }
+
+ private JsonNode recursiveCase(AbstractPlanNode node, MultiStageOperator.Type expectedType) {
+ MultiStageOperator.Type type = _stageStats.getOperatorType(_index);
+ if (type != expectedType) {
Review Comment:
~No, this is very rare. Mostly only happens on the stage 0 and in leaf stages.~
~In stage 0 always have two nodes. One is a transform IIRC and the other is a receive node and we only have stats for the latter.~
~In leaf stage, we may have several nodes, but the leaf stage compiles them into a different plan (as explained above).~
~In can add some comments~
I've made some tests and I think it should never be null. But I guess it is better to be safe to do not throw in that case. I'm going to add some comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1578099170
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java:
##########
@@ -51,6 +59,7 @@
"segmentStatistics", "traceInfo", "partialResult"
})
public class BrokerResponseNative implements BrokerResponse {
Review Comment:
I've reverted most of the changes and modified BrokerResponseNativeV2 to implement BrokerResponse instead of inherited this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1589060972
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
If we remove this case, then `v1ErrorWithExceptionsIsDecodedAsV2ErrorWithSameExceptions` fails because we decode as EOS instead of error.
The reason for that is because when I modified `MetadataBlock` I decided to keep the `_type` attribute. But I guess it doesn't make much sense if we define that an error block must always contain at least one exception (like in _V0_). I've just changed that and now all tests are passing and V1MetadataBlock can be moved to the test directory.
Anyway, one of my plans is to clean this code a lot in order to improve serialization/deserialization times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1586820017
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
Let's list down the behavior on reading v1 bytes as v2 and also reading v2 bytes as v1 so that user knows what to expect when upgrading.
With current way of keeping both versions, we don't need to read v1 bytes as v2, but we can still read v2 bytes as v1 block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1566907287
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/MultiStageQueryStats.java:
##########
@@ -0,0 +1,668 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.plan;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.avro.util.ByteBufferInputStream;
+import org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.query.runtime.operator.BaseMailboxReceiveOperator;
+import org.apache.pinot.query.runtime.operator.LeafStageTransferableBlockOperator;
+import org.apache.pinot.query.runtime.operator.LiteralValueOperator;
+import org.apache.pinot.query.runtime.operator.MailboxSendOperator;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The stats of a given query.
+ * <p>
+ * For the same query, multiple instances of this class may exist. Each of them will have a partial view of the stats.
+ * Specifically, while the query is being executed, each operator will return its own partial view of the stats when
+ * EOS block is sent.
+ * <p>
+ * Simple operations with a single upstream, like filters or transforms, would just add their own information to the
+ * stats. More complex operations, like joins or receiving mailboxes, will merge the stats from all their upstreams and
+ * add their own stats.
+ * <p>
+ * The complete stats for the query are obtained in the execution root (usually the broker) by merging the partial
+ * views.
+ * <p>
+ * In order to reduce allocation, this class is mutable. Some operators may create their own stats, but most of them
+ * will receive a stats object from the upstream operator and modify it by adding their own stats and sometimes merging
+ * them with other upstream stats.
+ */
+public class MultiStageQueryStats {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MultiStageQueryStats.class);
+ private final int _currentStageId;
+ private final StageStats.Open _currentStats;
+ /**
+ * Known stats for stages whose id is higher than the current one.
+ * <p>
+ * A stage may not know all the stats whose id is higher than the current one, so this list may contain null values.
+ * It may also grow in size when different merge methods are called.
+ * <p>
+ * For example the stats of the left hand side of a join may know stats of stages 3 and 4 and the right side may know
+ * stats of stages 5. When merging the stats of the join, the stats of stages 5 will be added to this list.
+ *
+ * @see #mergeUpstream(List)
+ * @see #mergeUpstream(MultiStageQueryStats)
+ * @see #mergeInOrder(MultiStageQueryStats, MultiStageOperator.Type, StatMap)
+ */
+ private final ArrayList<StageStats.Closed> _closedStats;
+ private static final MultiStageOperator.Type[] ALL_TYPES = MultiStageOperator.Type.values();
+
+ private MultiStageQueryStats(int stageId) {
+ _currentStageId = stageId;
+ _currentStats = new StageStats.Open();
+ _closedStats = new ArrayList<>();
+ }
+
+ private static MultiStageQueryStats create(int stageId, MultiStageOperator.Type type, @Nullable StatMap<?> opStats) {
+ MultiStageQueryStats multiStageQueryStats = new MultiStageQueryStats(stageId);
+ multiStageQueryStats.getCurrentStats().addLastOperator(type, opStats);
+ return multiStageQueryStats;
+ }
+
+ public static MultiStageQueryStats emptyStats(int stageId) {
+ return new MultiStageQueryStats(stageId);
+ }
+
+ public static MultiStageQueryStats createLeaf(int stageId,
+ StatMap<LeafStageTransferableBlockOperator.StatKey> opStats) {
+ return create(stageId, MultiStageOperator.Type.LEAF, opStats);
+ }
+
+ public static MultiStageQueryStats createLiteral(int stageId, StatMap<LiteralValueOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.LITERAL, statMap);
+ }
+
+ public static MultiStageQueryStats createCancelledSend(int stageId,
+ StatMap<MailboxSendOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_SEND, statMap);
+ }
+
+ public static MultiStageQueryStats createReceive(int stageId, StatMap<BaseMailboxReceiveOperator.StatKey> stats) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_RECEIVE, stats);
+ }
+
+ public int getCurrentStageId() {
+ return _currentStageId;
+ }
+
+ /**
+ * Serialize the current stats in a way it is compatible with {@link #mergeUpstream(List)}.
+ * <p>
+ * The serialized stats are returned in a list where the index is the stage id. Stages downstream or not related to
+ * the current one will be null.
+ */
+ public List<ByteBuffer> serialize()
+ throws IOException {
+
+ ArrayList<ByteBuffer> serializedStats = new ArrayList<>(getMaxStageId());
+ for (int i = 0; i < _currentStageId; i++) {
+ serializedStats.add(null);
+ }
+
+ try (UnsynchronizedByteArrayOutputStream baos = new UnsynchronizedByteArrayOutputStream.Builder().get();
+ DataOutputStream output = new DataOutputStream(baos)) {
+
+ _currentStats.serialize(output);
+ ByteBuffer currentBuf = ByteBuffer.wrap(baos.toByteArray());
+
+ serializedStats.add(currentBuf);
+
+ for (StageStats.Closed closedStats : _closedStats) {
+ if (closedStats == null) {
+ serializedStats.add(null);
+ continue;
+ }
+ baos.reset();
+ closedStats.serialize(output);
+ ByteBuffer buf = ByteBuffer.wrap(baos.toByteArray());
+ serializedStats.add(buf);
+ }
+ }
+ Preconditions.checkState(serializedStats.size() == getMaxStageId() + 1,
+ "Serialized stats size is different from expected size. Expected %s, got %s",
+ getMaxStageId() + 1, serializedStats.size());
+ return serializedStats;
+ }
+
+ public StageStats.Open getCurrentStats() {
+ return _currentStats;
+ }
+
+ /**
+ * Returns the higher stage id known by this object.
+ */
+ public int getMaxStageId() {
+ return _currentStageId + _closedStats.size();
+ }
+
+ /**
+ * Get the stats of a stage whose id is higher than the current one.
+ * <p>
+ * This method returns null in case the stage id is unknown by this stage or no stats are stored for it.
+ */
+ @Nullable
+ public StageStats.Closed getUpstreamStageStats(int stageId) {
+ if (stageId <= _currentStageId) {
+ throw new IllegalArgumentException("Stage " + stageId + " cannot be upstream of current stage "
+ + _currentStageId);
+ }
+
+ int index = stageId - _currentStageId - 1;
+ if (index >= _closedStats.size()) {
+ return null;
+ }
+ return _closedStats.get(index);
+ }
+
+ public void mergeInOrder(MultiStageQueryStats otherStats, MultiStageOperator.Type type,
+ StatMap<?> statMap) {
+ Preconditions.checkArgument(_currentStageId == otherStats._currentStageId,
+ "Cannot merge stats from different stages (%s and %s)", _currentStageId, otherStats._currentStageId);
+ mergeUpstream(otherStats);
+ StageStats.Open currentStats = getCurrentStats();
+ currentStats.concat(otherStats.getCurrentStats());
+ currentStats.addLastOperator(type, statMap);
+ }
+
+ private void growUpToStage(int stageId) {
+ _closedStats.ensureCapacity(stageId - _currentStageId);
+ while (getMaxStageId() < stageId) {
+ _closedStats.add(null);
+ }
+ }
+
+ /**
+ * Merge upstream stats from another MultiStageQueryStats object into this one.
+ * <p>
+ * Only the stages whose id is higher than the current one are merged. The reason to do so is that upstream stats
+ * should be already closed while current stage may need some extra tuning.
+ * <p>
+ * For example set operations may need to merge the stats from all its upstreams before concatenating stats of the
+ * current stage.
+ */
+ public void mergeUpstream(MultiStageQueryStats otherStats) {
+ Preconditions.checkArgument(_currentStageId <= otherStats._currentStageId,
+ "Cannot merge stats from early stage %s into stats of later stage %s",
+ otherStats._currentStageId, _currentStageId);
+
+ growUpToStage(otherStats.getMaxStageId());
+
+ int currentDiff = otherStats._currentStageId - _currentStageId;
+ if (currentDiff > 0) {
+ StageStats.Closed close = otherStats._currentStats.close();
+ int selfIdx = currentDiff - 1;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ if (myStats == null) {
+ _closedStats.set(selfIdx, close);
+ } else {
+ myStats.merge(close);
+ }
+ }
+
+ for (int i = 0; i < otherStats._closedStats.size(); i++) {
+ StageStats.Closed otherStatsForStage = otherStats._closedStats.get(i);
+ if (otherStatsForStage == null) {
+ continue;
+ }
+ int selfIdx = i + currentDiff;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ try {
+ if (myStats == null) {
+ _closedStats.set(selfIdx, otherStatsForStage);
+ assert getUpstreamStageStats(i + otherStats._currentStageId + 1) == otherStatsForStage;
+ } else {
+ myStats.merge(otherStatsForStage);
+ }
+ } catch (IllegalArgumentException | IllegalStateException ex) {
+ LOGGER.warn("Error merging stats on stage " + i + ". Ignoring the new stats", ex);
+ }
+ }
+ }
+
+ public void mergeUpstream(List<ByteBuffer> otherStats) {
+ for (int i = 0; i <= _currentStageId && i < otherStats.size(); i++) {
+ if (otherStats.get(i) != null) {
+ throw new IllegalArgumentException("Cannot merge stats from early stage " + i + " into stats of "
+ + "later stage " + _currentStageId);
+ }
+ }
+ growUpToStage(otherStats.size() - 1);
+
+ for (int i = _currentStageId + 1; i < otherStats.size(); i++) {
+ ByteBuffer otherBuf = otherStats.get(i);
+ if (otherBuf != null) {
+ StageStats.Closed myStats = getUpstreamStageStats(i);
+ try (InputStream is = new ByteBufferInputStream(Collections.singletonList(otherBuf));
+ DataInputStream dis = new DataInputStream(is)) {
+ if (myStats == null) {
+ StageStats.Closed deserialized = StageStats.Closed.deserialize(dis);
+ _closedStats.set(i - _currentStageId - 1, deserialized);
+ assert getUpstreamStageStats(i) == deserialized;
+ } else {
+ myStats.merge(dis);
+ }
+ } catch (IOException ex) {
+ boolean assertOn = false;
+ // *assigns* true if assertions are on.
+ //CHECKSTYLE:OFF
+ assert assertOn = true;
+ if (assertOn) {
+ throw new UncheckedIOException("Error deserializing stats on stage " + i, ex);
+ }
+ //CHECKSTYLE:ON
Review Comment:
I think it is useful. We don't want to kill the query in production when this happens but we want to catch the problem during testing. But we can remove it if you think it is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1566905246
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
Review Comment:
I think we are not going to add support for 2nd version in the code we have in master, but the code in this PR actually reads 1st and 2nd version. We don't need to do the former because we are lucky and the binary format should be compatible (though we may need to verify that adding a test in master).
The implementation in both branches mostly ignore metrics in the other, but even they ignore the content, they are able to read them. We are just ignoring the content because we think it is not worth to actually convert from one format to the other given the impact will be to do not have all stats during the upgrade, but we could do that if needed.
I think it is fine to increase the version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1566570540
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
Review Comment:
We use version to maintain backward compatible. When introducing a new version, we first add the support for both versions, then after all nodes are upgraded, use a flag to switch the versions. After it is stable, remove the flag and old version support.
Since we are not following the above process, version is not really checked when deserializing. I don't have strong opinion on whether to bump it. Just pointing it out that it is not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565379309
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNativeV2.java:
##########
@@ -61,36 +67,31 @@ public BrokerResponseNativeV2(List<ProcessingException> exceptions) {
super(exceptions);
}
- /** Generate EXPLAIN PLAN output when queries are evaluated by Broker without going to the Server. */
- private static BrokerResponseNativeV2 getBrokerResponseExplainPlanOutput() {
- BrokerResponseNativeV2 brokerResponse = BrokerResponseNativeV2.empty();
- List<Object[]> rows = new ArrayList<>();
- rows.add(new Object[]{"BROKER_EVALUATE", 0, -1});
- brokerResponse.setResultTable(new ResultTable(DataSchema.EXPLAIN_RESULT_SCHEMA, rows));
- return brokerResponse;
Review Comment:
This method was not used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1548253609
##########
pinot-common/src/main/java/org/apache/pinot/common/datatable/StatMap.java:
##########
@@ -0,0 +1,759 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.datatable;
+
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.utils.JsonUtils;
+
+
+/**
+ * A map that stores statistics.
+ *
+ * Statistics must be keyed by an enum that implements {@link StatMap.Key}.
+ *
+ * A stat map efficiently store, serialize and deserialize these statistics.
+ *
+ * Serialization and deserialization is backward and forward compatible as long as the only change in the keys are:
+ * <ul>
+ * <li>Adding new keys</li>
+ * <li>Change the name of the keys</li>
+ * </ul>
+ *
+ * Any other change (like changing the type of key, changing their literal order are not supported or removing keys)
+ * are backward incompatible changes.
+ * @param <K>
+ */
+public class StatMap<K extends Enum<K> & StatMap.Key> {
+ private final Family<K> _family;
+ @Nullable
+ private final int[] _intValues;
+ @Nullable
+ private final long[] _longValues;
+ @Nullable
+ private final boolean[] _booleanValues;
+ /**
+ * In Pinot 1.1.0 this can only store String values, but it is prepared to be able to store any kind of object in
+ * the future.
+ */
+ @Nullable
+ private final Object[] _referenceValues;
+
+ public StatMap(Class<K> keyClass) {
+ Family<K> family = new Family<>(keyClass);
+ _family = family;
+ _intValues = family.createIntValues();
+ _longValues = family.createLongValues();
+ _booleanValues = family.createBooleanValues();
+ _referenceValues = family.createReferenceValues();
+ }
+
+ private StatMap(Family<K> family, @Nullable int[] intValues, @Nullable long[] longValues,
+ @Nullable boolean[] booleanValues, @Nullable Object[] referenceValues) {
+ _family = family;
+ assert intValues == null || intValues.length == family._numIntsValues;
+ _intValues = intValues;
+ assert longValues == null || longValues.length == family._numLongValues;
+ _longValues = longValues;
+ assert booleanValues == null || booleanValues.length == family._numBooleanValues;
+ _booleanValues = booleanValues;
+ assert referenceValues == null || referenceValues.length == family._numReferenceValues;
+ _referenceValues = referenceValues;
+ }
+
+ public int getInt(K key) {
+ Preconditions.checkArgument(key.getType() == Type.INT);
+ int index = _family.getIndex(key);
+ assert _intValues != null : "Int values should not be null because " + key + " is of type INT";
+ return _intValues[index];
+ }
+
+ public void merge(K key, int value) {
+ if (key.getType() == Type.LONG) {
+ merge(key, (long) value);
+ return;
+ }
+ Preconditions.checkArgument(key.getType() == Type.INT);
+ int index = _family.getIndex(key);
+ assert _intValues != null : "Int values should not be null because " + key + " is of type INT";
+ _intValues[index] = key.merge(_intValues[index], value);
+ }
+
+ public long getLong(K key) {
+ if (key.getType() == Type.INT) {
+ return getInt(key);
+ }
+ Preconditions.checkArgument(key.getType() == Type.LONG);
+ int index = _family.getIndex(key);
+ assert _longValues != null : "Long values should not be null because " + key + " is of type LONG";
+ return _longValues[index];
+ }
+
+ public void merge(K key, long value) {
+ Preconditions.checkArgument(key.getType() == Type.LONG);
+ int index = _family.getIndex(key);
+ assert _longValues != null : "Long values should not be null because " + key + " is of type LONG";
+ _longValues[index] = key.merge(_longValues[index], value);
+ }
+
+ public boolean getBoolean(K key) {
+ Preconditions.checkArgument(key.getType() == Type.BOOLEAN);
+ int index = _family.getIndex(key);
+ assert _booleanValues != null : "Boolean values should not be null because " + key + " is of type BOOLEAN";
+ return _booleanValues[index];
+ }
+
+ public void merge(K key, boolean value) {
+ Preconditions.checkArgument(key.getType() == Type.BOOLEAN);
+ int index = _family.getIndex(key);
+ assert _booleanValues != null : "Boolean values should not be null because " + key + " is of type BOOLEAN";
+ _booleanValues[index] = key.merge(_booleanValues[index], value);
+ }
+
+ public String getString(K key) {
+ Preconditions.checkArgument(key.getType() == Type.STRING);
+ int index = _family.getIndex(key);
+ assert _referenceValues != null : "Reference values should not be null because " + key + " is of type STRING";
+ return (String) _referenceValues[index];
+ }
+
+ public void merge(K key, String value) {
+ Preconditions.checkArgument(key.getType() == Type.STRING);
+ int index = _family.getIndex(key);
+ assert _referenceValues != null : "Reference values should not be null because " + key + " is of type STRING";
+ _referenceValues[index] = key.merge((String) _referenceValues[index], value);
+ }
+
+ /**
+ * Returns the value associated with the key.
+ *
+ * Primitives will be boxed, so it is recommended to use the specific methods for each type.
+ */
+ public Object getAny(K key) {
+ switch (key.getType()) {
+ case BOOLEAN:
+ return getBoolean(key);
+ case INT:
+ return getInt(key);
+ case LONG:
+ return getLong(key);
+ case STRING:
+ return getString(key);
+ default:
+ throw new IllegalArgumentException("Unsupported type: " + key.getType());
+ }
+ }
+
+ /**
+ * Modifies this object to merge the values of the other object.
+ *
+ * Numbers will be added, booleans will be ORed, and strings will be set if they are null.
+ *
+ * @param other The object to merge with. This argument will not be modified.
+ */
+ public void merge(StatMap<K> other) {
+ Preconditions.checkState(_family._keyClass.equals(other._family._keyClass),
+ "Different key classes %s and %s", _family._keyClass, other._family._keyClass);
+ Preconditions.checkState(_family._numIntsValues == other._family._numIntsValues,
+ "Different number of int values");
+ for (int i = 0; i < _family._numIntsValues; i++) {
+ assert _intValues != null : "Int values should not be null because there are int keys";
+ assert other._intValues != null : "Int values should not be null because there are int keys";
+ K key = _family.getKey(i, Type.INT);
+ _intValues[i] = key.merge(_intValues[i], other._intValues[i]);
+ }
+
+ Preconditions.checkState(_family._numLongValues == other._family._numLongValues,
+ "Different number of long values");
+ for (int i = 0; i < _family._numLongValues; i++) {
+ assert _longValues != null : "Long values should not be null because there are long keys";
+ assert other._longValues != null : "Long values should not be null because there are long keys";
+ K key = _family.getKey(i, Type.LONG);
+ _longValues[i] = key.merge(_longValues[i], other._longValues[i]);
+ }
+
+ Preconditions.checkState(_family._numBooleanValues == other._family._numBooleanValues,
+ "Different number of boolean values");
+ for (int i = 0; i < _family._numBooleanValues; i++) {
+ assert _booleanValues != null : "Boolean values should not be null because there are boolean keys";
+ assert other._booleanValues != null : "Boolean values should not be null because there are boolean keys";
+ K key = _family.getKey(i, Type.BOOLEAN);
+ _booleanValues[i] = key.merge(_booleanValues[i], other._booleanValues[i]);
+ }
+
+ Preconditions.checkState(_family._numReferenceValues == other._family._numReferenceValues,
+ "Different number of reference values");
+ for (int i = 0; i < _family._numReferenceValues; i++) {
+ assert _referenceValues != null : "Reference values should not be null because there are reference keys";
+ assert other._referenceValues != null : "Reference values should not be null because there are reference keys";
+ if (_referenceValues[i] == null) {
+ _referenceValues[i] = other._referenceValues[i];
+ } else {
+ K key = _family.getKey(i, Type.STRING);
+ _referenceValues[i] = key.merge((String) _referenceValues[i], (String) other._referenceValues[i]);
+ }
+ }
+ }
+
+ public void merge(DataInput input)
+ throws IOException {
+ int bitsPerId = 32 - Integer.numberOfLeadingZeros(Math.abs(_family._maxIndex));
+ int maxBytesPerId = (bitsPerId + 7) / 8;
+
+ int bytesPerId = input.readInt();
+ Preconditions.checkArgument(bytesPerId <= maxBytesPerId, "Invalid bytes per id: %s. Max expected = %s",
+ bytesPerId, maxBytesPerId);
+
+ int[] keys;
+ keys = readKeys(input, bytesPerId, _family._numIntsValues);
+ if (_family._numIntsValues != 0) {
+ assert _intValues != null : "Int values should not be null because there are int keys";
+ for (int intKey : keys) {
+ int value = input.readInt();
+ assert value != 0;
+ K key = _family.getKey(intKey, Type.INT);
+ _intValues[intKey] = key.merge(_intValues[intKey], value);
+ }
+ }
+
+ keys = readKeys(input, bytesPerId, _family._numLongValues);
+ if (_family._numLongValues != 0) {
+ assert _longValues != null : "Long values should not be null because there are long keys";
+ for (int longKey : keys) {
+ long value = input.readLong();
+ assert value != 0;
+ K key = _family.getKey(longKey, Type.LONG);
+ _longValues[longKey] = key.merge(_longValues[longKey], value);
+ }
+ }
+
+ keys = readKeys(input, bytesPerId, _family._numBooleanValues);
+ if (_family._numBooleanValues != 0) {
+ assert _booleanValues != null : "Boolean values should not be null because there are boolean keys";
+ for (int booleanKey : keys) {
+ boolean value = input.readBoolean();
+ assert value;
+ K key = _family.getKey(booleanKey, Type.BOOLEAN);
+ _booleanValues[booleanKey] = key.merge(_booleanValues[booleanKey], true);
+ }
+ }
+
+ keys = readKeys(input, bytesPerId, _family._numReferenceValues);
+ if (_family._numReferenceValues != 0) {
+ assert _referenceValues != null : "Reference values should not be null because there are reference keys";
+ for (int refKeys : keys) {
+ // In case we add more reference keys, this should be changed
+ String value = input.readUTF();
+ K key = _family.getKey(refKeys, Type.STRING);
+ _referenceValues[refKeys] = key.merge((String) _referenceValues[refKeys], value);
+ }
+ }
+ }
+
+ public ObjectNode asJson() {
+ ObjectNode node = JsonUtils.newObjectNode();
+
+ if (_intValues != null) {
+ for (K key : _family.getKeysOfType(Type.INT)) {
+ int index = _family.getIndex(key);
+ int value = _intValues[index];
+ if (value != 0 || key.includeDefaultInJson()) {
+ node.put(key.getStatName(), value);
+ }
+ }
+ }
+
+ if (_longValues != null) {
+ for (K key : _family.getKeysOfType(Type.LONG)) {
+ int index = _family.getIndex(key);
+ long value = _longValues[index];
+ if (value != 0 || key.includeDefaultInJson()) {
+ node.put(key.getStatName(), value);
+ }
+ }
+ }
+
+ if (_booleanValues != null) {
+ for (K key : _family.getKeysOfType(Type.BOOLEAN)) {
+ int index = _family.getIndex(key);
+ boolean value = _booleanValues[index];
+ if (value || key.includeDefaultInJson()) {
+ node.put(key.getStatName(), true);
+ }
+ }
+ }
+
+ if (_referenceValues != null) {
+ for (K key : _family.getKeysOfType(Type.STRING)) {
+ int index = _family.getIndex(key);
+ String value = (String) _referenceValues[index];
+ if (value != null || key.includeDefaultInJson()) {
+ node.put(key.getStatName(), value);
+ }
+ }
+ }
+
+ return node;
+ }
+
+ public void serialize(DataOutput output)
+ throws IOException {
+ int maxIndex = _family._maxIndex;
+ int bitsPerId = 32 - Integer.numberOfLeadingZeros(Math.abs(maxIndex));
+ int bytesPerId = (bitsPerId + 7) / 8;
+
+ output.writeInt(bytesPerId);
+
+ writeInts(output, bytesPerId);
+ writeLongs(output, bytesPerId);
+ writeBooleans(output, bytesPerId);
+ writeStrings(output, bytesPerId);
+ }
+
+ private void writeInts(DataOutput output, int bytesPerId)
+ throws IOException {
+ if (_intValues == null) {
+ writeInt(output, bytesPerId, 0);
+ } else {
+ writeValuedKeys(output, Type.INT, bytesPerId, _intValues.length, i -> _intValues[i] != 0);
+ for (int intValue : _intValues) {
+ if (intValue != 0) {
+ output.writeInt(intValue);
+ }
+ }
+ }
+ }
+
+ private void writeLongs(DataOutput output, int bytesPerId)
+ throws IOException {
+ if (_longValues == null) {
+ writeInt(output, bytesPerId, 0);
+ } else {
+ writeValuedKeys(output, Type.LONG, bytesPerId, _longValues.length, i -> _longValues[i] != 0);
+ for (long longValue : _longValues) {
+ if (longValue != 0) {
+ output.writeLong(longValue);
+ }
+ }
+ }
+ }
+
+ private void writeBooleans(DataOutput output, int bytesPerId)
+ throws IOException {
+ if (_booleanValues == null) {
+ writeInt(output, bytesPerId, 0);
+ } else {
+ writeValuedKeys(output, Type.BOOLEAN, bytesPerId, _booleanValues.length, i -> _booleanValues[i]);
+ for (boolean booleanValue : _booleanValues) {
+ if (booleanValue) {
+ output.writeBoolean(true);
+ }
+ }
+ }
+ }
+
+ private void writeStrings(DataOutput output, int bytesPerId)
+ throws IOException {
+ if (_referenceValues == null) {
+ writeInt(output, bytesPerId, 0);
+ } else {
+ writeValuedKeys(output, Type.STRING, bytesPerId, _referenceValues.length, i -> _referenceValues[i] != null);
+ for (Object referenceValue : _referenceValues) {
+ if (referenceValue != null) {
+ output.writeUTF((String) referenceValue);
+ }
+ }
+ }
+ }
+
+ private void writeValuedKeys(DataOutput output, Type type, int bytesPerId, int length,
+ IntPredicate isValidIndex)
+ throws IOException {
+ List<K> allKeys = _family.getKeysOfType(type);
+ List<K> keys = new ArrayList<>(allKeys.size());
+ for (int i = 0; i < length; i++) {
+ if (isValidIndex.test(i)) {
+ keys.add(allKeys.get(i));
+ }
+ }
+ writeInt(output, bytesPerId, keys.size());
+ for (K key : keys) {
+ writeInt(output, bytesPerId, _family.getIndex(key));
+ }
+ }
+
+ public static <K extends Enum<K> & Key> StatMap<K> deserialize(DataInput input, Class<K> keyClass)
+ throws IOException {
+ Family<K> family = new Family<>(keyClass);
+
+ int bytesPerId = input.readInt();
+
+ int[] intKeys = readKeys(input, bytesPerId, family._numIntsValues);
+ int[] intValues;
+ if (family._numIntsValues == 0) {
+ intValues = null;
+ } else {
+ intValues = new int[family._numIntsValues];
+ for (int intKey : intKeys) {
+ int value = input.readInt();
+ assert value != 0;
+ intValues[intKey] = value;
+ }
+ }
+
+ int[] longKeys = readKeys(input, bytesPerId, family._numLongValues);
+ long[] longValues;
+ if (family._numLongValues == 0) {
+ longValues = null;
+ } else {
+ longValues = new long[family._numLongValues];
+ for (int longKey : longKeys) {
+ long value = input.readLong();
+ assert value != 0;
+ longValues[longKey] = value;
+ }
+ }
+
+ int[] booleanKeys = readKeys(input, bytesPerId, family._numBooleanValues);
+ boolean[] booleanValues;
+ if (family._numBooleanValues == 0) {
+ booleanValues = null;
+ } else {
+ booleanValues = new boolean[family._numBooleanValues];
+ for (int booleanKey : booleanKeys) {
+ boolean value = input.readBoolean();
+ assert value;
+ booleanValues[booleanKey] = true;
+ }
+ }
+
+ int[] stringKeys = readKeys(input, bytesPerId, family._numReferenceValues);
+ Object[] referenceValues;
+ if (family._numReferenceValues == 0) {
+ referenceValues = null;
+ } else {
+ referenceValues = new Object[family._numReferenceValues];
+ for (int stringKey : stringKeys) {
+ String value = input.readUTF();
+ referenceValues[stringKey] = value;
+ }
+ }
+
+ return new StatMap<>(family, intValues, longValues, booleanValues, referenceValues);
+ }
+
+ private static int[] readKeys(DataInput input, int bytesPerId, int maxExpectedKeys)
+ throws IOException {
+ // sender may have more or less keys than expected.
+ // In case it has less, receiver extra keys will not be populated
+ // In case it has more, sender extra keys will be ignored
+ int numKeys = Math.min(readInt(input, bytesPerId), maxExpectedKeys);
+ int[] keys;
+ keys = new int[numKeys];
+ for (int i = 0; i < numKeys; i++) {
+ keys[i] = readInt(input, bytesPerId);
+ }
+ return keys;
+ }
+
+ private static void writeInt(DataOutput dos, int bytesPerId, int number) throws IOException {
+ // Write the lowest bytes of the number into the DataOutputStream
+ for (int i = 0; i < bytesPerId; i++) {
+ byte b = (byte) (number & 0xFF); // Get the lowest byte of the number
+ dos.writeByte(b); // Write the byte into the DataOutputStream
+ number >>= 8; // Shift the number 8 bits to the right to get the next byte
+ }
+ }
+
+ private static int readInt(DataInput dis, int bytesPerInt) throws IOException {
+ // Read the lowest bytes from the DataInputStream and reconstruct the integer
+ int number = 0;
+ for (int i = 0; i < bytesPerInt; i++) {
+ byte b = dis.readByte(); // Read the next byte from the DataInputStream
+ number |= (b & 0xFF) << (8 * i); // Combine the byte into the integer
+ }
+ return number;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ StatMap<?> statMap = (StatMap<?>) o;
+ return Objects.equals(_family, statMap._family) && Arrays.equals(_intValues, statMap._intValues)
+ && Arrays.equals(_longValues, statMap._longValues) && Arrays.equals(_booleanValues, statMap._booleanValues)
+ && Arrays.equals(_referenceValues, statMap._referenceValues);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = Objects.hash(_family);
+ result = 31 * result + Arrays.hashCode(_intValues);
+ result = 31 * result + Arrays.hashCode(_longValues);
+ result = 31 * result + Arrays.hashCode(_booleanValues);
+ result = 31 * result + Arrays.hashCode(_referenceValues);
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return asJson().toString();
+ }
+
+ public Class<K> getKeyClass() {
+ return _family._keyClass;
+ }
+
+ public boolean isEmpty() {
+ if (_intValues != null) {
+ for (int intValue : _intValues) {
+ if (intValue != 0) {
+ return false;
+ }
+ }
+ }
+ if (_longValues != null) {
+ for (long longValue : _longValues) {
+ if (longValue != 0) {
+ return false;
+ }
+ }
+ }
+ if (_booleanValues != null) {
+ for (boolean booleanValue : _booleanValues) {
+ if (booleanValue) {
+ return false;
+ }
+ }
+ }
+ if (_referenceValues != null) {
+ for (Object referenceValue : _referenceValues) {
+ if (referenceValue != null) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ private static class Family<K extends Enum<K> & Key> {
+ private final Class<K> _keyClass;
+ private final int _numIntsValues;
Review Comment:
(minor)
```suggestion
private final int _numIntValues;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576184899
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java:
##########
@@ -51,6 +59,7 @@
"segmentStatistics", "traceInfo", "partialResult"
})
public class BrokerResponseNative implements BrokerResponse {
Review Comment:
The main reason to modify this file is to have an easier way to merge the stats from V2 into V1. Basically BrokerResponseNative exports the V1 metrics into a different format. In order to make it easy to populate these stats from V2 (mainly from leaf stage) I've modified `BrokerResponseNative` to be store a StatMap inside. The difference is very small (mostly change setters and getters) but it ends up being closer to an ideal state where all these metrics are stored in the same way.
I guess I can remove this change, but then `MultiStageOperator.Type.LEAF.mergeInto(BrokerResponseNativeV2 response, , StatMap<?> map)`. I can take to follow that approach given it doesn't look you like the idea of touching this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1578088791
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlock.java:
##########
@@ -19,44 +19,50 @@
package org.apache.pinot.query.runtime.blocks;
import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import javax.annotation.Nullable;
import org.apache.pinot.common.datablock.ColumnarDataBlock;
import org.apache.pinot.common.datablock.DataBlock;
import org.apache.pinot.common.datablock.DataBlockUtils;
import org.apache.pinot.common.datablock.MetadataBlock;
import org.apache.pinot.common.datablock.RowDataBlock;
-import org.apache.pinot.common.response.ProcessingException;
import org.apache.pinot.common.utils.DataSchema;
import org.apache.pinot.core.common.Block;
import org.apache.pinot.core.common.datablock.DataBlockBuilder;
import org.apache.pinot.core.util.DataBlockExtractUtils;
-import org.apache.pinot.query.runtime.operator.OperatorStats;
-import org.apache.pinot.query.runtime.operator.utils.OperatorUtils;
-
+import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
/**
* A {@code TransferableBlock} is a wrapper around {@link DataBlock} for transferring data using
* {@link org.apache.pinot.common.proto.Mailbox}.
*/
public class TransferableBlock implements Block {
private final DataBlock.Type _type;
+ @Nullable
private final DataSchema _dataSchema;
private final int _numRows;
private List<Object[]> _container;
private DataBlock _dataBlock;
private Map<Integer, String> _errCodeToExceptionMap;
+ @Nullable
+ private final MultiStageQueryStats _queryStats;
- public TransferableBlock(List<Object[]> container, DataSchema dataSchema, DataBlock.Type type) {
+ public TransferableBlock(List<Object[]> container, @Nullable DataSchema dataSchema, DataBlock.Type type) {
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582728941
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -798,7 +798,6 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
_brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_NUM_GROUPS_LIMIT_REACHED,
1);
}
- brokerResponse.setPartialResult(isPartialResult(brokerResponse));
Review Comment:
removed in 2d9649147559782e40fab44542dc93c0bb999249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565231863
##########
pinot-common/src/main/java/org/apache/pinot/common/datatable/DataTable.java:
##########
@@ -100,7 +110,7 @@ enum MetadataValueType {
* - NEVER decrease MAX_ID
* Otherwise, backward compatibility will be broken.
*/
- enum MetadataKey {
+ enum MetadataKey implements StatMap.Key {
Review Comment:
How many of the keys apply to leaf stage? Should we consider adding a separate key set for leaf stage as a clean start? I have a little bit concern on managing both v1 and v2 stats in the same class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565537934
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingMultiStreamConsumer.java:
##########
@@ -240,6 +253,15 @@ protected boolean isEos(TransferableBlock element) {
return element.isSuccessfulEndOfStreamBlock();
}
+ @Override
+ protected void onConsumerFinish(TransferableBlock element) {
+ if (element.getQueryStats() != null) {
+ _stats.mergeUpstream(element.getQueryStats());
+ } else {
+ _stats.mergeUpstream(element.getSerializedStatsByStage());
Review Comment:
By doing this we don't need to deserialize the stats when they are sent using memory mailboxes. But right now we are not prioritizing sending the stats there. That is something we can improve in future PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1546149252
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java:
##########
@@ -71,9 +71,6 @@
* +-----------------------------------------------+
* | VARIABLE_SIZE_DATA SECTION |
* +-----------------------------------------------+
- * | METADATA LENGTH |
- * | METADATA SECTION |
- * +-----------------------------------------------+
Review Comment:
This wasn't useful. All blocks but metadata blocks left this empty and in metadata blocks the content was equivalent to variable_size_data, but codified as a single string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576200948
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
+ brokerResponse.addStageStats(JsonUtils.newObjectNode());
+ } else {
+ stageStats.forEach((type, stats) -> type.mergeInto(brokerResponse, stats));
+
+ DispatchablePlanFragment dispatchablePlanFragment = dispatchableSubPlan.getQueryStageList().get(i);
+ MultiStageStatsTreeBuilder treeBuilder = new MultiStageStatsTreeBuilder(stageStats);
+ PlanNode fragmentRoot = dispatchablePlanFragment.getPlanFragment().getFragmentRoot();
+ JsonNode node = fragmentRoot.visit(treeBuilder, null);
+ brokerResponse.addStageStats(node);
+ }
+ }
+ }
+
+ public static class MultiStageStatsTreeBuilder implements PlanNodeVisitor<JsonNode, Void> {
+ private final MultiStageQueryStats.StageStats.Closed _stageStats;
+ private int _index;
+ private static final String CHILDREN_KEY = "children";
+
+ public MultiStageStatsTreeBuilder(MultiStageQueryStats.StageStats.Closed stageStats) {
+ _stageStats = stageStats;
+ _index = stageStats.getLastOperatorIndex();
+ }
+
+ private ObjectNode selfNode(MultiStageOperator.Type type) {
+ ObjectNode json = JsonUtils.newObjectNode();
+ json.put("type", type.toString());
+ Iterator<Map.Entry<String, JsonNode>> statsIt = _stageStats.getOperatorStats(_index).asJson().fields();
+ while (statsIt.hasNext()) {
+ Map.Entry<String, JsonNode> entry = statsIt.next();
+ json.set(entry.getKey(), entry.getValue());
+ }
+ return json;
+ }
+
+ private JsonNode recursiveCase(AbstractPlanNode node, MultiStageOperator.Type expectedType) {
+ MultiStageOperator.Type type = _stageStats.getOperatorType(_index);
+ if (type != expectedType) {
+ if (type == MultiStageOperator.Type.LEAF) {
+ // Leaf nodes compile the plan node into a single operator and therefore return a single stat
+ return selfNode(type);
+ }
+ int childrenSize = node.getInputs().size();
+ switch (childrenSize) {
+ case 0:
+ LOGGER.warn("Skipping unexpected node {} when stat of type {} was found at index {}",
+ node.getClass(), type, _index);
+ return JsonUtils.newObjectNode();
+ case 1:
+ LOGGER.warn("Skipping unexpected node {} when stat of type {} was found at index {}",
+ node.getClass(), type, _index);
+ return node.getInputs().get(0).visit(this, null);
+ default:
+ throw new IllegalStateException("Expected operator type: " + expectedType + ", but got: " + type + " with "
Review Comment:
I've modified the code to try to return something, even it doesn't make much sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1587275146
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
> Hmm, if we are encoding errors differently in v1 and v2, how does v1 understand the errors in v2?
In V1 the quality of being an error is reported in two different ways:
1. By having a map of exceptions, which is serialized in a specific byte buffer section.
2. By setting the `type` in the json `Contents` stored in the variable size section.
In V2 we just use the first option. I that is what the very old _V0_ (whose source code is in `MetadataBlockTest.V0MetadataBlock`) in version was doing. Given V1 can read V0, it works as expected.
Specifically, V1 is able to read bytes where the variable section is empty or `Contents.type` is not defined because `getType` is defined as:
```java
public MetadataBlock.MetadataBlockType getType() {
String type = _contents.getType();
// if type is null, then we're reading a legacy block where we didn't encode any
// data. assume that it is an EOS block if there's no exceptions and an ERROR block
// otherwise
return type == null
? (getExceptions().isEmpty() ? MetadataBlock.MetadataBlockType.EOS : MetadataBlock.MetadataBlockType.ERROR)
: MetadataBlock.MetadataBlockType.valueOf(type);
}
```
Given we don't set `Contents.getType`, its value is null and we return EOS or ERROR depending on whether exceptions is empty or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582735688
##########
pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java:
##########
@@ -123,6 +127,17 @@ String toJsonString()
*/
long getMinConsumingFreshnessTimeMs();
+ /**
+ * Get the max number of rows seen by a single operator in the query processing chain.
+ * <p>
+ * In single-stage this value is equal to {@link #getNumEntriesScannedPostFilter()} because single-stage operators
+ * cannot generate more rows than the number of rows received. This is not the case in multi-stage operators, where
+ * operators like join or union can emit more rows than the ones received on each of its children operators.
+ */
+ default long getMaxRowsInOperator() {
Review Comment:
~This is not a new method but a method that existed in `BrokerResponseNative` and was inherited by `BrokerResponseV2` . Given it is already there (even it doesn't make sense) and it was used to calculate `isPartialResponse` I think it is useful to have it in the base interface now that `BrokerResponseNativeV2` does not inherit `BrokerResponseNative` anymore.~
~The alternative would be to do not make `isPartialResponse` `default` and include the same code in both `BrokerResponseNative` and `BrokerResponseNativeV2`. If you think that is better I can change it, but it looks a bit repetitive. For sure currently how this stats are being declared doesn't make sense, but we can try to improve that in the future~
I thought this was `isMaxRowsInJoinReached` not `getMaxRowsInOperator`. That is why I wrote the two paragraphs above. I'm keeping them just to have the context.
The reason to define `getMaxRowsInOperator` is different. We need this method here because this is an important stat. This PR adds this value into the table that shows stats in Pinot UI and it is easier to assume this exists for both V1 and V2 than dealing with its absence. Also, it makes sense to have a stat that shows the max number of rows seen in any operator. In V1 that is limited by the number of elements returned by a filter, but in V2 that is not the case. I think having this attribute in both V1 and V2 is going to make the life easier for users who can be trained to just look for this value instead of reading `numEntriesScannedPostFilter` in V1 and `maxRowsInOperator` in V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583055716
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -135,14 +160,128 @@ protected TransferableBlock getNextBlock()
}
}
- private TransferableBlock constructMetadataBlock() {
- // All data blocks have been returned. Record the stats and return EOS.
- Map<String, String> executionStats = _executionStats;
+ private void mergeExecutionStats(@Nullable Map<String, String> executionStats) {
if (executionStats != null) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, getOperatorId());
- operatorStats.recordExecutionStats(executionStats);
+ for (Map.Entry<String, String> entry : executionStats.entrySet()) {
+ DataTable.MetadataKey key = DataTable.MetadataKey.getByName(entry.getKey());
+ if (key == null) {
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ continue;
+ }
+ switch (key) {
+ case UNKNOWN:
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ break;
+ case TABLE:
Review Comment:
Nah, it doesn't seem easy to add the table when the stat json is being build. Specifically in cases like the one where we have a pipeline breaker (semi join) the logical plan is quite different to the physical plan which makes it very difficult to infer the name of the table from the logical plan.
Even it is a bit hackish, I'm going to keep it the table as a stat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565391241
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -99,10 +105,32 @@ public LeafStageTransferableBlockOperator(OpChainExecutionContext context, List<
Integer maxStreamingPendingBlocks = QueryOptionsUtils.getMaxStreamingPendingBlocks(context.getOpChainMetadata());
_blockingQueue = new ArrayBlockingQueue<>(maxStreamingPendingBlocks != null ? maxStreamingPendingBlocks
: QueryOptionValue.DEFAULT_MAX_STREAMING_PENDING_BLOCKS);
+ String tableName = context.getLeafStageContext().getStagePlan().getStageMetadata().getTableName();
+ _statMap.merge(StatKey.TABLE, tableName);
Review Comment:
This is one of the places where we are modifying the map from different threads. Here the constructor is called by a different thread than the one executing the code.
In fact I found the concurrency issue in leaf stages stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565400481
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/AggregateOperator.java:
##########
@@ -132,29 +156,31 @@ public String toExplainString() {
@Override
protected TransferableBlock getNextBlock() {
if (_hasConstructedAggregateBlock) {
- return TransferableBlockUtils.getEndOfStreamTransferableBlock();
+ assert _eosBlock != null;
Review Comment:
Instead of an empty EOS block now we send a block with all the stats. This is a pattern that is repeated in all other operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565423170
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -19,107 +19,295 @@
package org.apache.pinot.query.runtime.operator;
import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import java.io.DataInput;
+import java.io.IOException;
import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.common.response.broker.BrokerResponseNativeV2;
import org.apache.pinot.core.common.Operator;
import org.apache.pinot.query.runtime.blocks.TransferableBlock;
import org.apache.pinot.query.runtime.blocks.TransferableBlockUtils;
+import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
import org.apache.pinot.query.runtime.plan.OpChainExecutionContext;
+import org.apache.pinot.query.runtime.plan.pipeline.PipelineBreakerOperator;
import org.apache.pinot.spi.exception.EarlyTerminationException;
import org.apache.pinot.spi.trace.InvocationScope;
import org.apache.pinot.spi.trace.Tracing;
import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-public abstract class MultiStageOperator implements Operator<TransferableBlock>, AutoCloseable {
- private static final Logger LOGGER = LoggerFactory.getLogger(MultiStageOperator.class);
+public abstract class MultiStageOperator<K extends Enum<K> & StatMap.Key>
+ implements Operator<TransferableBlock>, AutoCloseable {
protected final OpChainExecutionContext _context;
protected final String _operatorId;
- protected final OpChainStats _opChainStats;
+ protected final StatMap<K> _statMap;
protected boolean _isEarlyTerminated;
- public MultiStageOperator(OpChainExecutionContext context) {
+ public MultiStageOperator(OpChainExecutionContext context, Class<K> keyStatClass) {
_context = context;
_operatorId = Joiner.on("_").join(getClass().getSimpleName(), _context.getStageId(), _context.getServer());
- _opChainStats = _context.getStats();
_isEarlyTerminated = false;
+ _statMap = new StatMap<>(keyStatClass);
}
+ protected abstract Logger logger();
+
+ public abstract Type getOperatorType();
+
+ public abstract K getExecutionTimeKey();
+
+ public abstract K getEmittedRowsKey();
+
@Override
public TransferableBlock nextBlock() {
if (Tracing.ThreadAccountantOps.isInterrupted()) {
throw new EarlyTerminationException("Interrupted while processing next block");
}
+ if (logger().isDebugEnabled()) {
+ logger().debug("Operator {}: Reading next block", _operatorId);
+ }
try (InvocationScope ignored = Tracing.getTracer().createScope(getClass())) {
TransferableBlock nextBlock;
if (shouldCollectStats()) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, _operatorId);
- operatorStats.startTimer();
+ Stopwatch executeStopwatch = Stopwatch.createStarted();
Review Comment:
We discussed on whether we want to always collect time stats or not. Right now this PR is not modifying that, but it should be as easy as to always return true in `shouldCollectStats` or just remove this if
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576213860
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/ReceivingMailbox.java:
##########
@@ -72,11 +81,53 @@ public String getId() {
return _id;
}
+ /**
+ * Offers a raw block into the mailbox within the timeout specified, returns whether the block is successfully added.
+ * If the block is not added, an error block is added to the mailbox.
+ * <p>
+ * Contrary to {@link #offer(TransferableBlock, long)}, the block may be an
+ * {@link TransferableBlock#isErrorBlock() error block}.
+ */
+ public ReceivingMailboxStatus offerRaw(ByteBuffer byteBuffer, long timeoutMs)
+ throws IOException {
+ TransferableBlock block;
+ long now = System.currentTimeMillis();
+ _stats.merge(StatKey.WAIT_CPU_TIME_MS, now - _lastArriveTime);
+ _lastArriveTime = now;
+ _stats.merge(StatKey.DESERIALIZED_BYTES, byteBuffer.remaining());
+ _stats.merge(StatKey.DESERIALIZED_MESSAGES, 1);
+
+ now = System.currentTimeMillis();
+ DataBlock dataBlock = DataBlockUtils.getDataBlock(byteBuffer);
+ _stats.merge(StatKey.DESERIALIZATION_TIME_MS, System.currentTimeMillis() - now);
+
+ if (dataBlock instanceof MetadataBlock) {
+ Map<Integer, String> exceptions = dataBlock.getExceptions();
+ if (exceptions.isEmpty()) {
+ block = TransferableBlockUtils.wrap(dataBlock);
+ } else {
+ setErrorBlock(TransferableBlockUtils.getErrorTransferableBlock(exceptions));
+ return ReceivingMailboxStatus.FIRST_ERROR;
Review Comment:
Older code was treating this case in a bit different way. This method was mostly moved from MailboxContentObserver, which in this case the coded was doing:
```
_mailbox.setErrorBlock(TransferableBlockUtils.getErrorTransferableBlock(exceptions));
return;
```
The only reason to add this `FIRST_ERROR` enum is to keep the same behavior in `MailboxContentObserver`. I would prefer to just return `ERROR` here, but that would have the of printing the following log in `MailboxContentObserver`.
```
LOGGER.warn("Mailbox: {} already errored out (received error block before)", mailboxId);
```
I guess we can remove this new enum and change the code in `MailboxContentObserver` a little bit.
> Also seems this message doesn't apply to in-memory mailbox, some comments would help explain it
`InMemorySendingMailbox` does never call this code because this enum is only returned when there is an error deserializing the message. Given messages are always heap based in InMemorySendingMailbox, it can never fail in that way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1580275410
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNativeV2.java:
##########
@@ -77,20 +100,301 @@ public static BrokerResponseNativeV2 empty() {
return new BrokerResponseNativeV2();
}
- public static BrokerResponseNativeV2 fromJsonString(String jsonString)
- throws IOException {
- return JsonUtils.stringToObject(jsonString, BrokerResponseNativeV2.class);
+ public void addStageStats(JsonNode stageStats) {
+ ObjectNode node = JsonUtils.newObjectNode();
+ node.put("stage", _stageIdStats.size());
+ node.set("stats", stageStats);
+ _stageIdStats.add(node);
+ }
+
+ @JsonProperty
+ public List<JsonNode> getStageStats() {
+ return _stageIdStats;
+ }
+
+ @JsonProperty
+ public long getMaxRows() {
+ return _maxRows;
+ }
+
+ public void mergeMaxRows(long maxRows) {
+ _maxRows = Math.max(_maxRows, maxRows);
+ }
+
+ @Override
+ public void setTimeUsedMs(long timeUsedMs) {
+ _serverStats.merge(DataTable.MetadataKey.TIME_USED_MS, timeUsedMs);
+ }
+
+ @Override
+ public long getNumDocsScanned() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_DOCS_SCANNED);
+ }
+
+ @Override
+ public long getNumEntriesScannedInFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_IN_FILTER);
+ }
+
+ @Override
+ public long getNumEntriesScannedPostFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_POST_FILTER);
+ }
+
+ @Override
+ public long getNumSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getMinConsumingFreshnessTimeMs() {
+ return _serverStats.getLong(DataTable.MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS);
+ }
+
+ @Override
+ public long getTotalDocs() {
+ return _serverStats.getLong(DataTable.MetadataKey.TOTAL_DOCS);
+ }
+
+ @Override
+ public boolean isNumGroupsLimitReached() {
+ return _serverStats.getBoolean(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED);
+ }
+
+ public void mergeNumGroupsLimitReached(boolean numGroupsLimitReached) {
+ _serverStats.merge(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED, numGroupsLimitReached);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByServer() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_SERVER);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedInvalid() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_INVALID);
}
- public void addStageStat(Integer stageId, BrokerResponseStats brokerResponseStats) {
- // StageExecutionWallTime will always be there, other stats are optional such as OperatorStats
- if (brokerResponseStats.getStageExecWallTimeMs() != -1) {
- _stageIdStats.put(stageId, brokerResponseStats);
+ @Override
+ public long getNumSegmentsPrunedByLimit() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_LIMIT);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByValue() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_VALUE);
+ }
+
+ @Override
+ public long getExplainPlanNumEmptyFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getExplainPlanNumMatchAllFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getOfflineTotalCpuTimeNs() {
+ return getOfflineThreadCpuTimeNs() + getOfflineSystemActivitiesCpuTimeNs()
+ + getOfflineResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public long getRealtimeTotalCpuTimeNs() {
+ return getRealtimeThreadCpuTimeNs() + getRealtimeSystemActivitiesCpuTimeNs()
+ + getRealtimeResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public void setExceptions(List<ProcessingException> exceptions) {
+ for (ProcessingException exception : exceptions) {
+ _processingExceptions.add(new QueryProcessingException(exception.getErrorCode(), exception.getMessage()));
}
}
- @JsonProperty("stageStats")
- public Map<Integer, BrokerResponseStats> getStageIdStats() {
- return _stageIdStats;
+ public void addToExceptions(QueryProcessingException processingException) {
+ _processingExceptions.add(processingException);
+ }
+
+ @Override
+ public int getNumServersQueried() {
+ return _numServersQueried;
+ }
+
+ @Override
+ public void setNumServersQueried(int numServersQueried) {
+ _numServersQueried = numServersQueried;
+ }
+
+ @Override
+ public int getNumServersResponded() {
+ return _numServersResponded;
+ }
+
+ @Override
+ public void setNumServersResponded(int numServersResponded) {
+ _numServersResponded = numServersResponded;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public boolean isMaxRowsInJoinReached() {
+ return _maxRowsInJoinReached;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public void mergeMaxRowsInJoinReached(boolean maxRowsInJoinReached) {
+ _maxRowsInJoinReached |= maxRowsInJoinReached;
+ }
+
+ @Override
+ public int getExceptionsSize() {
+ return _processingExceptions.size();
+ }
+
+ @Override
+ public void setResultTable(ResultTable resultTable) {
+ _resultTable = resultTable;
+ }
+
+ @Override
+ public ResultTable getResultTable() {
+ return _resultTable;
+ }
+
+ @Override
+ public List<QueryProcessingException> getProcessingExceptions() {
+ return List.of();
+ }
+
+ @Override
+ public int getNumRowsResultSet() {
+ return 0;
+ }
+
+ @Override
+ public long getOfflineThreadCpuTimeNs() {
+ return _offlineThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeThreadCpuTimeNs() {
+ return _realtimeThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineSystemActivitiesCpuTimeNs() {
+ return _offlineSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeSystemActivitiesCpuTimeNs() {
+ return _realtimeSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineResponseSerializationCpuTimeNs() {
+ return _offlineResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeResponseSerializationCpuTimeNs() {
+ return _realtimeResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByBroker() {
+ return _numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public void setNumSegmentsPrunedByBroker(long numSegmentsPrunedByBroker) {
+ _numSegmentsPrunedByBroker = numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public String getRequestId() {
+ return _requestId;
+ }
+
+ @Override
+ public void setRequestId(String requestId) {
+ _requestId = requestId;
+ }
+
+ @Override
+ public String getBrokerId() {
+ return _brokerId;
+ }
+
+ @Override
+ public void setBrokerId(String requestId) {
+ _brokerId = requestId;
+ }
+
+ @Override
+ public long getBrokerReduceTimeMs() {
+ return _brokerReduceTimeMs;
+ }
+
+ @Override
+ public void setBrokerReduceTimeMs(long brokerReduceTimeMs) {
+ _brokerReduceTimeMs = brokerReduceTimeMs;
+ }
+
+ @JsonProperty(access = JsonProperty.Access.READ_ONLY)
+ @Override
+ public boolean isPartialResult() {
+ return isNumGroupsLimitReached() || getExceptionsSize() > 0 || isMaxRowsInJoinReached();
+ }
+
+ public void addServerStats(StatMap<DataTable.MetadataKey> serverStats) {
+ // Set execution statistics.
+ _serverStats.merge(serverStats);
+
+ long threadCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.THREAD_CPU_TIME_NS);
+ long systemActivitiesCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.SYSTEM_ACTIVITIES_CPU_TIME_NS);
+ long responseSerializationCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.RESPONSE_SER_CPU_TIME_NS);
+
+ String tableName = serverStats.getString(DataTable.MetadataKey.TABLE);
+ if (tableName != null) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
+
+ if (tableType == TableType.OFFLINE) {
+ _offlineThreadCpuTimeNs += threadCpuTimeNs;
+ _offlineSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _offlineResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ } else {
+ _realtimeThreadCpuTimeNs += threadCpuTimeNs;
+ _realtimeSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _realtimeResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ }
Review Comment:
Oh, I think I misunderstood the issue, and we are already reposting the stats in leaf stage operator. And now I understand better why we need to make V1 stats also implementing `StatMap.Key`.
I guess we can just leave them as 0. People can lookup the V2 stats to get the details
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576190502
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
+ brokerResponse.addStageStats(JsonUtils.newObjectNode());
+ } else {
+ stageStats.forEach((type, stats) -> type.mergeInto(brokerResponse, stats));
+
+ DispatchablePlanFragment dispatchablePlanFragment = dispatchableSubPlan.getQueryStageList().get(i);
+ MultiStageStatsTreeBuilder treeBuilder = new MultiStageStatsTreeBuilder(stageStats);
+ PlanNode fragmentRoot = dispatchablePlanFragment.getPlanFragment().getFragmentRoot();
+ JsonNode node = fragmentRoot.visit(treeBuilder, null);
+ brokerResponse.addStageStats(node);
+ }
+ }
+ }
+
+ public static class MultiStageStatsTreeBuilder implements PlanNodeVisitor<JsonNode, Void> {
+ private final MultiStageQueryStats.StageStats.Closed _stageStats;
+ private int _index;
+ private static final String CHILDREN_KEY = "children";
+
+ public MultiStageStatsTreeBuilder(MultiStageQueryStats.StageStats.Closed stageStats) {
+ _stageStats = stageStats;
+ _index = stageStats.getLastOperatorIndex();
+ }
+
+ private ObjectNode selfNode(MultiStageOperator.Type type) {
+ ObjectNode json = JsonUtils.newObjectNode();
+ json.put("type", type.toString());
+ Iterator<Map.Entry<String, JsonNode>> statsIt = _stageStats.getOperatorStats(_index).asJson().fields();
+ while (statsIt.hasNext()) {
+ Map.Entry<String, JsonNode> entry = statsIt.next();
+ json.set(entry.getKey(), entry.getValue());
+ }
+ return json;
+ }
+
+ private JsonNode recursiveCase(AbstractPlanNode node, MultiStageOperator.Type expectedType) {
+ MultiStageOperator.Type type = _stageStats.getOperatorType(_index);
+ if (type != expectedType) {
Review Comment:
No, this is very rare. Mostly only happens on the stage 0 and in leaf stages.
In stage 0 always have two nodes. One is a transform IIRC and the other is a receive node and we only have stats for the latter.
In leaf stage, we may have several nodes, but the leaf stage compiles them into a different plan (as explained above).
In can add some comments
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
Review Comment:
I've made some tests and I think it should never be null. But I guess it is better to be safe to do not throw in that case. I'm going to add some comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1581631413
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -798,7 +798,6 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
_brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_NUM_GROUPS_LIMIT_REACHED,
1);
}
- brokerResponse.setPartialResult(isPartialResult(brokerResponse));
Review Comment:
(minor) Also remove `isPartialResult()` method
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -419,4 +550,122 @@ public void send(BaseResultsBlock block)
addResultsBlock(block);
}
}
+
+ public enum StatKey implements StatMap.Key {
+ TABLE(StatMap.Type.STRING),
+ EXECUTION_TIME_MS(StatMap.Type.LONG, null, DataTable.MetadataKey.TIME_USED_MS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ EMITTED_ROWS(StatMap.Type.LONG, null, DataTable.MetadataKey.NUM_ROWS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ NUM_DOCS_SCANNED(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(StatMap.Type.LONG),
+ NUM_SEGMENTS_QUERIED(StatMap.Type.INT),
+ NUM_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(StatMap.Type.INT),
+ // the timestamp indicating the freshness of the data queried in consuming segments.
+ // This can be ingestion timestamp if provided by the stream, or the last index time
+ MIN_CONSUMING_FRESHNESS_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ TOTAL_DOCS(StatMap.Type.LONG),
+ NUM_GROUPS_LIMIT_REACHED(StatMap.Type.BOOLEAN),
+ //TRACE_INFO(StatMap.Type.STRING),
+ //REQUEST_ID(StatMap.Type.LONG),
+ NUM_RESIZES(StatMap.Type.INT),
+ RESIZE_TIME_MS(StatMap.Type.LONG),
+ THREAD_CPU_TIME_NS(StatMap.Type.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(StatMap.Type.LONG),
+ RESPONSE_SER_CPU_TIME_NS(StatMap.Type.LONG, "responseSerializationCpuTimeNs"),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_BLOCKS(StatMap.Type.INT),
+ OPERATOR_EXECUTION_TIME_MS(StatMap.Type.LONG),
+ OPERATOR_ID(StatMap.Type.STRING),
+ OPERATOR_EXEC_START_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ OPERATOR_EXEC_END_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return Math.max(value1, value2);
+ }
+ },;
+ private final StatMap.Type _type;
+ @Nullable
+ private final DataTable.MetadataKey _v1Key;
+ private final String _statName;
+
+ StatKey(StatMap.Type type) {
+ this(type, null);
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = DataTable.MetadataKey.getByName(getStatName());
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName, @Nullable DataTable.MetadataKey v1Key) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = v1Key == null ? DataTable.MetadataKey.getByName(getStatName()) : v1Key;
+ }
+
+ @Override
+ public String getStatName() {
+ return _statName;
+ }
+
+ @Override
+ public StatMap.Type getType() {
+ return _type;
+ }
+
+ public void updateV1Metadata(StatMap<DataTable.MetadataKey> oldMetadata, StatMap<StatKey> stats) {
Review Comment:
Here we maintained a map from v2 key to v1 key, and use v2 stats to fill the v1 stats, then use v1 stats to fill the broker response. IMO this will make future decoupling of v1 and v2 hard.
Does it work if we do not fill the v1 stats, but directly use v2 stats to fill the broker response? I'm not sure if you did this to simplify the broker response V1 handling, but since we already decoupled broker response v1 and v2, should we also decouple this?
The v1 stats key to v2 mapping is done through the switch case in `mergeExecutionStats()`. If we remove the mapping from v2 to v1, can we make v1 stats not implementing `StatMap.Key`? The overall goal here is to decouple the v2 and v1 handling so that in the future we can remove v1 easier if necessary.
##########
pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java:
##########
@@ -123,6 +127,17 @@ String toJsonString()
*/
long getMinConsumingFreshnessTimeMs();
+ /**
+ * Get the max number of rows seen by a single operator in the query processing chain.
+ * <p>
+ * In single-stage this value is equal to {@link #getNumEntriesScannedPostFilter()} because single-stage operators
+ * cannot generate more rows than the number of rows received. This is not the case in multi-stage operators, where
+ * operators like join or union can emit more rows than the ones received on each of its children operators.
+ */
+ default long getMaxRowsInOperator() {
Review Comment:
I'd suggest not adding new stats on the top level.
In the future we might want to re-design the UI for V2 stats, and all the V2 stats should come from `StatMap`.
If this stats is truly useful, we may revisit it in a separate PR.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -135,14 +160,128 @@ protected TransferableBlock getNextBlock()
}
}
- private TransferableBlock constructMetadataBlock() {
- // All data blocks have been returned. Record the stats and return EOS.
- Map<String, String> executionStats = _executionStats;
+ private void mergeExecutionStats(@Nullable Map<String, String> executionStats) {
if (executionStats != null) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, getOperatorId());
- operatorStats.recordExecutionStats(executionStats);
+ for (Map.Entry<String, String> entry : executionStats.entrySet()) {
+ DataTable.MetadataKey key = DataTable.MetadataKey.getByName(entry.getKey());
+ if (key == null) {
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ continue;
+ }
+ switch (key) {
+ case UNKNOWN:
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ break;
+ case TABLE:
Review Comment:
Do we need table? Seems we are simply overriding it?
I think we can remove it from v2 stats
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -135,14 +160,128 @@ protected TransferableBlock getNextBlock()
}
}
- private TransferableBlock constructMetadataBlock() {
- // All data blocks have been returned. Record the stats and return EOS.
- Map<String, String> executionStats = _executionStats;
+ private void mergeExecutionStats(@Nullable Map<String, String> executionStats) {
if (executionStats != null) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, getOperatorId());
- operatorStats.recordExecutionStats(executionStats);
+ for (Map.Entry<String, String> entry : executionStats.entrySet()) {
+ DataTable.MetadataKey key = DataTable.MetadataKey.getByName(entry.getKey());
+ if (key == null) {
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ continue;
+ }
+ switch (key) {
+ case UNKNOWN:
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ break;
+ case TABLE:
Review Comment:
After removing `TABLE` and `OPERATOR_ID`, we might be able to remove `STRING` type. string stats doesn't make much sense
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
I'm still confused. Is this the only place where v1 metadata block is used? Does it handle error differently from v2 metadata block?
Does it work if we ignore the version and always `return new MetadataBlock(byteBuffer);`?
If the error handling is different, then sending v2 metadata block to old server will cause backward incompatibility right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583822172
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -135,14 +160,128 @@ protected TransferableBlock getNextBlock()
}
}
- private TransferableBlock constructMetadataBlock() {
- // All data blocks have been returned. Record the stats and return EOS.
- Map<String, String> executionStats = _executionStats;
+ private void mergeExecutionStats(@Nullable Map<String, String> executionStats) {
if (executionStats != null) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, getOperatorId());
- operatorStats.recordExecutionStats(executionStats);
+ for (Map.Entry<String, String> entry : executionStats.entrySet()) {
+ DataTable.MetadataKey key = DataTable.MetadataKey.getByName(entry.getKey());
+ if (key == null) {
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ continue;
+ }
+ switch (key) {
+ case UNKNOWN:
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ break;
+ case TABLE:
Review Comment:
Can you explain how these 2 stats are used? I don't see them used in the broker response, and seems the merge rule is overriding, meaning if we gathered 2 tables/operatorIds, the second one will override the first one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1586803055
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
Hmm, if we are encoding errors differently in v1 and v2, how does v1 understand the errors in v2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1575464681
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java:
##########
@@ -51,6 +59,7 @@
"segmentStatistics", "traceInfo", "partialResult"
})
public class BrokerResponseNative implements BrokerResponse {
Review Comment:
I'm trying to follow the changes in this class. This class represents the response for v1 queries. Currently (before this PR) we have a `BrokerResponseNativeV2` which extends the v1 query response with just an additional stage id stats list. I feel we should use this opportunity to clean them up and decouple the v2 response format from the v1 format because a lot of things do not apply for v2, and with v2 stats we don't really need the extra fields in this class.
I guess the intention here is to keep things compatible with the current interface. Shall we keep the changes within `BrokerResponseNativeV2` if possible? We can make it directly implements `BrokerResponse` so that we can easily decouple v1 and v2 in the future
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
Review Comment:
For my understanding, will this ever contain `null` value? Is this for the broker stage (stage 0)? If so, some comments can help understanding the code
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/ReceivingMailbox.java:
##########
@@ -72,11 +81,53 @@ public String getId() {
return _id;
}
+ /**
+ * Offers a raw block into the mailbox within the timeout specified, returns whether the block is successfully added.
+ * If the block is not added, an error block is added to the mailbox.
+ * <p>
+ * Contrary to {@link #offer(TransferableBlock, long)}, the block may be an
+ * {@link TransferableBlock#isErrorBlock() error block}.
+ */
+ public ReceivingMailboxStatus offerRaw(ByteBuffer byteBuffer, long timeoutMs)
+ throws IOException {
+ TransferableBlock block;
+ long now = System.currentTimeMillis();
+ _stats.merge(StatKey.WAIT_CPU_TIME_MS, now - _lastArriveTime);
+ _lastArriveTime = now;
+ _stats.merge(StatKey.DESERIALIZED_BYTES, byteBuffer.remaining());
+ _stats.merge(StatKey.DESERIALIZED_MESSAGES, 1);
+
+ now = System.currentTimeMillis();
+ DataBlock dataBlock = DataBlockUtils.getDataBlock(byteBuffer);
+ _stats.merge(StatKey.DESERIALIZATION_TIME_MS, System.currentTimeMillis() - now);
+
+ if (dataBlock instanceof MetadataBlock) {
+ Map<Integer, String> exceptions = dataBlock.getExceptions();
+ if (exceptions.isEmpty()) {
+ block = TransferableBlockUtils.wrap(dataBlock);
+ } else {
+ setErrorBlock(TransferableBlockUtils.getErrorTransferableBlock(exceptions));
+ return ReceivingMailboxStatus.FIRST_ERROR;
Review Comment:
`FIRST_ERROR` is a little bit confusing. I think this means the error block is successfully set? Maybe make it more explicit like `ERROR_BLOCK_SET`?
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/V1MetadataBlock.java:
##########
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.datablock;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import java.util.Map;
+
+
+/**
+ * The datablock used before the introduction of {@link org.apache.pinot.common.datatable.StatMap}.
+ * <p>
+ * This version stored the metadata in a {@code Map<String, String>} which was encoded as JSON and stored in the
+ * variable size data buffer.
+ * <p>
+ * Instances of this class are not actually seen by the operators. Instead, they are converted to {@link MetadataBlock}
+ * in {@link MetadataBlock#deserialize(ByteBuffer, int)}.
+ * <p>
+ * The reason to keep it here is mostly for backwards compatibility and testing. In order to simplify the code, the
+ * stats engine just ignores the metadata of these objects, but we need to be able to deserialize them anyway.
+ */
+public class V1MetadataBlock extends BaseDataBlock {
Review Comment:
Shall we mark it deprecated? Or maybe moving it to test package since it is just for testing purpose?
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
+ brokerResponse.addStageStats(JsonUtils.newObjectNode());
+ } else {
+ stageStats.forEach((type, stats) -> type.mergeInto(brokerResponse, stats));
+
+ DispatchablePlanFragment dispatchablePlanFragment = dispatchableSubPlan.getQueryStageList().get(i);
+ MultiStageStatsTreeBuilder treeBuilder = new MultiStageStatsTreeBuilder(stageStats);
+ PlanNode fragmentRoot = dispatchablePlanFragment.getPlanFragment().getFragmentRoot();
+ JsonNode node = fragmentRoot.visit(treeBuilder, null);
+ brokerResponse.addStageStats(node);
+ }
+ }
+ }
+
+ public static class MultiStageStatsTreeBuilder implements PlanNodeVisitor<JsonNode, Void> {
+ private final MultiStageQueryStats.StageStats.Closed _stageStats;
+ private int _index;
+ private static final String CHILDREN_KEY = "children";
+
+ public MultiStageStatsTreeBuilder(MultiStageQueryStats.StageStats.Closed stageStats) {
+ _stageStats = stageStats;
+ _index = stageStats.getLastOperatorIndex();
+ }
+
+ private ObjectNode selfNode(MultiStageOperator.Type type) {
+ ObjectNode json = JsonUtils.newObjectNode();
+ json.put("type", type.toString());
+ Iterator<Map.Entry<String, JsonNode>> statsIt = _stageStats.getOperatorStats(_index).asJson().fields();
+ while (statsIt.hasNext()) {
+ Map.Entry<String, JsonNode> entry = statsIt.next();
+ json.set(entry.getKey(), entry.getValue());
+ }
+ return json;
+ }
+
+ private JsonNode recursiveCase(AbstractPlanNode node, MultiStageOperator.Type expectedType) {
+ MultiStageOperator.Type type = _stageStats.getOperatorType(_index);
+ if (type != expectedType) {
Review Comment:
Is this regular case? Some comments can help explain the logic
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
+ brokerResponse.addStageStats(JsonUtils.newObjectNode());
+ } else {
+ stageStats.forEach((type, stats) -> type.mergeInto(brokerResponse, stats));
+
+ DispatchablePlanFragment dispatchablePlanFragment = dispatchableSubPlan.getQueryStageList().get(i);
+ MultiStageStatsTreeBuilder treeBuilder = new MultiStageStatsTreeBuilder(stageStats);
+ PlanNode fragmentRoot = dispatchablePlanFragment.getPlanFragment().getFragmentRoot();
+ JsonNode node = fragmentRoot.visit(treeBuilder, null);
+ brokerResponse.addStageStats(node);
+ }
+ }
+ }
+
+ public static class MultiStageStatsTreeBuilder implements PlanNodeVisitor<JsonNode, Void> {
+ private final MultiStageQueryStats.StageStats.Closed _stageStats;
+ private int _index;
+ private static final String CHILDREN_KEY = "children";
+
+ public MultiStageStatsTreeBuilder(MultiStageQueryStats.StageStats.Closed stageStats) {
+ _stageStats = stageStats;
+ _index = stageStats.getLastOperatorIndex();
+ }
+
+ private ObjectNode selfNode(MultiStageOperator.Type type) {
+ ObjectNode json = JsonUtils.newObjectNode();
+ json.put("type", type.toString());
+ Iterator<Map.Entry<String, JsonNode>> statsIt = _stageStats.getOperatorStats(_index).asJson().fields();
+ while (statsIt.hasNext()) {
+ Map.Entry<String, JsonNode> entry = statsIt.next();
+ json.set(entry.getKey(), entry.getValue());
+ }
+ return json;
+ }
+
+ private JsonNode recursiveCase(AbstractPlanNode node, MultiStageOperator.Type expectedType) {
+ MultiStageOperator.Type type = _stageStats.getOperatorType(_index);
+ if (type != expectedType) {
+ if (type == MultiStageOperator.Type.LEAF) {
+ // Leaf nodes compile the plan node into a single operator and therefore return a single stat
+ return selfNode(type);
+ }
+ int childrenSize = node.getInputs().size();
+ switch (childrenSize) {
+ case 0:
+ LOGGER.warn("Skipping unexpected node {} when stat of type {} was found at index {}",
+ node.getClass(), type, _index);
+ return JsonUtils.newObjectNode();
+ case 1:
+ LOGGER.warn("Skipping unexpected node {} when stat of type {} was found at index {}",
+ node.getClass(), type, _index);
+ return node.getInputs().get(0).visit(this, null);
+ default:
+ throw new IllegalStateException("Expected operator type: " + expectedType + ", but got: " + type + " with "
Review Comment:
Do we want to fail the whole query when stats processing fails?
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlock.java:
##########
@@ -19,44 +19,50 @@
package org.apache.pinot.query.runtime.blocks;
import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import javax.annotation.Nullable;
import org.apache.pinot.common.datablock.ColumnarDataBlock;
import org.apache.pinot.common.datablock.DataBlock;
import org.apache.pinot.common.datablock.DataBlockUtils;
import org.apache.pinot.common.datablock.MetadataBlock;
import org.apache.pinot.common.datablock.RowDataBlock;
-import org.apache.pinot.common.response.ProcessingException;
import org.apache.pinot.common.utils.DataSchema;
import org.apache.pinot.core.common.Block;
import org.apache.pinot.core.common.datablock.DataBlockBuilder;
import org.apache.pinot.core.util.DataBlockExtractUtils;
-import org.apache.pinot.query.runtime.operator.OperatorStats;
-import org.apache.pinot.query.runtime.operator.utils.OperatorUtils;
-
+import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
/**
* A {@code TransferableBlock} is a wrapper around {@link DataBlock} for transferring data using
* {@link org.apache.pinot.common.proto.Mailbox}.
*/
public class TransferableBlock implements Block {
private final DataBlock.Type _type;
+ @Nullable
private final DataSchema _dataSchema;
private final int _numRows;
private List<Object[]> _container;
private DataBlock _dataBlock;
private Map<Integer, String> _errCodeToExceptionMap;
+ @Nullable
+ private final MultiStageQueryStats _queryStats;
- public TransferableBlock(List<Object[]> container, DataSchema dataSchema, DataBlock.Type type) {
+ public TransferableBlock(List<Object[]> container, @Nullable DataSchema dataSchema, DataBlock.Type type) {
Review Comment:
We shouldn't use this constructor when schema doesn't exist
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/ReceivingMailbox.java:
##########
@@ -72,11 +81,53 @@ public String getId() {
return _id;
}
+ /**
+ * Offers a raw block into the mailbox within the timeout specified, returns whether the block is successfully added.
+ * If the block is not added, an error block is added to the mailbox.
+ * <p>
+ * Contrary to {@link #offer(TransferableBlock, long)}, the block may be an
+ * {@link TransferableBlock#isErrorBlock() error block}.
+ */
+ public ReceivingMailboxStatus offerRaw(ByteBuffer byteBuffer, long timeoutMs)
+ throws IOException {
+ TransferableBlock block;
+ long now = System.currentTimeMillis();
+ _stats.merge(StatKey.WAIT_CPU_TIME_MS, now - _lastArriveTime);
+ _lastArriveTime = now;
+ _stats.merge(StatKey.DESERIALIZED_BYTES, byteBuffer.remaining());
+ _stats.merge(StatKey.DESERIALIZED_MESSAGES, 1);
+
+ now = System.currentTimeMillis();
+ DataBlock dataBlock = DataBlockUtils.getDataBlock(byteBuffer);
+ _stats.merge(StatKey.DESERIALIZATION_TIME_MS, System.currentTimeMillis() - now);
+
+ if (dataBlock instanceof MetadataBlock) {
+ Map<Integer, String> exceptions = dataBlock.getExceptions();
+ if (exceptions.isEmpty()) {
+ block = TransferableBlockUtils.wrap(dataBlock);
+ } else {
+ setErrorBlock(TransferableBlockUtils.getErrorTransferableBlock(exceptions));
+ return ReceivingMailboxStatus.FIRST_ERROR;
Review Comment:
Also seems this message doesn't apply to in-memory mailbox, some comments would help explain it
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/MultiStageQueryStats.java:
##########
@@ -0,0 +1,668 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.plan;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.avro.util.ByteBufferInputStream;
+import org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.query.runtime.operator.BaseMailboxReceiveOperator;
+import org.apache.pinot.query.runtime.operator.LeafStageTransferableBlockOperator;
+import org.apache.pinot.query.runtime.operator.LiteralValueOperator;
+import org.apache.pinot.query.runtime.operator.MailboxSendOperator;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The stats of a given query.
+ * <p>
+ * For the same query, multiple instances of this class may exist. Each of them will have a partial view of the stats.
+ * Specifically, while the query is being executed, each operator will return its own partial view of the stats when
+ * EOS block is sent.
+ * <p>
+ * Simple operations with a single upstream, like filters or transforms, would just add their own information to the
+ * stats. More complex operations, like joins or receiving mailboxes, will merge the stats from all their upstreams and
+ * add their own stats.
+ * <p>
+ * The complete stats for the query are obtained in the execution root (usually the broker) by merging the partial
+ * views.
+ * <p>
+ * In order to reduce allocation, this class is mutable. Some operators may create their own stats, but most of them
+ * will receive a stats object from the upstream operator and modify it by adding their own stats and sometimes merging
+ * them with other upstream stats.
+ */
+public class MultiStageQueryStats {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MultiStageQueryStats.class);
+ private final int _currentStageId;
+ private final StageStats.Open _currentStats;
+ /**
+ * Known stats for stages whose id is higher than the current one.
+ * <p>
+ * A stage may not know all the stats whose id is higher than the current one, so this list may contain null values.
+ * It may also grow in size when different merge methods are called.
+ * <p>
+ * For example the stats of the left hand side of a join may know stats of stages 3 and 4 and the right side may know
+ * stats of stages 5. When merging the stats of the join, the stats of stages 5 will be added to this list.
+ *
+ * @see #mergeUpstream(List)
+ * @see #mergeUpstream(MultiStageQueryStats)
+ * @see #mergeInOrder(MultiStageQueryStats, MultiStageOperator.Type, StatMap)
+ */
+ private final ArrayList<StageStats.Closed> _closedStats;
+ private static final MultiStageOperator.Type[] ALL_TYPES = MultiStageOperator.Type.values();
+
+ private MultiStageQueryStats(int stageId) {
+ _currentStageId = stageId;
+ _currentStats = new StageStats.Open();
+ _closedStats = new ArrayList<>();
+ }
+
+ private static MultiStageQueryStats create(int stageId, MultiStageOperator.Type type, @Nullable StatMap<?> opStats) {
+ MultiStageQueryStats multiStageQueryStats = new MultiStageQueryStats(stageId);
+ multiStageQueryStats.getCurrentStats().addLastOperator(type, opStats);
+ return multiStageQueryStats;
+ }
+
+ public static MultiStageQueryStats emptyStats(int stageId) {
+ return new MultiStageQueryStats(stageId);
+ }
+
+ public static MultiStageQueryStats createLeaf(int stageId,
+ StatMap<LeafStageTransferableBlockOperator.StatKey> opStats) {
+ return create(stageId, MultiStageOperator.Type.LEAF, opStats);
+ }
+
+ public static MultiStageQueryStats createLiteral(int stageId, StatMap<LiteralValueOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.LITERAL, statMap);
+ }
+
+ public static MultiStageQueryStats createCancelledSend(int stageId,
+ StatMap<MailboxSendOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_SEND, statMap);
+ }
+
+ public static MultiStageQueryStats createReceive(int stageId, StatMap<BaseMailboxReceiveOperator.StatKey> stats) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_RECEIVE, stats);
+ }
+
+ public int getCurrentStageId() {
+ return _currentStageId;
+ }
+
+ /**
+ * Serialize the current stats in a way it is compatible with {@link #mergeUpstream(List)}.
+ * <p>
+ * The serialized stats are returned in a list where the index is the stage id. Stages downstream or not related to
+ * the current one will be null.
+ */
+ public List<ByteBuffer> serialize()
+ throws IOException {
+
+ ArrayList<ByteBuffer> serializedStats = new ArrayList<>(getMaxStageId());
+ for (int i = 0; i < _currentStageId; i++) {
+ serializedStats.add(null);
+ }
+
+ try (UnsynchronizedByteArrayOutputStream baos = new UnsynchronizedByteArrayOutputStream.Builder().get();
+ DataOutputStream output = new DataOutputStream(baos)) {
+
+ _currentStats.serialize(output);
+ ByteBuffer currentBuf = ByteBuffer.wrap(baos.toByteArray());
+
+ serializedStats.add(currentBuf);
+
+ for (StageStats.Closed closedStats : _closedStats) {
+ if (closedStats == null) {
+ serializedStats.add(null);
+ continue;
+ }
+ baos.reset();
+ closedStats.serialize(output);
+ ByteBuffer buf = ByteBuffer.wrap(baos.toByteArray());
+ serializedStats.add(buf);
+ }
+ }
+ Preconditions.checkState(serializedStats.size() == getMaxStageId() + 1,
+ "Serialized stats size is different from expected size. Expected %s, got %s",
+ getMaxStageId() + 1, serializedStats.size());
+ return serializedStats;
+ }
+
+ public StageStats.Open getCurrentStats() {
+ return _currentStats;
+ }
+
+ /**
+ * Returns the higher stage id known by this object.
+ */
+ public int getMaxStageId() {
+ return _currentStageId + _closedStats.size();
+ }
+
+ /**
+ * Get the stats of a stage whose id is higher than the current one.
+ * <p>
+ * This method returns null in case the stage id is unknown by this stage or no stats are stored for it.
+ */
+ @Nullable
+ public StageStats.Closed getUpstreamStageStats(int stageId) {
+ if (stageId <= _currentStageId) {
+ throw new IllegalArgumentException("Stage " + stageId + " cannot be upstream of current stage "
+ + _currentStageId);
+ }
+
+ int index = stageId - _currentStageId - 1;
+ if (index >= _closedStats.size()) {
+ return null;
+ }
+ return _closedStats.get(index);
+ }
+
+ public void mergeInOrder(MultiStageQueryStats otherStats, MultiStageOperator.Type type,
+ StatMap<?> statMap) {
+ Preconditions.checkArgument(_currentStageId == otherStats._currentStageId,
+ "Cannot merge stats from different stages (%s and %s)", _currentStageId, otherStats._currentStageId);
+ mergeUpstream(otherStats);
+ StageStats.Open currentStats = getCurrentStats();
+ currentStats.concat(otherStats.getCurrentStats());
+ currentStats.addLastOperator(type, statMap);
+ }
+
+ private void growUpToStage(int stageId) {
+ _closedStats.ensureCapacity(stageId - _currentStageId);
+ while (getMaxStageId() < stageId) {
+ _closedStats.add(null);
+ }
+ }
+
+ /**
+ * Merge upstream stats from another MultiStageQueryStats object into this one.
+ * <p>
+ * Only the stages whose id is higher than the current one are merged. The reason to do so is that upstream stats
+ * should be already closed while current stage may need some extra tuning.
+ * <p>
+ * For example set operations may need to merge the stats from all its upstreams before concatenating stats of the
+ * current stage.
+ */
+ public void mergeUpstream(MultiStageQueryStats otherStats) {
+ Preconditions.checkArgument(_currentStageId <= otherStats._currentStageId,
+ "Cannot merge stats from early stage %s into stats of later stage %s",
+ otherStats._currentStageId, _currentStageId);
+
+ growUpToStage(otherStats.getMaxStageId());
+
+ int currentDiff = otherStats._currentStageId - _currentStageId;
+ if (currentDiff > 0) {
+ StageStats.Closed close = otherStats._currentStats.close();
+ int selfIdx = currentDiff - 1;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ if (myStats == null) {
+ _closedStats.set(selfIdx, close);
+ } else {
+ myStats.merge(close);
+ }
+ }
+
+ for (int i = 0; i < otherStats._closedStats.size(); i++) {
+ StageStats.Closed otherStatsForStage = otherStats._closedStats.get(i);
+ if (otherStatsForStage == null) {
+ continue;
+ }
+ int selfIdx = i + currentDiff;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ try {
+ if (myStats == null) {
+ _closedStats.set(selfIdx, otherStatsForStage);
+ assert getUpstreamStageStats(i + otherStats._currentStageId + 1) == otherStatsForStage;
+ } else {
+ myStats.merge(otherStatsForStage);
+ }
+ } catch (IllegalArgumentException | IllegalStateException ex) {
+ LOGGER.warn("Error merging stats on stage " + i + ". Ignoring the new stats", ex);
+ }
+ }
+ }
+
+ public void mergeUpstream(List<ByteBuffer> otherStats) {
+ for (int i = 0; i <= _currentStageId && i < otherStats.size(); i++) {
+ if (otherStats.get(i) != null) {
+ throw new IllegalArgumentException("Cannot merge stats from early stage " + i + " into stats of "
+ + "later stage " + _currentStageId);
+ }
+ }
+ growUpToStage(otherStats.size() - 1);
+
+ for (int i = _currentStageId + 1; i < otherStats.size(); i++) {
+ ByteBuffer otherBuf = otherStats.get(i);
+ if (otherBuf != null) {
+ StageStats.Closed myStats = getUpstreamStageStats(i);
+ try (InputStream is = new ByteBufferInputStream(Collections.singletonList(otherBuf));
+ DataInputStream dis = new DataInputStream(is)) {
+ if (myStats == null) {
+ StageStats.Closed deserialized = StageStats.Closed.deserialize(dis);
+ _closedStats.set(i - _currentStageId - 1, deserialized);
+ assert getUpstreamStageStats(i) == deserialized;
+ } else {
+ myStats.merge(dis);
+ }
+ } catch (IOException ex) {
+ boolean assertOn = false;
+ // *assigns* true if assertions are on.
+ //CHECKSTYLE:OFF
+ assert assertOn = true;
+ if (assertOn) {
+ throw new UncheckedIOException("Error deserializing stats on stage " + i, ex);
+ }
+ //CHECKSTYLE:ON
Review Comment:
I feel it is safer to have consistent behavior in test and production. In the test we can check for the stats value to verify it is correctly handled.
I don't have strong opinion, so either way is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576190502
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -276,6 +264,161 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
return brokerResponse;
}
+ private void fillOldBrokerResponseStats(BrokerResponseNativeV2 brokerResponse,
+ List<MultiStageQueryStats.StageStats.Closed> queryStats, DispatchableSubPlan dispatchableSubPlan) {
+ for (int i = 0; i < queryStats.size(); i++) {
+ MultiStageQueryStats.StageStats.Closed stageStats = queryStats.get(i);
+ if (stageStats == null) {
+ brokerResponse.addStageStats(JsonUtils.newObjectNode());
+ } else {
+ stageStats.forEach((type, stats) -> type.mergeInto(brokerResponse, stats));
+
+ DispatchablePlanFragment dispatchablePlanFragment = dispatchableSubPlan.getQueryStageList().get(i);
+ MultiStageStatsTreeBuilder treeBuilder = new MultiStageStatsTreeBuilder(stageStats);
+ PlanNode fragmentRoot = dispatchablePlanFragment.getPlanFragment().getFragmentRoot();
+ JsonNode node = fragmentRoot.visit(treeBuilder, null);
+ brokerResponse.addStageStats(node);
+ }
+ }
+ }
+
+ public static class MultiStageStatsTreeBuilder implements PlanNodeVisitor<JsonNode, Void> {
+ private final MultiStageQueryStats.StageStats.Closed _stageStats;
+ private int _index;
+ private static final String CHILDREN_KEY = "children";
+
+ public MultiStageStatsTreeBuilder(MultiStageQueryStats.StageStats.Closed stageStats) {
+ _stageStats = stageStats;
+ _index = stageStats.getLastOperatorIndex();
+ }
+
+ private ObjectNode selfNode(MultiStageOperator.Type type) {
+ ObjectNode json = JsonUtils.newObjectNode();
+ json.put("type", type.toString());
+ Iterator<Map.Entry<String, JsonNode>> statsIt = _stageStats.getOperatorStats(_index).asJson().fields();
+ while (statsIt.hasNext()) {
+ Map.Entry<String, JsonNode> entry = statsIt.next();
+ json.set(entry.getKey(), entry.getValue());
+ }
+ return json;
+ }
+
+ private JsonNode recursiveCase(AbstractPlanNode node, MultiStageOperator.Type expectedType) {
+ MultiStageOperator.Type type = _stageStats.getOperatorType(_index);
+ if (type != expectedType) {
Review Comment:
No, this is very rare. Mostly only happens on the stage 0 and in leaf stages.
In stage 0 always have two nodes. One is a transform IIRC and the other is a receive node and we only have stats for the latter.
In leaf stage, we may have several nodes, but the leaf stage compiles them into a different plan (as explained above).
In can add some comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12704:
URL: https://github.com/apache/pinot/pull/12704#issuecomment-2075217016
> Can you share the race condition you find during debugging? I want to learn the context
If found it in https://github.com/apache/pinot/pull/12704/commits/dcb197cd5469824f30aeaaab0c54a5ef40246ec2, whose tests are still in https://github.com/apache/pinot/actions/runs/8659408355/job/23754450579. But previous commits failed in other cases.
Basically it looks like there is a concurrency issue if we remove the Synchronized map in `StatMap`. Specifically, I was able to reproduce it by running one of the test that failed 1000 times in a loop. After a couple of minutes it failed. In that case, it always failed in the stats of the leaf operator. In that operator I modify the StatMap in the constructor.
I have to say that in that version of the code the StatMap was initialized in the super class, so maybe the constructor with final attribute was not applying by then. Now that it applies, it may be the case that we don't need to actually synchronize, but I wouldn't feel very comfortable relaying on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582728941
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -798,7 +798,6 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
_brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_NUM_GROUPS_LIMIT_REACHED,
1);
}
- brokerResponse.setPartialResult(isPartialResult(brokerResponse));
Review Comment:
removed in commit 2d9649147559782e40fab44542dc93c0bb999249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12704:
URL: https://github.com/apache/pinot/pull/12704#issuecomment-2051326915
There seems to be an randomly detected by integration tests where we end up timing out. I'm trying to find why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1588535547
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
Given the above chart, do we need to have the special handling here?
This is a non-blocking comment, we may iterate on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565384127
##########
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/ExecutionStatsAggregator.java:
##########
@@ -364,25 +357,6 @@ public void setStats(@Nullable String rawTableName, BrokerResponseNative brokerR
}
}
- public void setStageLevelStats(@Nullable String rawTableName, BrokerResponseStats brokerResponseStats,
- @Nullable BrokerMetrics brokerMetrics) {
- if (_enableTrace) {
- setStats(rawTableName, brokerResponseStats, brokerMetrics);
- brokerResponseStats.setOperatorStats(_operatorStats);
- }
-
- brokerResponseStats.setNumBlocks(_numBlocks);
- brokerResponseStats.setNumRows(_numRows);
- brokerResponseStats.setMaxRowsInJoinReached(_maxRowsInJoinReached);
- brokerResponseStats.setNumGroupsLimitReached(_numGroupsLimitReached);
- brokerResponseStats.setStageExecutionTimeMs(_stageExecutionTimeMs);
- brokerResponseStats.setStageExecutionUnit(_stageExecutionUnit);
- brokerResponseStats.setTableNames(new ArrayList<>(_tableNames));
- if (_stageExecStartTimeMs >= 0 && _stageExecEndTimeMs >= 0) {
- brokerResponseStats.setStageExecWallTimeMs(_stageExecEndTimeMs - _stageExecStartTimeMs);
- }
- }
-
Review Comment:
This method is not used anymore. Instead we merge stats in `MultiStageOperator.Type.mergeInto`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565227184
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlock.java:
##########
@@ -80,8 +80,6 @@ byte[] toBytes()
@Nullable
RoaringBitmap getNullRowIds(int colId);
- DataBlock toMetadataOnlyDataTable();
-
DataBlock toDataOnlyDataTable();
Review Comment:
Can we also cleanup this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1566574337
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/MultiStageQueryStats.java:
##########
@@ -0,0 +1,668 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.plan;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.avro.util.ByteBufferInputStream;
+import org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.query.runtime.operator.BaseMailboxReceiveOperator;
+import org.apache.pinot.query.runtime.operator.LeafStageTransferableBlockOperator;
+import org.apache.pinot.query.runtime.operator.LiteralValueOperator;
+import org.apache.pinot.query.runtime.operator.MailboxSendOperator;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The stats of a given query.
+ * <p>
+ * For the same query, multiple instances of this class may exist. Each of them will have a partial view of the stats.
+ * Specifically, while the query is being executed, each operator will return its own partial view of the stats when
+ * EOS block is sent.
+ * <p>
+ * Simple operations with a single upstream, like filters or transforms, would just add their own information to the
+ * stats. More complex operations, like joins or receiving mailboxes, will merge the stats from all their upstreams and
+ * add their own stats.
+ * <p>
+ * The complete stats for the query are obtained in the execution root (usually the broker) by merging the partial
+ * views.
+ * <p>
+ * In order to reduce allocation, this class is mutable. Some operators may create their own stats, but most of them
+ * will receive a stats object from the upstream operator and modify it by adding their own stats and sometimes merging
+ * them with other upstream stats.
+ */
+public class MultiStageQueryStats {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MultiStageQueryStats.class);
+ private final int _currentStageId;
+ private final StageStats.Open _currentStats;
+ /**
+ * Known stats for stages whose id is higher than the current one.
+ * <p>
+ * A stage may not know all the stats whose id is higher than the current one, so this list may contain null values.
+ * It may also grow in size when different merge methods are called.
+ * <p>
+ * For example the stats of the left hand side of a join may know stats of stages 3 and 4 and the right side may know
+ * stats of stages 5. When merging the stats of the join, the stats of stages 5 will be added to this list.
+ *
+ * @see #mergeUpstream(List)
+ * @see #mergeUpstream(MultiStageQueryStats)
+ * @see #mergeInOrder(MultiStageQueryStats, MultiStageOperator.Type, StatMap)
+ */
+ private final ArrayList<StageStats.Closed> _closedStats;
+ private static final MultiStageOperator.Type[] ALL_TYPES = MultiStageOperator.Type.values();
+
+ private MultiStageQueryStats(int stageId) {
+ _currentStageId = stageId;
+ _currentStats = new StageStats.Open();
+ _closedStats = new ArrayList<>();
+ }
+
+ private static MultiStageQueryStats create(int stageId, MultiStageOperator.Type type, @Nullable StatMap<?> opStats) {
+ MultiStageQueryStats multiStageQueryStats = new MultiStageQueryStats(stageId);
+ multiStageQueryStats.getCurrentStats().addLastOperator(type, opStats);
+ return multiStageQueryStats;
+ }
+
+ public static MultiStageQueryStats emptyStats(int stageId) {
+ return new MultiStageQueryStats(stageId);
+ }
+
+ public static MultiStageQueryStats createLeaf(int stageId,
+ StatMap<LeafStageTransferableBlockOperator.StatKey> opStats) {
+ return create(stageId, MultiStageOperator.Type.LEAF, opStats);
+ }
+
+ public static MultiStageQueryStats createLiteral(int stageId, StatMap<LiteralValueOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.LITERAL, statMap);
+ }
+
+ public static MultiStageQueryStats createCancelledSend(int stageId,
+ StatMap<MailboxSendOperator.StatKey> statMap) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_SEND, statMap);
+ }
+
+ public static MultiStageQueryStats createReceive(int stageId, StatMap<BaseMailboxReceiveOperator.StatKey> stats) {
+ return create(stageId, MultiStageOperator.Type.MAILBOX_RECEIVE, stats);
+ }
+
+ public int getCurrentStageId() {
+ return _currentStageId;
+ }
+
+ /**
+ * Serialize the current stats in a way it is compatible with {@link #mergeUpstream(List)}.
+ * <p>
+ * The serialized stats are returned in a list where the index is the stage id. Stages downstream or not related to
+ * the current one will be null.
+ */
+ public List<ByteBuffer> serialize()
+ throws IOException {
+
+ ArrayList<ByteBuffer> serializedStats = new ArrayList<>(getMaxStageId());
+ for (int i = 0; i < _currentStageId; i++) {
+ serializedStats.add(null);
+ }
+
+ try (UnsynchronizedByteArrayOutputStream baos = new UnsynchronizedByteArrayOutputStream.Builder().get();
+ DataOutputStream output = new DataOutputStream(baos)) {
+
+ _currentStats.serialize(output);
+ ByteBuffer currentBuf = ByteBuffer.wrap(baos.toByteArray());
+
+ serializedStats.add(currentBuf);
+
+ for (StageStats.Closed closedStats : _closedStats) {
+ if (closedStats == null) {
+ serializedStats.add(null);
+ continue;
+ }
+ baos.reset();
+ closedStats.serialize(output);
+ ByteBuffer buf = ByteBuffer.wrap(baos.toByteArray());
+ serializedStats.add(buf);
+ }
+ }
+ Preconditions.checkState(serializedStats.size() == getMaxStageId() + 1,
+ "Serialized stats size is different from expected size. Expected %s, got %s",
+ getMaxStageId() + 1, serializedStats.size());
+ return serializedStats;
+ }
+
+ public StageStats.Open getCurrentStats() {
+ return _currentStats;
+ }
+
+ /**
+ * Returns the higher stage id known by this object.
+ */
+ public int getMaxStageId() {
+ return _currentStageId + _closedStats.size();
+ }
+
+ /**
+ * Get the stats of a stage whose id is higher than the current one.
+ * <p>
+ * This method returns null in case the stage id is unknown by this stage or no stats are stored for it.
+ */
+ @Nullable
+ public StageStats.Closed getUpstreamStageStats(int stageId) {
+ if (stageId <= _currentStageId) {
+ throw new IllegalArgumentException("Stage " + stageId + " cannot be upstream of current stage "
+ + _currentStageId);
+ }
+
+ int index = stageId - _currentStageId - 1;
+ if (index >= _closedStats.size()) {
+ return null;
+ }
+ return _closedStats.get(index);
+ }
+
+ public void mergeInOrder(MultiStageQueryStats otherStats, MultiStageOperator.Type type,
+ StatMap<?> statMap) {
+ Preconditions.checkArgument(_currentStageId == otherStats._currentStageId,
+ "Cannot merge stats from different stages (%s and %s)", _currentStageId, otherStats._currentStageId);
+ mergeUpstream(otherStats);
+ StageStats.Open currentStats = getCurrentStats();
+ currentStats.concat(otherStats.getCurrentStats());
+ currentStats.addLastOperator(type, statMap);
+ }
+
+ private void growUpToStage(int stageId) {
+ _closedStats.ensureCapacity(stageId - _currentStageId);
+ while (getMaxStageId() < stageId) {
+ _closedStats.add(null);
+ }
+ }
+
+ /**
+ * Merge upstream stats from another MultiStageQueryStats object into this one.
+ * <p>
+ * Only the stages whose id is higher than the current one are merged. The reason to do so is that upstream stats
+ * should be already closed while current stage may need some extra tuning.
+ * <p>
+ * For example set operations may need to merge the stats from all its upstreams before concatenating stats of the
+ * current stage.
+ */
+ public void mergeUpstream(MultiStageQueryStats otherStats) {
+ Preconditions.checkArgument(_currentStageId <= otherStats._currentStageId,
+ "Cannot merge stats from early stage %s into stats of later stage %s",
+ otherStats._currentStageId, _currentStageId);
+
+ growUpToStage(otherStats.getMaxStageId());
+
+ int currentDiff = otherStats._currentStageId - _currentStageId;
+ if (currentDiff > 0) {
+ StageStats.Closed close = otherStats._currentStats.close();
+ int selfIdx = currentDiff - 1;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ if (myStats == null) {
+ _closedStats.set(selfIdx, close);
+ } else {
+ myStats.merge(close);
+ }
+ }
+
+ for (int i = 0; i < otherStats._closedStats.size(); i++) {
+ StageStats.Closed otherStatsForStage = otherStats._closedStats.get(i);
+ if (otherStatsForStage == null) {
+ continue;
+ }
+ int selfIdx = i + currentDiff;
+ StageStats.Closed myStats = _closedStats.get(selfIdx);
+ try {
+ if (myStats == null) {
+ _closedStats.set(selfIdx, otherStatsForStage);
+ assert getUpstreamStageStats(i + otherStats._currentStageId + 1) == otherStatsForStage;
+ } else {
+ myStats.merge(otherStatsForStage);
+ }
+ } catch (IllegalArgumentException | IllegalStateException ex) {
+ LOGGER.warn("Error merging stats on stage " + i + ". Ignoring the new stats", ex);
+ }
+ }
+ }
+
+ public void mergeUpstream(List<ByteBuffer> otherStats) {
+ for (int i = 0; i <= _currentStageId && i < otherStats.size(); i++) {
+ if (otherStats.get(i) != null) {
+ throw new IllegalArgumentException("Cannot merge stats from early stage " + i + " into stats of "
+ + "later stage " + _currentStageId);
+ }
+ }
+ growUpToStage(otherStats.size() - 1);
+
+ for (int i = _currentStageId + 1; i < otherStats.size(); i++) {
+ ByteBuffer otherBuf = otherStats.get(i);
+ if (otherBuf != null) {
+ StageStats.Closed myStats = getUpstreamStageStats(i);
+ try (InputStream is = new ByteBufferInputStream(Collections.singletonList(otherBuf));
+ DataInputStream dis = new DataInputStream(is)) {
+ if (myStats == null) {
+ StageStats.Closed deserialized = StageStats.Closed.deserialize(dis);
+ _closedStats.set(i - _currentStageId - 1, deserialized);
+ assert getUpstreamStageStats(i) == deserialized;
+ } else {
+ myStats.merge(dis);
+ }
+ } catch (IOException ex) {
+ boolean assertOn = false;
+ // *assigns* true if assertions are on.
+ //CHECKSTYLE:OFF
+ assert assertOn = true;
+ if (assertOn) {
+ throw new UncheckedIOException("Error deserializing stats on stage " + i, ex);
+ }
+ //CHECKSTYLE:ON
Review Comment:
Is the intention to fail it during testing?
Having different behavior with/without assert on seems weird to me though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582896361
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -135,14 +160,128 @@ protected TransferableBlock getNextBlock()
}
}
- private TransferableBlock constructMetadataBlock() {
- // All data blocks have been returned. Record the stats and return EOS.
- Map<String, String> executionStats = _executionStats;
+ private void mergeExecutionStats(@Nullable Map<String, String> executionStats) {
if (executionStats != null) {
- OperatorStats operatorStats = _opChainStats.getOperatorStats(_context, getOperatorId());
- operatorStats.recordExecutionStats(executionStats);
+ for (Map.Entry<String, String> entry : executionStats.entrySet()) {
+ DataTable.MetadataKey key = DataTable.MetadataKey.getByName(entry.getKey());
+ if (key == null) {
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ continue;
+ }
+ switch (key) {
+ case UNKNOWN:
+ LOGGER.debug("Skipping unknown execution stat: {}", entry.getKey());
+ break;
+ case TABLE:
Review Comment:
I think this stat is very useful to indicate the table in the json stats returned to the client. Specifically, in the json that correspond to the leaf stage. Right now it is very difficult to map from stages to the original SQL and this is one of the few hints we have to understand how stages are being generated. See for example the stat map in the description of this PR. Without the `table` attribute it is very difficult to map from these stats to the original SQL and therefore understand how to improve your query.
Said that, we can do that without adding _table_ as an stat. We can add add the _table_ json field on `MultiStageBrokerRequestHandler` when the json is being created. I've pushed a change that does that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583030085
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
> Is this the only place where v1 metadata block is used?
Yes. It is used only here and in the tests.
> Does it work if we ignore the version and always return new MetadataBlock(byteBuffer);?
Yes.
> If the error handling is different, then sending v2 metadata block to old server will cause backward incompatibility right?
It is no different.
Remember that the main reason to have this class is to be able to run tests with it. The fact that it is useful here is just an extra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz merged PR #12704:
URL: https://github.com/apache/pinot/pull/12704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1587305238
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
+ @Nullable
+ private List<ByteBuffer> _statsByStage;
- public enum MetadataBlockType {
- /**
- * Indicates that this block is the final block to be sent
- * (End Of Stream) as part of an operator chain computation.
- */
- EOS,
+ private final MetadataBlockType _type;
- /**
- * An {@code ERROR} metadata block indicates that there was
- * some error during computation. To retrieve the error that
- * occurred, use {@link MetadataBlock#getExceptions()}
- */
- ERROR
+ public MetadataBlock(MetadataBlockType type) {
+ this(type, Collections.emptyList());
}
- /**
- * Used to serialize the contents of the metadata block conveniently and in
- * a backwards compatible way. Use JSON because the performance of metadata block
- * SerDe should not be a bottleneck.
- */
- @JsonIgnoreProperties(ignoreUnknown = true)
- @VisibleForTesting
- static class Contents {
-
- private String _type;
- private Map<String, String> _stats;
-
- @JsonCreator
- public Contents(@JsonProperty("type") String type, @JsonProperty("stats") Map<String, String> stats) {
- _type = type;
- _stats = stats;
- }
-
- @JsonCreator
- public Contents() {
- this(null, new HashMap<>());
- }
-
- public String getType() {
- return _type;
- }
+ public MetadataBlock(MetadataBlockType type, List<ByteBuffer> statsByStage) {
+ super(0, null, new String[0], new byte[]{(byte) (type.ordinal() & 0xFF)}, new byte[0]);
+ _type = type;
+ _statsByStage = statsByStage;
+ }
- public void setType(String type) {
- _type = type;
+ MetadataBlock(ByteBuffer byteBuffer)
+ throws IOException {
+ super(byteBuffer);
+ // Remember: At this point deserializeMetadata is already being called.
+ if (_fixedSizeDataBytes == null) {
+ if (_errCodeToExceptionMap.isEmpty()) {
+ _type = MetadataBlockType.EOS;
+ } else {
+ _type = MetadataBlockType.ERROR;
+ }
+ } else {
+ _type = MetadataBlockType.values()[_fixedSizeDataBytes[0]];
}
+ }
- public Map<String, String> getStats() {
- return _stats;
+ @Override
+ protected void serializeMetadata(DataOutputStream output)
+ throws IOException {
+ if (_statsByStage == null) {
+ output.writeInt(0);
+ return;
}
-
- public void setStats(Map<String, String> stats) {
- _stats = stats;
+ int size = _statsByStage.size();
+ output.writeInt(size);
+ if (size > 0) {
+ byte[] bytes = new byte[4096];
+ for (ByteBuffer stat : _statsByStage) {
+ if (stat == null) {
+ output.writeBoolean(false);
+ } else {
+ output.writeBoolean(true);
+ output.writeInt(stat.remaining());
+ ByteBuffer duplicate = stat.duplicate();
+ while (duplicate.hasRemaining()) {
+ int length = Math.min(duplicate.remaining(), bytes.length);
+ duplicate.get(bytes, 0, length);
+ output.write(bytes, 0, length);
+ }
+ }
+ }
}
}
- private final Contents _contents;
-
- public MetadataBlock(MetadataBlockType type) {
- this(type, new HashMap<>());
- }
-
- public MetadataBlock(MetadataBlockType type, Map<String, String> stats) {
- super(0, null, new String[0], new byte[]{0}, toContents(new Contents(type.name(), stats)));
- _contents = new Contents(type.name(), stats);
- }
-
- private static byte[] toContents(Contents type) {
- try {
- return JSON.writeValueAsBytes(type);
- } catch (JsonProcessingException e) {
- throw new RuntimeException(e);
+ public static MetadataBlock deserialize(ByteBuffer byteBuffer, int version)
+ throws IOException {
+ switch (version) {
+ case 1: {
+ V1MetadataBlock decoded = new V1MetadataBlock(byteBuffer);
Review Comment:
> Let's list down the behavior on reading v1 bytes as v2 and also reading v2 bytes as v1 so that user knows what to expect when upgrading.
The compatibility matrix is verified by tests. I've just renamed the name of the tests and I'm adding here the test table to show which tests verifies each scenario:
| input version | input type | input content | possible? | read in version | output type | output content | tested by |
|:-------------:|:----------:|:-------------:|:---------:|:---------------:|:-----------:|:--------------:|:---------------------------------------------:|
| v1 | eos | empty | yes | v2 | eos | empty | v1EosWithStatsIsDecodedAsV2EosWithoutStats |
| v1 | eos | not empty | yes | v2 | eos | empty | v1EosWithoutStatsIsDecodedAsV2EosWithoutStats |
| v1 | error | empty | no | - | - | - | - |
| v1 | error | not empty | yes | v2 | error | same as input | v1ErrorIsDecodedAsV2ErrorWithSameExceptions |
| v2 | eos | empty | yes | v1 | eos | empty | v2EosWithoutStatsIsReadInV1AsEosWithoutStats |
| v2 | eos | not empty | yes | v1 | eos | empty | v2EosWithStatsIsReadInV1AsEosWithoutStats |
| v2 | error | empty | no | - | - | - | - |
| v2 | error | not empty | yes | v1 | error | same as input | v2ErrorIsReadInV1AsErrorWithSameExceptions |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1576220503
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlock.java:
##########
@@ -19,44 +19,50 @@
package org.apache.pinot.query.runtime.blocks;
import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import javax.annotation.Nullable;
import org.apache.pinot.common.datablock.ColumnarDataBlock;
import org.apache.pinot.common.datablock.DataBlock;
import org.apache.pinot.common.datablock.DataBlockUtils;
import org.apache.pinot.common.datablock.MetadataBlock;
import org.apache.pinot.common.datablock.RowDataBlock;
-import org.apache.pinot.common.response.ProcessingException;
import org.apache.pinot.common.utils.DataSchema;
import org.apache.pinot.core.common.Block;
import org.apache.pinot.core.common.datablock.DataBlockBuilder;
import org.apache.pinot.core.util.DataBlockExtractUtils;
-import org.apache.pinot.query.runtime.operator.OperatorStats;
-import org.apache.pinot.query.runtime.operator.utils.OperatorUtils;
-
+import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
/**
* A {@code TransferableBlock} is a wrapper around {@link DataBlock} for transferring data using
* {@link org.apache.pinot.common.proto.Mailbox}.
*/
public class TransferableBlock implements Block {
private final DataBlock.Type _type;
+ @Nullable
private final DataSchema _dataSchema;
private final int _numRows;
private List<Object[]> _container;
private DataBlock _dataBlock;
private Map<Integer, String> _errCodeToExceptionMap;
+ @Nullable
+ private final MultiStageQueryStats _queryStats;
- public TransferableBlock(List<Object[]> container, DataSchema dataSchema, DataBlock.Type type) {
+ public TransferableBlock(List<Object[]> container, @Nullable DataSchema dataSchema, DataBlock.Type type) {
Review Comment:
I don't remember where, but I've found a path in the code this method could be called with a null data schema.
I may also be wrong, maybe I've just found that call during my debugs in a moment there was an error in my code. I guess I can remove the annotation and just add it again in case I find the problematic call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1580608897
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNativeV2.java:
##########
@@ -77,20 +100,301 @@ public static BrokerResponseNativeV2 empty() {
return new BrokerResponseNativeV2();
}
- public static BrokerResponseNativeV2 fromJsonString(String jsonString)
- throws IOException {
- return JsonUtils.stringToObject(jsonString, BrokerResponseNativeV2.class);
+ public void addStageStats(JsonNode stageStats) {
+ ObjectNode node = JsonUtils.newObjectNode();
+ node.put("stage", _stageIdStats.size());
+ node.set("stats", stageStats);
+ _stageIdStats.add(node);
+ }
+
+ @JsonProperty
+ public List<JsonNode> getStageStats() {
+ return _stageIdStats;
+ }
+
+ @JsonProperty
+ public long getMaxRows() {
+ return _maxRows;
+ }
+
+ public void mergeMaxRows(long maxRows) {
+ _maxRows = Math.max(_maxRows, maxRows);
+ }
+
+ @Override
+ public void setTimeUsedMs(long timeUsedMs) {
+ _serverStats.merge(DataTable.MetadataKey.TIME_USED_MS, timeUsedMs);
+ }
+
+ @Override
+ public long getNumDocsScanned() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_DOCS_SCANNED);
+ }
+
+ @Override
+ public long getNumEntriesScannedInFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_IN_FILTER);
+ }
+
+ @Override
+ public long getNumEntriesScannedPostFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_POST_FILTER);
+ }
+
+ @Override
+ public long getNumSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getMinConsumingFreshnessTimeMs() {
+ return _serverStats.getLong(DataTable.MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS);
+ }
+
+ @Override
+ public long getTotalDocs() {
+ return _serverStats.getLong(DataTable.MetadataKey.TOTAL_DOCS);
+ }
+
+ @Override
+ public boolean isNumGroupsLimitReached() {
+ return _serverStats.getBoolean(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED);
+ }
+
+ public void mergeNumGroupsLimitReached(boolean numGroupsLimitReached) {
+ _serverStats.merge(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED, numGroupsLimitReached);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByServer() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_SERVER);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedInvalid() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_INVALID);
}
- public void addStageStat(Integer stageId, BrokerResponseStats brokerResponseStats) {
- // StageExecutionWallTime will always be there, other stats are optional such as OperatorStats
- if (brokerResponseStats.getStageExecWallTimeMs() != -1) {
- _stageIdStats.put(stageId, brokerResponseStats);
+ @Override
+ public long getNumSegmentsPrunedByLimit() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_LIMIT);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByValue() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_VALUE);
+ }
+
+ @Override
+ public long getExplainPlanNumEmptyFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getExplainPlanNumMatchAllFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getOfflineTotalCpuTimeNs() {
+ return getOfflineThreadCpuTimeNs() + getOfflineSystemActivitiesCpuTimeNs()
+ + getOfflineResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public long getRealtimeTotalCpuTimeNs() {
+ return getRealtimeThreadCpuTimeNs() + getRealtimeSystemActivitiesCpuTimeNs()
+ + getRealtimeResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public void setExceptions(List<ProcessingException> exceptions) {
+ for (ProcessingException exception : exceptions) {
+ _processingExceptions.add(new QueryProcessingException(exception.getErrorCode(), exception.getMessage()));
}
}
- @JsonProperty("stageStats")
- public Map<Integer, BrokerResponseStats> getStageIdStats() {
- return _stageIdStats;
+ public void addToExceptions(QueryProcessingException processingException) {
+ _processingExceptions.add(processingException);
+ }
+
+ @Override
+ public int getNumServersQueried() {
+ return _numServersQueried;
+ }
+
+ @Override
+ public void setNumServersQueried(int numServersQueried) {
+ _numServersQueried = numServersQueried;
+ }
+
+ @Override
+ public int getNumServersResponded() {
+ return _numServersResponded;
+ }
+
+ @Override
+ public void setNumServersResponded(int numServersResponded) {
+ _numServersResponded = numServersResponded;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public boolean isMaxRowsInJoinReached() {
+ return _maxRowsInJoinReached;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public void mergeMaxRowsInJoinReached(boolean maxRowsInJoinReached) {
+ _maxRowsInJoinReached |= maxRowsInJoinReached;
+ }
+
+ @Override
+ public int getExceptionsSize() {
+ return _processingExceptions.size();
+ }
+
+ @Override
+ public void setResultTable(ResultTable resultTable) {
+ _resultTable = resultTable;
+ }
+
+ @Override
+ public ResultTable getResultTable() {
+ return _resultTable;
+ }
+
+ @Override
+ public List<QueryProcessingException> getProcessingExceptions() {
+ return List.of();
+ }
+
+ @Override
+ public int getNumRowsResultSet() {
+ return 0;
+ }
+
+ @Override
+ public long getOfflineThreadCpuTimeNs() {
+ return _offlineThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeThreadCpuTimeNs() {
+ return _realtimeThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineSystemActivitiesCpuTimeNs() {
+ return _offlineSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeSystemActivitiesCpuTimeNs() {
+ return _realtimeSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineResponseSerializationCpuTimeNs() {
+ return _offlineResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeResponseSerializationCpuTimeNs() {
+ return _realtimeResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByBroker() {
+ return _numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public void setNumSegmentsPrunedByBroker(long numSegmentsPrunedByBroker) {
+ _numSegmentsPrunedByBroker = numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public String getRequestId() {
+ return _requestId;
+ }
+
+ @Override
+ public void setRequestId(String requestId) {
+ _requestId = requestId;
+ }
+
+ @Override
+ public String getBrokerId() {
+ return _brokerId;
+ }
+
+ @Override
+ public void setBrokerId(String requestId) {
+ _brokerId = requestId;
+ }
+
+ @Override
+ public long getBrokerReduceTimeMs() {
+ return _brokerReduceTimeMs;
+ }
+
+ @Override
+ public void setBrokerReduceTimeMs(long brokerReduceTimeMs) {
+ _brokerReduceTimeMs = brokerReduceTimeMs;
+ }
+
+ @JsonProperty(access = JsonProperty.Access.READ_ONLY)
+ @Override
+ public boolean isPartialResult() {
+ return isNumGroupsLimitReached() || getExceptionsSize() > 0 || isMaxRowsInJoinReached();
+ }
+
+ public void addServerStats(StatMap<DataTable.MetadataKey> serverStats) {
+ // Set execution statistics.
+ _serverStats.merge(serverStats);
+
+ long threadCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.THREAD_CPU_TIME_NS);
+ long systemActivitiesCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.SYSTEM_ACTIVITIES_CPU_TIME_NS);
+ long responseSerializationCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.RESPONSE_SER_CPU_TIME_NS);
+
+ String tableName = serverStats.getString(DataTable.MetadataKey.TABLE);
+ if (tableName != null) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
+
+ if (tableType == TableType.OFFLINE) {
+ _offlineThreadCpuTimeNs += threadCpuTimeNs;
+ _offlineSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _offlineResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ } else {
+ _realtimeThreadCpuTimeNs += threadCpuTimeNs;
+ _realtimeSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _realtimeResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ }
Review Comment:
These are not V1 or V2 stats. They _broker_ stats. They are populated in the broker code.
V1 collects `threadCpuTimeNs`, `systemActivitiesCpuTimeNs` and `responseSerializationCpuTimeNs` but doesn't care whether they are realtime or offline. That information reaches the broker and depending on the table type, it stores the value in the realtime or offline version and that is what is returned to the customer.
In the code we have in master right now these stats are not being populated when the query is executed in V2. Specifically in `ExecutionStatsAggregator`, the code:
```java
TableType tableType = null;
String instanceName = null;
if (routingInstance != null) {
tableType = routingInstance.getTableType();
instanceName = routingInstance.getShortName();;
} else if (tableName != null) {
tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
instanceName = tableName;
} else {
tableType = null;
instanceName = null;
}
```
Ends up setting `tableType` to null, these stats are never calculated.
I guess we can remove these lines and keep the current behavior. We can try to add support for this in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565307236
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##########
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER = LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
Review Comment:
I think it is cleaner to read a different version and explicitly write in the code what we do when a block of an older version is received. Instead of crossing our fingers knowing that v1 uses varsize while v2 uses metadata, now in `MetadataBlock.deserialize` we an explicit code saying what we are doing.
I'm open to increase the version of the other blocks as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565358229
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlockUtils.java:
##########
@@ -62,13 +67,13 @@ public static MetadataBlock getEndOfStreamDataBlock() {
return new MetadataBlock(MetadataBlock.MetadataBlockType.EOS);
}
- public static MetadataBlock getEndOfStreamDataBlock(Map<String, String> stats) {
- return new MetadataBlock(MetadataBlock.MetadataBlockType.EOS, stats);
+ public static int readVersionType(ByteBuffer byteBuffer) {
+ return byteBuffer.getInt();
Review Comment:
This method is here to be called from a test, but can be useful in other cases in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565387972
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcSendingMailbox.java:
##########
@@ -64,25 +69,31 @@ public void send(TransferableBlock block)
return;
}
if (LOGGER.isDebugEnabled()) {
- LOGGER.debug("==[GRPC SEND]== sending data to: " + _id);
+ LOGGER.debug("==[GRPC SEND]== sending message " + block + " to: " + _id);
}
if (_contentObserver == null) {
_contentObserver = getContentObserver();
}
_contentObserver.onNext(toMailboxContent(block));
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("==[GRPC SEND]== message " + block + " sent to: " + _id);
+ }
Review Comment:
I've found this logs very useful to debug a concurrency issue in this PR. All these logs are either not in the hot path or warded with `isDebugEnabled` in order to do not allocate if disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565395943
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentObserver.java:
##########
@@ -62,22 +58,9 @@ public void onNext(MailboxContent mailboxContent) {
_mailbox = _mailboxService.getReceivingMailbox(mailboxId);
}
try {
- TransferableBlock block;
- DataBlock dataBlock = DataBlockUtils.getDataBlock(mailboxContent.getPayload().asReadOnlyByteBuffer());
- if (dataBlock instanceof MetadataBlock) {
- Map<Integer, String> exceptions = dataBlock.getExceptions();
- if (exceptions.isEmpty()) {
- block = TransferableBlockUtils.getEndOfStreamTransferableBlock(((MetadataBlock) dataBlock).getStats());
- } else {
- _mailbox.setErrorBlock(TransferableBlockUtils.getErrorTransferableBlock(exceptions));
- return;
- }
- } else {
- block = new TransferableBlock(dataBlock);
- }
-
Review Comment:
This code has been moved to `ReceivingMailbox.offerRaw` in order to be able to capture the stats there. Also, I think the responsibility of doing this should be in ReceivingMailbox and not in the pipeline itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565397287
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentObserver.java:
##########
@@ -88,6 +71,8 @@ public void onNext(MailboxContent mailboxContent) {
LOGGER.warn("Mailbox: {} already cancelled from upstream", mailboxId);
cancelStream();
break;
+ case FIRST_ERROR:
+ return;
Review Comment:
I added this enum to keep the same behavior in this class, which just aborted the execution when exceptions are found in a message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1578077425
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNativeV2.java:
##########
@@ -77,20 +100,301 @@ public static BrokerResponseNativeV2 empty() {
return new BrokerResponseNativeV2();
}
- public static BrokerResponseNativeV2 fromJsonString(String jsonString)
- throws IOException {
- return JsonUtils.stringToObject(jsonString, BrokerResponseNativeV2.class);
+ public void addStageStats(JsonNode stageStats) {
+ ObjectNode node = JsonUtils.newObjectNode();
+ node.put("stage", _stageIdStats.size());
+ node.set("stats", stageStats);
+ _stageIdStats.add(node);
+ }
+
+ @JsonProperty
+ public List<JsonNode> getStageStats() {
+ return _stageIdStats;
+ }
+
+ @JsonProperty
+ public long getMaxRows() {
+ return _maxRows;
+ }
+
+ public void mergeMaxRows(long maxRows) {
+ _maxRows = Math.max(_maxRows, maxRows);
+ }
+
+ @Override
+ public void setTimeUsedMs(long timeUsedMs) {
+ _serverStats.merge(DataTable.MetadataKey.TIME_USED_MS, timeUsedMs);
+ }
+
+ @Override
+ public long getNumDocsScanned() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_DOCS_SCANNED);
+ }
+
+ @Override
+ public long getNumEntriesScannedInFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_IN_FILTER);
+ }
+
+ @Override
+ public long getNumEntriesScannedPostFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_POST_FILTER);
+ }
+
+ @Override
+ public long getNumSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getMinConsumingFreshnessTimeMs() {
+ return _serverStats.getLong(DataTable.MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS);
+ }
+
+ @Override
+ public long getTotalDocs() {
+ return _serverStats.getLong(DataTable.MetadataKey.TOTAL_DOCS);
+ }
+
+ @Override
+ public boolean isNumGroupsLimitReached() {
+ return _serverStats.getBoolean(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED);
+ }
+
+ public void mergeNumGroupsLimitReached(boolean numGroupsLimitReached) {
+ _serverStats.merge(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED, numGroupsLimitReached);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByServer() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_SERVER);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedInvalid() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_INVALID);
}
- public void addStageStat(Integer stageId, BrokerResponseStats brokerResponseStats) {
- // StageExecutionWallTime will always be there, other stats are optional such as OperatorStats
- if (brokerResponseStats.getStageExecWallTimeMs() != -1) {
- _stageIdStats.put(stageId, brokerResponseStats);
+ @Override
+ public long getNumSegmentsPrunedByLimit() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_LIMIT);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByValue() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_VALUE);
+ }
+
+ @Override
+ public long getExplainPlanNumEmptyFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getExplainPlanNumMatchAllFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getOfflineTotalCpuTimeNs() {
+ return getOfflineThreadCpuTimeNs() + getOfflineSystemActivitiesCpuTimeNs()
+ + getOfflineResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public long getRealtimeTotalCpuTimeNs() {
+ return getRealtimeThreadCpuTimeNs() + getRealtimeSystemActivitiesCpuTimeNs()
+ + getRealtimeResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public void setExceptions(List<ProcessingException> exceptions) {
+ for (ProcessingException exception : exceptions) {
+ _processingExceptions.add(new QueryProcessingException(exception.getErrorCode(), exception.getMessage()));
}
}
- @JsonProperty("stageStats")
- public Map<Integer, BrokerResponseStats> getStageIdStats() {
- return _stageIdStats;
+ public void addToExceptions(QueryProcessingException processingException) {
+ _processingExceptions.add(processingException);
+ }
+
+ @Override
+ public int getNumServersQueried() {
+ return _numServersQueried;
+ }
+
+ @Override
+ public void setNumServersQueried(int numServersQueried) {
+ _numServersQueried = numServersQueried;
+ }
+
+ @Override
+ public int getNumServersResponded() {
+ return _numServersResponded;
+ }
+
+ @Override
+ public void setNumServersResponded(int numServersResponded) {
+ _numServersResponded = numServersResponded;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public boolean isMaxRowsInJoinReached() {
+ return _maxRowsInJoinReached;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public void mergeMaxRowsInJoinReached(boolean maxRowsInJoinReached) {
+ _maxRowsInJoinReached |= maxRowsInJoinReached;
+ }
+
+ @Override
+ public int getExceptionsSize() {
+ return _processingExceptions.size();
+ }
+
+ @Override
+ public void setResultTable(ResultTable resultTable) {
+ _resultTable = resultTable;
+ }
+
+ @Override
+ public ResultTable getResultTable() {
+ return _resultTable;
+ }
+
+ @Override
+ public List<QueryProcessingException> getProcessingExceptions() {
+ return List.of();
+ }
+
+ @Override
+ public int getNumRowsResultSet() {
+ return 0;
+ }
+
+ @Override
+ public long getOfflineThreadCpuTimeNs() {
+ return _offlineThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeThreadCpuTimeNs() {
+ return _realtimeThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineSystemActivitiesCpuTimeNs() {
+ return _offlineSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeSystemActivitiesCpuTimeNs() {
+ return _realtimeSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineResponseSerializationCpuTimeNs() {
+ return _offlineResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeResponseSerializationCpuTimeNs() {
+ return _realtimeResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByBroker() {
+ return _numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public void setNumSegmentsPrunedByBroker(long numSegmentsPrunedByBroker) {
+ _numSegmentsPrunedByBroker = numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public String getRequestId() {
+ return _requestId;
+ }
+
+ @Override
+ public void setRequestId(String requestId) {
+ _requestId = requestId;
+ }
+
+ @Override
+ public String getBrokerId() {
+ return _brokerId;
+ }
+
+ @Override
+ public void setBrokerId(String requestId) {
+ _brokerId = requestId;
+ }
+
+ @Override
+ public long getBrokerReduceTimeMs() {
+ return _brokerReduceTimeMs;
+ }
+
+ @Override
+ public void setBrokerReduceTimeMs(long brokerReduceTimeMs) {
+ _brokerReduceTimeMs = brokerReduceTimeMs;
+ }
+
+ @JsonProperty(access = JsonProperty.Access.READ_ONLY)
+ @Override
+ public boolean isPartialResult() {
+ return isNumGroupsLimitReached() || getExceptionsSize() > 0 || isMaxRowsInJoinReached();
+ }
+
+ public void addServerStats(StatMap<DataTable.MetadataKey> serverStats) {
+ // Set execution statistics.
+ _serverStats.merge(serverStats);
+
+ long threadCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.THREAD_CPU_TIME_NS);
+ long systemActivitiesCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.SYSTEM_ACTIVITIES_CPU_TIME_NS);
+ long responseSerializationCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.RESPONSE_SER_CPU_TIME_NS);
+
+ String tableName = serverStats.getString(DataTable.MetadataKey.TABLE);
+ if (tableName != null) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
+
+ if (tableType == TableType.OFFLINE) {
+ _offlineThreadCpuTimeNs += threadCpuTimeNs;
+ _offlineSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _offlineResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ } else {
+ _realtimeThreadCpuTimeNs += threadCpuTimeNs;
+ _realtimeSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _realtimeResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ }
Review Comment:
This is not correct because tableName never contains the _OFFLINE or _REALTIME suffix, so these stats are always counted as realtime.
Right now as far as I can test, these stats are not being filled in V2. @Jackie-Jiang do you think we should just remove this or always count the times in either offline or realtime as we are doing right now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1580263654
##########
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNativeV2.java:
##########
@@ -77,20 +100,301 @@ public static BrokerResponseNativeV2 empty() {
return new BrokerResponseNativeV2();
}
- public static BrokerResponseNativeV2 fromJsonString(String jsonString)
- throws IOException {
- return JsonUtils.stringToObject(jsonString, BrokerResponseNativeV2.class);
+ public void addStageStats(JsonNode stageStats) {
+ ObjectNode node = JsonUtils.newObjectNode();
+ node.put("stage", _stageIdStats.size());
+ node.set("stats", stageStats);
+ _stageIdStats.add(node);
+ }
+
+ @JsonProperty
+ public List<JsonNode> getStageStats() {
+ return _stageIdStats;
+ }
+
+ @JsonProperty
+ public long getMaxRows() {
+ return _maxRows;
+ }
+
+ public void mergeMaxRows(long maxRows) {
+ _maxRows = Math.max(_maxRows, maxRows);
+ }
+
+ @Override
+ public void setTimeUsedMs(long timeUsedMs) {
+ _serverStats.merge(DataTable.MetadataKey.TIME_USED_MS, timeUsedMs);
+ }
+
+ @Override
+ public long getNumDocsScanned() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_DOCS_SCANNED);
+ }
+
+ @Override
+ public long getNumEntriesScannedInFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_IN_FILTER);
+ }
+
+ @Override
+ public long getNumEntriesScannedPostFilter() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_ENTRIES_SCANNED_POST_FILTER);
+ }
+
+ @Override
+ public long getNumSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsQueried() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_QUERIED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsProcessed() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_PROCESSED);
+ }
+
+ @Override
+ public long getNumConsumingSegmentsMatched() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_CONSUMING_SEGMENTS_MATCHED);
+ }
+
+ @Override
+ public long getMinConsumingFreshnessTimeMs() {
+ return _serverStats.getLong(DataTable.MetadataKey.MIN_CONSUMING_FRESHNESS_TIME_MS);
+ }
+
+ @Override
+ public long getTotalDocs() {
+ return _serverStats.getLong(DataTable.MetadataKey.TOTAL_DOCS);
+ }
+
+ @Override
+ public boolean isNumGroupsLimitReached() {
+ return _serverStats.getBoolean(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED);
+ }
+
+ public void mergeNumGroupsLimitReached(boolean numGroupsLimitReached) {
+ _serverStats.merge(DataTable.MetadataKey.NUM_GROUPS_LIMIT_REACHED, numGroupsLimitReached);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByServer() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_SERVER);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedInvalid() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_INVALID);
}
- public void addStageStat(Integer stageId, BrokerResponseStats brokerResponseStats) {
- // StageExecutionWallTime will always be there, other stats are optional such as OperatorStats
- if (brokerResponseStats.getStageExecWallTimeMs() != -1) {
- _stageIdStats.put(stageId, brokerResponseStats);
+ @Override
+ public long getNumSegmentsPrunedByLimit() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_LIMIT);
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByValue() {
+ return _serverStats.getLong(DataTable.MetadataKey.NUM_SEGMENTS_PRUNED_BY_VALUE);
+ }
+
+ @Override
+ public long getExplainPlanNumEmptyFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getExplainPlanNumMatchAllFilterSegments() {
+ return _serverStats.getLong(DataTable.MetadataKey.EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS);
+ }
+
+ @Override
+ public long getOfflineTotalCpuTimeNs() {
+ return getOfflineThreadCpuTimeNs() + getOfflineSystemActivitiesCpuTimeNs()
+ + getOfflineResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public long getRealtimeTotalCpuTimeNs() {
+ return getRealtimeThreadCpuTimeNs() + getRealtimeSystemActivitiesCpuTimeNs()
+ + getRealtimeResponseSerializationCpuTimeNs();
+ }
+
+ @Override
+ public void setExceptions(List<ProcessingException> exceptions) {
+ for (ProcessingException exception : exceptions) {
+ _processingExceptions.add(new QueryProcessingException(exception.getErrorCode(), exception.getMessage()));
}
}
- @JsonProperty("stageStats")
- public Map<Integer, BrokerResponseStats> getStageIdStats() {
- return _stageIdStats;
+ public void addToExceptions(QueryProcessingException processingException) {
+ _processingExceptions.add(processingException);
+ }
+
+ @Override
+ public int getNumServersQueried() {
+ return _numServersQueried;
+ }
+
+ @Override
+ public void setNumServersQueried(int numServersQueried) {
+ _numServersQueried = numServersQueried;
+ }
+
+ @Override
+ public int getNumServersResponded() {
+ return _numServersResponded;
+ }
+
+ @Override
+ public void setNumServersResponded(int numServersResponded) {
+ _numServersResponded = numServersResponded;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public boolean isMaxRowsInJoinReached() {
+ return _maxRowsInJoinReached;
+ }
+
+ @JsonProperty("maxRowsInJoinReached")
+ public void mergeMaxRowsInJoinReached(boolean maxRowsInJoinReached) {
+ _maxRowsInJoinReached |= maxRowsInJoinReached;
+ }
+
+ @Override
+ public int getExceptionsSize() {
+ return _processingExceptions.size();
+ }
+
+ @Override
+ public void setResultTable(ResultTable resultTable) {
+ _resultTable = resultTable;
+ }
+
+ @Override
+ public ResultTable getResultTable() {
+ return _resultTable;
+ }
+
+ @Override
+ public List<QueryProcessingException> getProcessingExceptions() {
+ return List.of();
+ }
+
+ @Override
+ public int getNumRowsResultSet() {
+ return 0;
+ }
+
+ @Override
+ public long getOfflineThreadCpuTimeNs() {
+ return _offlineThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeThreadCpuTimeNs() {
+ return _realtimeThreadCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineSystemActivitiesCpuTimeNs() {
+ return _offlineSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeSystemActivitiesCpuTimeNs() {
+ return _realtimeSystemActivitiesCpuTimeNs;
+ }
+
+ @Override
+ public long getOfflineResponseSerializationCpuTimeNs() {
+ return _offlineResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getRealtimeResponseSerializationCpuTimeNs() {
+ return _realtimeResponseSerializationCpuTimeNs;
+ }
+
+ @Override
+ public long getNumSegmentsPrunedByBroker() {
+ return _numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public void setNumSegmentsPrunedByBroker(long numSegmentsPrunedByBroker) {
+ _numSegmentsPrunedByBroker = numSegmentsPrunedByBroker;
+ }
+
+ @Override
+ public String getRequestId() {
+ return _requestId;
+ }
+
+ @Override
+ public void setRequestId(String requestId) {
+ _requestId = requestId;
+ }
+
+ @Override
+ public String getBrokerId() {
+ return _brokerId;
+ }
+
+ @Override
+ public void setBrokerId(String requestId) {
+ _brokerId = requestId;
+ }
+
+ @Override
+ public long getBrokerReduceTimeMs() {
+ return _brokerReduceTimeMs;
+ }
+
+ @Override
+ public void setBrokerReduceTimeMs(long brokerReduceTimeMs) {
+ _brokerReduceTimeMs = brokerReduceTimeMs;
+ }
+
+ @JsonProperty(access = JsonProperty.Access.READ_ONLY)
+ @Override
+ public boolean isPartialResult() {
+ return isNumGroupsLimitReached() || getExceptionsSize() > 0 || isMaxRowsInJoinReached();
+ }
+
+ public void addServerStats(StatMap<DataTable.MetadataKey> serverStats) {
+ // Set execution statistics.
+ _serverStats.merge(serverStats);
+
+ long threadCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.THREAD_CPU_TIME_NS);
+ long systemActivitiesCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.SYSTEM_ACTIVITIES_CPU_TIME_NS);
+ long responseSerializationCpuTimeNs = serverStats.getLong(DataTable.MetadataKey.RESPONSE_SER_CPU_TIME_NS);
+
+ String tableName = serverStats.getString(DataTable.MetadataKey.TABLE);
+ if (tableName != null) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
+
+ if (tableType == TableType.OFFLINE) {
+ _offlineThreadCpuTimeNs += threadCpuTimeNs;
+ _offlineSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _offlineResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ } else {
+ _realtimeThreadCpuTimeNs += threadCpuTimeNs;
+ _realtimeSystemActivitiesCpuTimeNs += systemActivitiesCpuTimeNs;
+ _realtimeResponseSerializationCpuTimeNs += responseSerializationCpuTimeNs;
+ }
Review Comment:
Good catch! I guess this is an example where we cannot directly expose V1 stats as V2 stats even for leaf stage. V1 server request only touch one table (either OFFLINE or REALTIME), but V2 leaf stage can touch 1 or 2 tables. To solve this, I think we should read the V1 stats in leaf stage, and convert them into leaf stage stats there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1546149252
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java:
##########
@@ -71,9 +71,6 @@
* +-----------------------------------------------+
* | VARIABLE_SIZE_DATA SECTION |
* +-----------------------------------------------+
- * | METADATA LENGTH |
- * | METADATA SECTION |
- * +-----------------------------------------------+
Review Comment:
This wasn't useful. All blocks but metadata blocks left this empty and in metadata blocks the content was equivalent to variable_size_data, but codified as a single string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12704:
URL: https://github.com/apache/pinot/pull/12704#issuecomment-2039443675
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12704?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `65.53156%` with `415 lines` in your changes are missing coverage. Please review.
> Project coverage is 61.70%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`dec1e22`)](https://app.codecov.io/gh/apache/pinot/pull/12704?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 213 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12704?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...pinot/query/runtime/plan/MultiStageQueryStats.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Fplan%2FMultiStageQueryStats.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL011bHRpU3RhZ2VRdWVyeVN0YXRzLmphdmE=) | 64.45% | [66 Missing and 25 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...requesthandler/MultiStageBrokerRequestHandler.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fbroker%2Frequesthandler%2FMultiStageBrokerRequestHandler.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvTXVsdGlTdGFnZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | 0.00% | [86 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/operator/LeafStageTransferableBlockOperator.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Foperator%2FLeafStageTransferableBlockOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9MZWFmU3RhZ2VUcmFuc2ZlcmFibGVCbG9ja09wZXJhdG9yLmphdmE=) | 58.53% | [66 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/pinot/common/datatable/StatMap.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fdatatable%2FStatMap.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZGF0YXRhYmxlL1N0YXRNYXAuamF2YQ==) | 74.61% | [30 Missing and 19 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...t/common/response/broker/BrokerResponseNative.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fresponse%2Fbroker%2FBrokerResponseNative.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvYnJva2VyL0Jyb2tlclJlc3BvbnNlTmF0aXZlLmphdmE=) | 64.63% | [26 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...y/runtime/operator/BaseMailboxReceiveOperator.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Foperator%2FBaseMailboxReceiveOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9CYXNlTWFpbGJveFJlY2VpdmVPcGVyYXRvci5qYXZh) | 62.06% | [7 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...common/response/broker/BrokerResponseNativeV2.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fresponse%2Fbroker%2FBrokerResponseNativeV2.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvYnJva2VyL0Jyb2tlclJlc3BvbnNlTmF0aXZlVjIuamF2YQ==) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../pinot/query/runtime/blocks/TransferableBlock.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Fblocks%2FTransferableBlock.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9ibG9ja3MvVHJhbnNmZXJhYmxlQmxvY2suamF2YQ==) | 77.14% | [6 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...g/apache/pinot/common/datablock/MetadataBlock.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fdatablock%2FMetadataBlock.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZGF0YWJsb2NrL01ldGFkYXRhQmxvY2suamF2YQ==) | 87.27% | [6 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ot/query/runtime/operator/MailboxSendOperator.java](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&filepath=pinot-query-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fquery%2Fruntime%2Foperator%2FMailboxSendOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9vcGVyYXRvci9NYWlsYm94U2VuZE9wZXJhdG9yLmphdmE=) | 78.57% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [20 more](https://app.codecov.io/gh/apache/pinot/pull/12704?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12704 +/- ##
============================================
- Coverage 61.75% 61.70% -0.05%
- Complexity 207 1202 +995
============================================
Files 2436 2464 +28
Lines 133233 135647 +2414
Branches 20636 20969 +333
============================================
+ Hits 82274 83700 +1426
- Misses 44911 45706 +795
- Partials 6048 6241 +193
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.70% <65.53%> (+0.07%)` | :arrow_up: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.50% <70.57%> (-15.25%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.80% <0.00%> (+0.07%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.70% <65.53%> (-0.05%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `61.70% <65.53%> (-0.05%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.50% <70.57%> (-0.39%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12704/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.80% <0.00%> (+0.07%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12704?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565227466
##########
pinot-broker/src/test/java/org/apache/pinot/broker/querylog/QueryLoggerTest.java:
##########
@@ -259,11 +259,9 @@ private QueryLogger.QueryLogParams generateParams(boolean isGroupLimitHit, int n
response.setExceptions(
IntStream.range(0, numExceptions)
.mapToObj(i -> new ProcessingException()).collect(Collectors.toList()));
- response.setOfflineTotalCpuTimeNs(14);
Review Comment:
Are these 2 removal intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565356393
##########
pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java:
##########
@@ -572,14 +585,9 @@ private Map<Integer, String> deserializeExceptions(ByteBuffer buffer)
@Override
public String toString() {
if (_dataSchema == null) {
- return _metadata.toString();
+ return "{}";
Review Comment:
We don't have the ability to decode what is in metadata now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565537934
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingMultiStreamConsumer.java:
##########
@@ -240,6 +253,15 @@ protected boolean isEos(TransferableBlock element) {
return element.isSuccessfulEndOfStreamBlock();
}
+ @Override
+ protected void onConsumerFinish(TransferableBlock element) {
+ if (element.getQueryStats() != null) {
+ _stats.mergeUpstream(element.getQueryStats());
+ } else {
+ _stats.mergeUpstream(element.getSerializedStatsByStage());
Review Comment:
By doing this we don't need to deserialize the stats when they are sent using memory mailboxes. But right now we are not prioritizing sending the stats there. That is something we can improve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12704:
URL: https://github.com/apache/pinot/pull/12704#issuecomment-2055488944
Will there be downtime when broker and servers are running different versions? Please add a section in the PR description describing what to expect during upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582764851
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -419,4 +550,122 @@ public void send(BaseResultsBlock block)
addResultsBlock(block);
}
}
+
+ public enum StatKey implements StatMap.Key {
+ TABLE(StatMap.Type.STRING),
+ EXECUTION_TIME_MS(StatMap.Type.LONG, null, DataTable.MetadataKey.TIME_USED_MS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ EMITTED_ROWS(StatMap.Type.LONG, null, DataTable.MetadataKey.NUM_ROWS) {
+ @Override
+ public boolean includeDefaultInJson() {
+ return true;
+ }
+ },
+ NUM_DOCS_SCANNED(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(StatMap.Type.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(StatMap.Type.LONG),
+ NUM_SEGMENTS_QUERIED(StatMap.Type.INT),
+ NUM_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(StatMap.Type.INT),
+ // the timestamp indicating the freshness of the data queried in consuming segments.
+ // This can be ingestion timestamp if provided by the stream, or the last index time
+ MIN_CONSUMING_FRESHNESS_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ TOTAL_DOCS(StatMap.Type.LONG),
+ NUM_GROUPS_LIMIT_REACHED(StatMap.Type.BOOLEAN),
+ //TRACE_INFO(StatMap.Type.STRING),
+ //REQUEST_ID(StatMap.Type.LONG),
+ NUM_RESIZES(StatMap.Type.INT),
+ RESIZE_TIME_MS(StatMap.Type.LONG),
+ THREAD_CPU_TIME_NS(StatMap.Type.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(StatMap.Type.LONG),
+ RESPONSE_SER_CPU_TIME_NS(StatMap.Type.LONG, "responseSerializationCpuTimeNs"),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(StatMap.Type.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(StatMap.Type.INT),
+ //EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(StatMap.Type.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(StatMap.Type.INT),
+ NUM_BLOCKS(StatMap.Type.INT),
+ OPERATOR_EXECUTION_TIME_MS(StatMap.Type.LONG),
+ OPERATOR_ID(StatMap.Type.STRING),
+ OPERATOR_EXEC_START_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return StatMap.Key.minPositive(value1, value2);
+ }
+ },
+ OPERATOR_EXEC_END_TIME_MS(StatMap.Type.LONG) {
+ @Override
+ public long merge(long value1, long value2) {
+ return Math.max(value1, value2);
+ }
+ },;
+ private final StatMap.Type _type;
+ @Nullable
+ private final DataTable.MetadataKey _v1Key;
+ private final String _statName;
+
+ StatKey(StatMap.Type type) {
+ this(type, null);
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = DataTable.MetadataKey.getByName(getStatName());
+ }
+
+ StatKey(StatMap.Type type, @Nullable String statName, @Nullable DataTable.MetadataKey v1Key) {
+ _type = type;
+ _statName = statName == null ? StatMap.getDefaultStatName(this) : statName;
+ _v1Key = v1Key == null ? DataTable.MetadataKey.getByName(getStatName()) : v1Key;
+ }
+
+ @Override
+ public String getStatName() {
+ return _statName;
+ }
+
+ @Override
+ public StatMap.Type getType() {
+ return _type;
+ }
+
+ public void updateV1Metadata(StatMap<DataTable.MetadataKey> oldMetadata, StatMap<StatKey> stats) {
Review Comment:
The stats returned by `BrokerResponseNative` are based on the stats in `MetadataKey` and extend them with some extra stats not present in there. But instead of using the map from `MetadataKey` to values we already have, `BrokerResponseNative` stores all of them in specific attributes, which makes `BrokerResponseNative` a bit difficult to understand, specially to understand which attributes are unique to `BrokerResponseNative` and which are just a copy of some `MetadataKey`.
In master, `BrokerResponseNativeV2` extends `BrokerResponseNative`, so it inherits the same pattern. This PR is improving the situation by storing a `StatMap` with the `MetadataKey` values. Now it is easy to see which values are just `MetadataKey` and which ones are actual new concepts added in the broker.
> If we remove the mapping from v2 to v1, can we make v1 stats not implementing StatMap.Key?
This is not the first time you ask about that. My answer is: Why is that a target? What is the issue of `MetadataKey` implementing `StatMap.Key`? To me it looks like an improvement. Now, for example, `MetadataKey`s can define how are they merged in a nice way. `StatMap` doesn't have to be limited to store the V2 stats but any stat like feature in a efficient way.
> Does it work if we do not fill the v1 stats, but directly use v2 stats to fill the broker response?
There is no V2 stat that can assume this role. Because `BrokerResponse` stats are showing an aggregated view of the query stats, while V2 stats are operation base. For example, in a query we may have 2 different aggregations. One of them may have reached its num groups limit and the other may not.
In that case, there would be a single value for `BrokerResponse.isNumGroupsLimitReached`, which will be true. But from the V2 stats point of view there are two instances of a `Map<LeafStageTransferableBlockOperator.StatKey>` (or `Map<AggregateOperator.StatKey>`). One of them will say that the limit has being reached and the other will say the opposite. In order to generate the actual `BrokerResponse` we need to merge both.
Then we would need to create a new StatKey for BrokerResponse and that new enum would end up being almost exactly equal to `MetadataKey`.
Could we implement `BrokerNativeResponseV2` without using a `StatMap`? Yes, we could, but then we would need to add a lot of logic. We already have that in `ExecutionStatsAggregator.setStats` for V1 and we already had that for V2. Now instead of having at least two almost identical large method merging the stats, we have that merge logic defined in the stats, so it is more difficult to mess it up. These large methods are problematic because they are are mostly adding stats, but sometimes the merge logic is different and we need to be aware of that each time we merge the stats. With `StatKey` we end up having smaller methods which call `StatKey.merge` which is where the actual logic is defined.
> The overall goal here is to decouple the v2 and v1 handling so that in the future we can remove v1 easier if necessary.
This PR doesn't make more difficult to do so, in fact it makes it easier. Apart from collecting stats in V1, the main blocker to remove `MetadataKey` (the V1 stats) is that `BrokerResponse` and `MetadataKey` are very coupled.
The idea of mapping V2 stats into V1 is there just to make it easier to return them in `BrokerResponse`. If in the future we want to remove `MetadataKey`, we can just create a new class that implements `StatKey` and map from V2 to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1583790911
##########
pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java:
##########
@@ -123,6 +127,17 @@ String toJsonString()
*/
long getMinConsumingFreshnessTimeMs();
+ /**
+ * Get the max number of rows seen by a single operator in the query processing chain.
+ * <p>
+ * In single-stage this value is equal to {@link #getNumEntriesScannedPostFilter()} because single-stage operators
+ * cannot generate more rows than the number of rows received. This is not the case in multi-stage operators, where
+ * operators like join or union can emit more rows than the ones received on each of its children operators.
+ */
+ default long getMaxRowsInOperator() {
Review Comment:
To me this is actually a little bit confusing. Correct me if my understanding is wrong:
- In `v1` it is a query level stats on entries (rows * columns projected), which is the total `numEntriesScannedPostFilter` across all segments queried
- In `v2` it is the maximum rows returned by the same operator (there can be multiple operators of the same type, but they are counted separately) across all the workers. It is essentially the maximum of all the `EMITTED_ROWS` in the stats map.
In `v1` the closest matching might be `numDocsScanned`.
I feel we may discuss about this and add it as a separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1584390473
##########
pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java:
##########
@@ -123,6 +127,17 @@ String toJsonString()
*/
long getMinConsumingFreshnessTimeMs();
+ /**
+ * Get the max number of rows seen by a single operator in the query processing chain.
+ * <p>
+ * In single-stage this value is equal to {@link #getNumEntriesScannedPostFilter()} because single-stage operators
+ * cannot generate more rows than the number of rows received. This is not the case in multi-stage operators, where
+ * operators like join or union can emit more rows than the ones received on each of its children operators.
+ */
+ default long getMaxRowsInOperator() {
Review Comment:
Removed in 02ef0b23076064ae44b1d2496f948d6ed00dff36. We can open a future PR to add this if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1582728941
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -798,7 +798,6 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
_brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_NUM_GROUPS_LIMIT_REACHED,
1);
}
- brokerResponse.setPartialResult(isPartialResult(brokerResponse));
Review Comment:
removed in 2d9649147559782e40fab44542dc93c0bb999249 a
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -798,7 +798,6 @@ protected BrokerResponse handleRequest(long requestId, String query, @Nullable S
_brokerMetrics.addMeteredTableValue(rawTableName, BrokerMeter.BROKER_RESPONSES_WITH_NUM_GROUPS_LIMIT_REACHED,
1);
}
- brokerResponse.setPartialResult(isPartialResult(brokerResponse));
Review Comment:
removed in 2d9649147559782e40fab44542dc93c0bb999249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org