You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/12/13 19:55:03 UTC
[GitHub] [druid] kfaraz opened a new pull request, #13559: Track input processedBytes with MSQ ingestion
kfaraz opened a new pull request, #13559:
URL: https://github.com/apache/druid/pull/13559
Follow up to #13520
### Description
Bytes processed are currently tracked for intermediate stages in MSQ ingestion. This PR adds the capability to track the bytes processed by an MSQ controller task while reading from an external input source.
### Changes
- Create an instance of `InputStats` for every `InputSource` in `ExternalInputSliceReader`
- Track `processedBytes` in `InputStats` by invoking `InputSourceReader.read(inputStats)`
- Update `ChannelCounters` with the above obtained `processedBytes`
- Update task report structure in docs
### Total input processed bytes
The total input processed bytes can be obtained by summing the `processedBytes` as follows:
```
totalBytes = 0
for every worker in stage 0:
for every input channel: (i.e. channels with prefix "input", e.g. "input0", "input1", etc.)
totalBytes += processedBytes
```
### Pending changes
- [ ] Verify with multiple input sources and workers
- [ ] Tests
### Release note
- [ ] TODO
### Screenshots
`processedBytes` with sample "wikipedia" datasource:
- MSQ batch:
<img width="295" alt="msq_bytes" src="https://user-images.githubusercontent.com/18635897/207430326-3cc44205-df41-4c49-984b-f57acf5179fe.png">
- Classic batch:
<img width="369" alt="batch_bytes" src="https://user-images.githubusercontent.com/18635897/207430937-47608d23-739b-4820-8a5a-7c7bdbd66fb9.png">
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] cryptoe commented on a diff in pull request #13559: Track input processedBytes with MSQ ingestion
Posted by GitBox <gi...@apache.org>.
cryptoe commented on code in PR #13559:
URL: https://github.com/apache/druid/pull/13559#discussion_r1049237812
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/MSQTestBase.java:
##########
@@ -613,6 +617,11 @@ public IngestTester testIngestQuery()
return new IngestTester();
}
+ public static CounterSnapshotBuilder channelCountersWith()
Review Comment:
Nit: syle I think CounterSnapshotBuilder can be a new class of its own and then this method gets pushed there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz merged pull request #13559: Track input processedBytes with MSQ ingestion
Posted by GitBox <gi...@apache.org>.
kfaraz merged PR #13559:
URL: https://github.com/apache/druid/pull/13559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13559: Track input processedBytes with MSQ ingestion
Posted by GitBox <gi...@apache.org>.
kfaraz commented on code in PR #13559:
URL: https://github.com/apache/druid/pull/13559#discussion_r1049285207
##########
extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/MSQTestBase.java:
##########
@@ -613,6 +617,11 @@ public IngestTester testIngestQuery()
return new IngestTester();
}
+ public static CounterSnapshotBuilder channelCountersWith()
Review Comment:
Sure, I will move it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13559: Track input processedBytes with MSQ ingestion
Posted by GitBox <gi...@apache.org>.
kfaraz commented on PR #13559:
URL: https://github.com/apache/druid/pull/13559#issuecomment-1352914412
Added some more validations for SELECT and REPLACE.
The bytes are currently being validated in tests only for external input, not for segment input.
To validate bytes read from the segment, the following things might need to be done:
- `QueryableIndex` could expose a `getSize()`
- the segments created during the initial setup of `SqlTestFramework.createQuerySegmentWalker()` could be reused in the `MSQTestBase.getSupplierForSegment()`
- the corresponding `DataSegment` objects should be retained in the `BaseCalciteQueryTest` to be used for expected size.
Since doing all of this would be outside the scope of this PR, we can address it separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org