You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/01/08 07:24:23 UTC
[GitHub] [iotdb] neuyilan opened a new pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
neuyilan opened a new pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!-- Thanks for trying to help us make Apache IoTDB be the best it can be!
Please fill out as much of the following information as is possible (where relevant, and remove it
when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
## Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue,
it's not necessary to repeat the description here,
however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section
for each of them. For example: -->
### Content1 ...
### Content2 ...
### Content3 ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error
conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition,
design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design
(or naming) decision point and compare the alternatives with the designs that you've implemented
(or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere
(e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list),
link to that discussion from this PR description and explain what have changed in your final design
compared to your original proposal or the consensus version in the end of the discussion.
If something hasn't changed since the original discussion, you can omit a detailed discussion of
those aspects of the design here, perhaps apart from brief mentioning for the sake of readability
of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] concurrent read
- [ ] concurrent write
- [ ] concurrent read and write
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods.
- [ ] added or updated version, __license__, or notice information
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious
for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold
for code coverage.
- [ ] added integration tests.
- [ ] been tested in a test IoTDB cluster.
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items
apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items
from the checklist above are strictly necessary, but it would be very helpful if you at least
self-review the PR. -->
<hr>
##### Key changed/added classes (or packages if there are too many classes) in this PR
<!-- Template copied and modified from Apache Druid-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757411187
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-756623738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757073453
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759901022
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759892476
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757409324
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan edited a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan edited a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-762230593
The test code is shown as below, and when called `insertTabletsSecondFailed()`, the following error will be occurred because of datatype not match.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-758360991
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757933395
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-756623738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757073902
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan edited a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan edited a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-762230593
The test code is shown as below, and when called `insertTabletsSecondFailed()`, the following error will be occurred because of datatype not match.
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
import org.apache.iotdb.session.SessionDataSet;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.utils.Binary;
import org.apache.iotdb.tsfile.write.record.Tablet;
import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
public class SessionExample {
private static Session session;
private static final String ROOT_SG1_D1 = "root.sg1.d1";
public static void main(String[] args)
throws IoTDBConnectionException, StatementExecutionException {
session = new Session("127.0.0.1", 55560, "root", "root");
session.open(false);
//set session fetchSize
session.setFetchSize(10000);
insertTabletsFirstSuccess();
insertTabletsSecondFailed();
query();
session.close();
}
private static void query() throws IoTDBConnectionException, StatementExecutionException {
SessionDataSet dataSet = session.executeQueryStatement("select * from root");
System.out.println(dataSet.getColumnNames());
dataSet.setFetchSize(1024); // default is 10000
while (dataSet.hasNext()) {
System.out.println(dataSet.next());
}
dataSet.closeOperationHandle();
}
private static void insertTabletsFirstSuccess()
throws IoTDBConnectionException, StatementExecutionException {
// The schema of measurements of one device
// only measurementId and data type in MeasurementSchema take effects in Tablet
List<MeasurementSchema> schemaList = new ArrayList<>();
schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s3", TSDataType.INT64));
Tablet tablet1 = new Tablet(ROOT_SG1_D1, schemaList, 100);
Tablet tablet2 = new Tablet("root.sg1.d2", schemaList, 100);
Tablet tablet3 = new Tablet("root.sg1.d3", schemaList, 100);
Map<String, Tablet> tabletMap = new HashMap<>();
tabletMap.put(ROOT_SG1_D1, tablet1);
tabletMap.put("root.sg1.d2", tablet2);
tabletMap.put("root.sg1.d3", tablet3);
//Method 1 to add tablet data
long timestamp = System.currentTimeMillis();
for (long row = 0; row < 100; row++) {
int row1 = tablet1.rowSize++;
int row2 = tablet2.rowSize++;
int row3 = tablet3.rowSize++;
tablet1.addTimestamp(row1, timestamp);
tablet2.addTimestamp(row2, timestamp);
tablet3.addTimestamp(row3, timestamp);
for (int i = 0; i < 3; i++) {
long value = new Random().nextLong();
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, value);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, value);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, value);
}
if (tablet1.rowSize == tablet1.getMaxRowNumber()) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
timestamp++;
}
if (tablet1.rowSize != 0) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
}
private static void insertTabletsSecondFailed()
throws IoTDBConnectionException, StatementExecutionException {
// The schema of measurements of one device
// only measurementId and data type in MeasurementSchema take effects in Tablet
List<MeasurementSchema> schemaList = new ArrayList<>();
schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s3", TSDataType.TEXT));
Tablet tablet1 = new Tablet(ROOT_SG1_D1, schemaList, 100);
Tablet tablet2 = new Tablet("root.sg1.d2", schemaList, 100);
Tablet tablet3 = new Tablet("root.sg1.d3", schemaList, 100);
Map<String, Tablet> tabletMap = new HashMap<>();
tabletMap.put(ROOT_SG1_D1, tablet1);
tabletMap.put("root.sg1.d2", tablet2);
tabletMap.put("root.sg1.d3", tablet3);
//Method 1 to add tablet data
long timestamp = System.currentTimeMillis();
for (long row = 0; row < 100; row++) {
int row1 = tablet1.rowSize++;
int row2 = tablet2.rowSize++;
int row3 = tablet3.rowSize++;
tablet1.addTimestamp(row1, timestamp);
tablet2.addTimestamp(row2, timestamp);
tablet3.addTimestamp(row3, timestamp);
for (int i = 0; i < 3; i++) {
long value = new Random().nextLong();
if (i == 2) {
Binary b = new Binary(value + "-str");
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, b);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, b);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, b);
} else {
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, value);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, value);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, value);
}
}
if (tablet1.rowSize == tablet1.getMaxRowNumber()) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
timestamp++;
}
if (tablet1.rowSize != 0) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
}
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r554687342
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/applier/AsyncDataLogApplier.java
##########
@@ -131,7 +132,7 @@ private PartialPath getPlanKey(PhysicalPlan plan) throws StorageGroupNotSetExcep
private PartialPath getPlanSG(PhysicalPlan plan) throws StorageGroupNotSetException {
PartialPath sgPath = null;
- if (plan instanceof InsertPlan) {
+ if (plan instanceof InsertPlan && !(plan instanceof InsertMultiTabletPlan)) {
Review comment:
Sure, I'll try to do this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r554983918
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/query/ClusterPlanRouter.java
##########
@@ -155,6 +158,40 @@ private PartitionGroup routePlan(ShowChildPathsPlan plan) {
return Collections.singletonMap(plan, partitionGroup);
}
+ private Map<PhysicalPlan, PartitionGroup> splitAndRoutePlan(InsertMultiTabletPlan plan)
+ throws MetadataException {
+ Map<PartitionGroup, InsertMultiTabletPlan> pgInsertMultiTabletPlanMap = new HashMap<>();
+ for (int i = 0; i < plan.getInsertTabletPlanList().size(); i++) {
+ InsertTabletPlan insertTabletPlan = plan.getInsertTabletPlanList().get(i);
+ Map<PhysicalPlan, PartitionGroup> tmpResult = splitAndRoutePlan(insertTabletPlan);
+ for (Map.Entry<PhysicalPlan, PartitionGroup> entry : tmpResult.entrySet()) {
+ InsertTabletPlan tmpPlan = (InsertTabletPlan) entry.getKey();
+ PartitionGroup tmpPg = entry.getValue();
+
+ InsertMultiTabletPlan pgPlan = pgInsertMultiTabletPlanMap.get(tmpPg);
+ if (pgPlan == null) {
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ InsertMultiTabletPlan insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+ pgInsertMultiTabletPlanMap.put(tmpPg, insertMultiTabletPlan);
+ } else {
+ pgPlan.addInsertTabletPlan(tmpPlan, i);
+ pgInsertMultiTabletPlanMap.put(tmpPg, pgPlan);
Review comment:
it's unnecessary, I'll remove it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] LebronAl commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r557000864
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is zero;
+ * <p>
+ * suppose we originally have three InsertTabletPlans in one InsertMultiTabletPlan, then the
+ * initial InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={0,0,0} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlans, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlans, InsertTabletPlan_3 is split into four sub InsertTabletPlans.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belong to two different raft data groups, so will generate two new
+ * InsertMultiTabletPlans
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsertTabletPlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsertTabletPlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsertTabletPlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsertTabletPlanIndexList() {
+ return parentInsertTabletPlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMaxTime()) {
+ maxTime = insertTabletPlan.getMaxTime();
+ }
+ }
+ return maxTime;
+ }
+
+ public int getTabletsSize() {
+ return insertTabletPlanList.size();
+ }
+
+ public Map<Integer, TSStatus> getResults() {
+ return results;
+ }
+
+ /**
+ * @return the total row of the whole InsertTabletPlan
+ */
+ public int getTotalRowCount() {
+ int rowCount = 0;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ rowCount += insertTabletPlan.getRowCount();
+ }
+ return rowCount;
+ }
+
+ /**
+ * Gets the number of rows in the InsertTabletPlan of the index, zero will be returned if the
+ * index is out of bound.
+ *
+ * @param index the index of the insertTabletPlanList
+ * @return the total row count of the insertTabletPlanList.get(i)
+ */
+ public int getRowCount(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return 0;
+ }
+ return insertTabletPlanList.get(index).getRowCount();
+ }
+
+ public PartialPath getFirstDeviceId() {
+ return insertTabletPlanList.get(0).getDeviceId();
+ }
+
+ public InsertTabletPlan getInsertTabletPlan(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return null;
+ }
+ return insertTabletPlanList.get(index);
+ }
+
+ /**
+ * @param index the index of the sub plan in this InsertMultiTabletPlan
+ * @return the parent's index in the parent InsertMultiTabletPlan
+ */
+ public int getParentIndex(int index) {
+ if (index >= parentInsertTabletPlanIndexList.size() || index < 0) {
+ return -1;
+ }
+ return parentInsertTabletPlanIndexList.get(index);
+ }
+
+ @Override
+ public void checkIntegrity() throws QueryProcessException {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.checkIntegrity();
+ }
+ }
+
+ public void setParentInsertTabletPlanIndexList(List<Integer> parentInsertTabletPlanIndexList) {
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void setInsertTabletPlanList(List<InsertTabletPlan> insertTabletPlanList) {
+ this.insertTabletPlanList = insertTabletPlanList;
+ }
+
+ public void setResults(
+ Map<Integer, TSStatus> results) {
+ this.results = results;
+ }
+
+ @Override
+ public void recoverFromFailure() {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.recoverFromFailure();
+ }
+ }
+
+ @Override
+ public void serialize(ByteBuffer buffer) {
+ int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+ buffer.put((byte) type);
+ buffer.putInt(insertTabletPlanList.size());
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.subSerialize(buffer);
+ }
+
+ buffer.putInt(parentInsertTabletPlanIndexList.size());
+ for (Integer index : parentInsertTabletPlanIndexList) {
+ buffer.putInt(index);
+ }
+ }
+
+ @Override
+ public void serialize(DataOutputStream stream) throws IOException {
+ int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+ stream.writeByte((byte) type);
+ stream.writeInt(insertTabletPlanList.size());
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.subSerialize(stream);
+ }
+
+ stream.writeInt(parentInsertTabletPlanIndexList.size());
Review comment:
same
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is zero;
+ * <p>
+ * suppose we originally have three InsertTabletPlans in one InsertMultiTabletPlan, then the
+ * initial InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={0,0,0} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlans, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlans, InsertTabletPlan_3 is split into four sub InsertTabletPlans.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belong to two different raft data groups, so will generate two new
+ * InsertMultiTabletPlans
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsertTabletPlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsertTabletPlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsertTabletPlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsertTabletPlanIndexList() {
+ return parentInsertTabletPlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMaxTime()) {
+ maxTime = insertTabletPlan.getMaxTime();
+ }
+ }
+ return maxTime;
+ }
+
+ public int getTabletsSize() {
+ return insertTabletPlanList.size();
+ }
+
+ public Map<Integer, TSStatus> getResults() {
+ return results;
+ }
+
+ /**
+ * @return the total row of the whole InsertTabletPlan
+ */
+ public int getTotalRowCount() {
+ int rowCount = 0;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ rowCount += insertTabletPlan.getRowCount();
+ }
+ return rowCount;
+ }
+
+ /**
+ * Gets the number of rows in the InsertTabletPlan of the index, zero will be returned if the
+ * index is out of bound.
+ *
+ * @param index the index of the insertTabletPlanList
+ * @return the total row count of the insertTabletPlanList.get(i)
+ */
+ public int getRowCount(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return 0;
+ }
+ return insertTabletPlanList.get(index).getRowCount();
+ }
+
+ public PartialPath getFirstDeviceId() {
+ return insertTabletPlanList.get(0).getDeviceId();
+ }
+
+ public InsertTabletPlan getInsertTabletPlan(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return null;
+ }
+ return insertTabletPlanList.get(index);
+ }
+
+ /**
+ * @param index the index of the sub plan in this InsertMultiTabletPlan
+ * @return the parent's index in the parent InsertMultiTabletPlan
+ */
+ public int getParentIndex(int index) {
+ if (index >= parentInsertTabletPlanIndexList.size() || index < 0) {
+ return -1;
+ }
+ return parentInsertTabletPlanIndexList.get(index);
+ }
+
+ @Override
+ public void checkIntegrity() throws QueryProcessException {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.checkIntegrity();
+ }
+ }
+
+ public void setParentInsertTabletPlanIndexList(List<Integer> parentInsertTabletPlanIndexList) {
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void setInsertTabletPlanList(List<InsertTabletPlan> insertTabletPlanList) {
+ this.insertTabletPlanList = insertTabletPlanList;
+ }
+
+ public void setResults(
+ Map<Integer, TSStatus> results) {
+ this.results = results;
+ }
+
+ @Override
+ public void recoverFromFailure() {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.recoverFromFailure();
+ }
+ }
+
+ @Override
+ public void serialize(ByteBuffer buffer) {
+ int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+ buffer.put((byte) type);
+ buffer.putInt(insertTabletPlanList.size());
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.subSerialize(buffer);
+ }
+
+ buffer.putInt(parentInsertTabletPlanIndexList.size());
Review comment:
It seems that `insertTabletPlanList.size()` is always same to `parentInsertTabletPlanIndexList.size()`,then maybe it's unnecessary to serialize it.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is zero;
+ * <p>
+ * suppose we originally have three InsertTabletPlans in one InsertMultiTabletPlan, then the
+ * initial InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={0,0,0} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlans, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlans, InsertTabletPlan_3 is split into four sub InsertTabletPlans.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belong to two different raft data groups, so will generate two new
+ * InsertMultiTabletPlans
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsertTabletPlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsertTabletPlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsertTabletPlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsertTabletPlanIndexList() {
+ return parentInsertTabletPlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMaxTime()) {
+ maxTime = insertTabletPlan.getMaxTime();
+ }
+ }
+ return maxTime;
+ }
+
+ public int getTabletsSize() {
+ return insertTabletPlanList.size();
+ }
+
+ public Map<Integer, TSStatus> getResults() {
+ return results;
+ }
+
+ /**
+ * @return the total row of the whole InsertTabletPlan
+ */
+ public int getTotalRowCount() {
+ int rowCount = 0;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ rowCount += insertTabletPlan.getRowCount();
+ }
+ return rowCount;
+ }
+
+ /**
+ * Gets the number of rows in the InsertTabletPlan of the index, zero will be returned if the
+ * index is out of bound.
+ *
+ * @param index the index of the insertTabletPlanList
+ * @return the total row count of the insertTabletPlanList.get(i)
+ */
+ public int getRowCount(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return 0;
+ }
+ return insertTabletPlanList.get(index).getRowCount();
+ }
+
+ public PartialPath getFirstDeviceId() {
+ return insertTabletPlanList.get(0).getDeviceId();
+ }
+
+ public InsertTabletPlan getInsertTabletPlan(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return null;
+ }
+ return insertTabletPlanList.get(index);
+ }
+
+ /**
+ * @param index the index of the sub plan in this InsertMultiTabletPlan
+ * @return the parent's index in the parent InsertMultiTabletPlan
+ */
+ public int getParentIndex(int index) {
+ if (index >= parentInsertTabletPlanIndexList.size() || index < 0) {
+ return -1;
+ }
+ return parentInsertTabletPlanIndexList.get(index);
+ }
+
+ @Override
+ public void checkIntegrity() throws QueryProcessException {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.checkIntegrity();
+ }
+ }
+
+ public void setParentInsertTabletPlanIndexList(List<Integer> parentInsertTabletPlanIndexList) {
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void setInsertTabletPlanList(List<InsertTabletPlan> insertTabletPlanList) {
+ this.insertTabletPlanList = insertTabletPlanList;
+ }
+
+ public void setResults(
+ Map<Integer, TSStatus> results) {
+ this.results = results;
+ }
+
+ @Override
+ public void recoverFromFailure() {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.recoverFromFailure();
+ }
+ }
+
+ @Override
+ public void serialize(ByteBuffer buffer) {
+ int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+ buffer.put((byte) type);
+ buffer.putInt(insertTabletPlanList.size());
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.subSerialize(buffer);
+ }
+
+ buffer.putInt(parentInsertTabletPlanIndexList.size());
+ for (Integer index : parentInsertTabletPlanIndexList) {
+ buffer.putInt(index);
+ }
+ }
+
+ @Override
+ public void serialize(DataOutputStream stream) throws IOException {
+ int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+ stream.writeByte((byte) type);
+ stream.writeInt(insertTabletPlanList.size());
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.subSerialize(stream);
+ }
+
+ stream.writeInt(parentInsertTabletPlanIndexList.size());
+ for (Integer index : parentInsertTabletPlanIndexList) {
+ stream.writeInt(index);
+ }
+ }
+
+ @Override
+ public void deserialize(ByteBuffer buffer) throws IllegalPathException {
+ int tmpInsertTabletPlanListSize = buffer.getInt();
+ this.insertTabletPlanList = new ArrayList<>(tmpInsertTabletPlanListSize);
+ for (int i = 0; i < tmpInsertTabletPlanListSize; i++) {
+ InsertTabletPlan tmpPlan = new InsertTabletPlan();
+ tmpPlan.deserialize(buffer);
+ this.insertTabletPlanList.add(tmpPlan);
+ }
+
+ int tmpParentInsetTablePlanIndexListSize = buffer.getInt();
Review comment:
same
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r557819410
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is zero;
+ * <p>
+ * suppose we originally have three InsertTabletPlans in one InsertMultiTabletPlan, then the
+ * initial InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={0,0,0} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlans, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlans, InsertTabletPlan_3 is split into four sub InsertTabletPlans.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belong to two different raft data groups, so will generate two new
+ * InsertMultiTabletPlans
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsertTabletPlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsertTabletPlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsertTabletPlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsertTabletPlanIndexList() {
+ return parentInsertTabletPlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMaxTime()) {
+ maxTime = insertTabletPlan.getMaxTime();
+ }
+ }
+ return maxTime;
+ }
+
+ public int getTabletsSize() {
+ return insertTabletPlanList.size();
+ }
+
+ public Map<Integer, TSStatus> getResults() {
+ return results;
+ }
+
+ /**
+ * @return the total row of the whole InsertTabletPlan
+ */
+ public int getTotalRowCount() {
+ int rowCount = 0;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ rowCount += insertTabletPlan.getRowCount();
+ }
+ return rowCount;
+ }
+
+ /**
+ * Gets the number of rows in the InsertTabletPlan of the index, zero will be returned if the
+ * index is out of bound.
+ *
+ * @param index the index of the insertTabletPlanList
+ * @return the total row count of the insertTabletPlanList.get(i)
+ */
+ public int getRowCount(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return 0;
+ }
+ return insertTabletPlanList.get(index).getRowCount();
+ }
+
+ public PartialPath getFirstDeviceId() {
+ return insertTabletPlanList.get(0).getDeviceId();
+ }
+
+ public InsertTabletPlan getInsertTabletPlan(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return null;
+ }
+ return insertTabletPlanList.get(index);
+ }
+
+ /**
+ * @param index the index of the sub plan in this InsertMultiTabletPlan
+ * @return the parent's index in the parent InsertMultiTabletPlan
+ */
+ public int getParentIndex(int index) {
+ if (index >= parentInsertTabletPlanIndexList.size() || index < 0) {
+ return -1;
+ }
+ return parentInsertTabletPlanIndexList.get(index);
+ }
+
+ @Override
+ public void checkIntegrity() throws QueryProcessException {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.checkIntegrity();
+ }
+ }
+
+ public void setParentInsertTabletPlanIndexList(List<Integer> parentInsertTabletPlanIndexList) {
+ this.parentInsertTabletPlanIndexList = parentInsertTabletPlanIndexList;
+ }
+
+ public void setInsertTabletPlanList(List<InsertTabletPlan> insertTabletPlanList) {
+ this.insertTabletPlanList = insertTabletPlanList;
+ }
+
+ public void setResults(
+ Map<Integer, TSStatus> results) {
+ this.results = results;
+ }
+
+ @Override
+ public void recoverFromFailure() {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.recoverFromFailure();
+ }
+ }
+
+ @Override
+ public void serialize(ByteBuffer buffer) {
+ int type = PhysicalPlanType.MULTI_BATCH_INSERT.ordinal();
+ buffer.put((byte) type);
+ buffer.putInt(insertTabletPlanList.size());
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.subSerialize(buffer);
+ }
+
+ buffer.putInt(parentInsertTabletPlanIndexList.size());
Review comment:
Thanks, I'll fix it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jt2594838 commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r556995766
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/query/ClusterPlanRouter.java
##########
@@ -155,6 +158,78 @@ private PartitionGroup routePlan(ShowChildPathsPlan plan) {
return Collections.singletonMap(plan, partitionGroup);
}
+ /**
+ * @param plan InsertMultiTabletPlan
+ * @return key is InsertMultiTabletPlan, value is the partition group the plan belongs to, all
+ * InsertTabletPlans in InsertMultiTabletPlan belongs to one same storage group.
+ */
+ private Map<PhysicalPlan, PartitionGroup> splitAndRoutePlan(InsertMultiTabletPlan plan)
+ throws MetadataException {
+ /*
+ * the key of pgSgPathPlanMap is the partition group; the value is one map,
+ * the key of the map is storage group, the value is the InsertMultiTabletPlan,
+ * all InsertTabletPlans in InsertMultiTabletPlan belongs to one same storage group.
+ */
+ Map<PartitionGroup, Map<PartialPath, InsertMultiTabletPlan>> pgSgPathPlanMap = new HashMap<>();
+ for (int i = 0; i < plan.getInsertTabletPlanList().size(); i++) {
+ InsertTabletPlan insertTabletPlan = plan.getInsertTabletPlanList().get(i);
+ Map<PhysicalPlan, PartitionGroup> tmpResult = splitAndRoutePlan(insertTabletPlan);
+ for (Map.Entry<PhysicalPlan, PartitionGroup> entry : tmpResult.entrySet()) {
+ //1. handle the value returned by call splitAndRoutePlan(InsertTabletPlan)
+ InsertTabletPlan tmpPlan = (InsertTabletPlan) entry.getKey();
+ PartitionGroup tmpPg = entry.getValue();
+ //1.1 the sg that the plan(actually calculated based on device) belongs to
+ PartialPath tmpSgPath = IoTDB.metaManager.getStorageGroupPath(tmpPlan.getDeviceId());
+ Map<PartialPath, InsertMultiTabletPlan> sgPathPlanMap = pgSgPathPlanMap.get(tmpPg);
+ if (sgPathPlanMap == null) {
+ // 2.1 construct the InsertMultiTabletPlan
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ InsertMultiTabletPlan insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+
+ // 2.2 construct the sgPathPlanMap
+ sgPathPlanMap = new HashMap<>();
+ sgPathPlanMap.put(tmpSgPath, insertMultiTabletPlan);
+
+ // 2.3 put the sgPathPlanMap to the pgSgPathPlanMap
+ pgSgPathPlanMap.put(tmpPg, sgPathPlanMap);
+ } else {
+ InsertMultiTabletPlan insertMultiTabletPlan = sgPathPlanMap.get(tmpSgPath);
+ if (insertMultiTabletPlan == null) {
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+
+ // 2.4 put the sgPathPlanMap to the pgSgPathPlanMap
+ sgPathPlanMap.put(tmpSgPath, insertMultiTabletPlan);
+ pgSgPathPlanMap.put(tmpPg, sgPathPlanMap);
Review comment:
This `put` seems unnecessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jt2594838 merged pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
jt2594838 merged pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757119784
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759159680
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759159680
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757073453
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r554983918
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/query/ClusterPlanRouter.java
##########
@@ -155,6 +158,40 @@ private PartitionGroup routePlan(ShowChildPathsPlan plan) {
return Collections.singletonMap(plan, partitionGroup);
}
+ private Map<PhysicalPlan, PartitionGroup> splitAndRoutePlan(InsertMultiTabletPlan plan)
+ throws MetadataException {
+ Map<PartitionGroup, InsertMultiTabletPlan> pgInsertMultiTabletPlanMap = new HashMap<>();
+ for (int i = 0; i < plan.getInsertTabletPlanList().size(); i++) {
+ InsertTabletPlan insertTabletPlan = plan.getInsertTabletPlanList().get(i);
+ Map<PhysicalPlan, PartitionGroup> tmpResult = splitAndRoutePlan(insertTabletPlan);
+ for (Map.Entry<PhysicalPlan, PartitionGroup> entry : tmpResult.entrySet()) {
+ InsertTabletPlan tmpPlan = (InsertTabletPlan) entry.getKey();
+ PartitionGroup tmpPg = entry.getValue();
+
+ InsertMultiTabletPlan pgPlan = pgInsertMultiTabletPlanMap.get(tmpPg);
+ if (pgPlan == null) {
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ InsertMultiTabletPlan insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+ pgInsertMultiTabletPlanMap.put(tmpPg, insertMultiTabletPlan);
+ } else {
+ pgPlan.addInsertTabletPlan(tmpPlan, i);
+ pgInsertMultiTabletPlanMap.put(tmpPg, pgPlan);
Review comment:
its unnecessary
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757411187
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757073902
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757592002
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-756623738
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759139196
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='1.5%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [1.5% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r555008256
##########
File path: server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java
##########
@@ -1560,29 +1551,68 @@ public TSStatus insertTablets(TSInsertTabletsReq req) {
}
}
- public List<TSStatus> insertTabletsInternal(TSInsertTabletsReq req) throws IllegalPathException {
- List<TSStatus> statusList = new ArrayList<>();
+ private InsertTabletPlan constructInsertTabletPlan(TSInsertTabletsReq req, int i)
+ throws IllegalPathException {
+ InsertTabletPlan insertTabletPlan = new InsertTabletPlan(
+ new PartialPath(req.deviceIds.get(i)),
+ req.measurementsList.get(i));
+ insertTabletPlan.setTimes(
+ QueryDataSetUtils.readTimesFromBuffer(req.timestampsList.get(i), req.sizeList.get(i)));
+ insertTabletPlan.setColumns(
+ QueryDataSetUtils.readValuesFromBuffer(
+ req.valuesList.get(i), req.typesList.get(i), req.measurementsList.get(i).size(),
+ req.sizeList.get(i)));
+ insertTabletPlan.setRowCount(req.sizeList.get(i));
+ insertTabletPlan.setDataTypes(req.typesList.get(i));
+ return insertTabletPlan;
+ }
+ /**
+ * process the TSInsertTabletsReq by Split it into many InsertTabletPlan
+ */
+ public TSStatus insertTabletsInternalV1(TSInsertTabletsReq req) throws IllegalPathException {
Review comment:
I originally wanted to keep this version, I'll remove it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-758355971
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan edited a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan edited a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-762230593
The test code is shown as below, and when called `insertTabletsSecondFailed()`, the following error will be occurred because of datatype not match.
[test_code.txt](https://github.com/apache/iotdb/files/5833065/test_code.txt)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jt2594838 commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r554661270
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/applier/AsyncDataLogApplier.java
##########
@@ -131,7 +132,10 @@ private PartialPath getPlanKey(PhysicalPlan plan) throws StorageGroupNotSetExcep
private PartialPath getPlanSG(PhysicalPlan plan) throws StorageGroupNotSetException {
PartialPath sgPath = null;
- if (plan instanceof InsertPlan) {
+ if (plan instanceof InsertMultiTabletPlan) {
+ PartialPath deviceId = ((InsertMultiTabletPlan) plan).getFirstDeviceId();
+ sgPath = IoTDB.metaManager.getStorageGroupPath(deviceId);
+ } else if (plan instanceof InsertPlan) {
Review comment:
Applying InertMultiTabletPlan parallelly with other insertions is dangerous, for example:
There are two plans, one InsertMultiTabletPlan and one InsertTabletPlan, and the InsertMultiTabletPlan contains 2 insertions: insert timestamp=0,value=0 to root.sg1.s0 and insert timestamp=0,value=0 to root.sg2.s0; while the InsertTabletPlan has only one insertion: insert timestamp=0,value=100 to root.sg2.s0.
Because the first deviceId is used as the plan key, the two plans may be executed in parallel, so the actual order is not guaranteed on different replicas, that is, on NodeA, InsertMultiTabletPlan may be executed lastly so root.sg2.s0 is 0 at timestamp=0, but on NodeB, InsertTabletPlan is executed later so root.sg2.s0 is 100 at timestamp=0, thus data becomes inconsistent.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/query/ClusterPlanRouter.java
##########
@@ -155,6 +158,40 @@ private PartitionGroup routePlan(ShowChildPathsPlan plan) {
return Collections.singletonMap(plan, partitionGroup);
}
+ private Map<PhysicalPlan, PartitionGroup> splitAndRoutePlan(InsertMultiTabletPlan plan)
+ throws MetadataException {
+ Map<PartitionGroup, InsertMultiTabletPlan> pgInsertMultiTabletPlanMap = new HashMap<>();
+ for (int i = 0; i < plan.getInsertTabletPlanList().size(); i++) {
+ InsertTabletPlan insertTabletPlan = plan.getInsertTabletPlanList().get(i);
+ Map<PhysicalPlan, PartitionGroup> tmpResult = splitAndRoutePlan(insertTabletPlan);
+ for (Map.Entry<PhysicalPlan, PartitionGroup> entry : tmpResult.entrySet()) {
+ InsertTabletPlan tmpPlan = (InsertTabletPlan) entry.getKey();
+ PartitionGroup tmpPg = entry.getValue();
+
+ InsertMultiTabletPlan pgPlan = pgInsertMultiTabletPlanMap.get(tmpPg);
+ if (pgPlan == null) {
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ InsertMultiTabletPlan insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+ pgInsertMultiTabletPlanMap.put(tmpPg, insertMultiTabletPlan);
+ } else {
+ pgPlan.addInsertTabletPlan(tmpPlan, i);
+ pgInsertMultiTabletPlanMap.put(tmpPg, pgPlan);
Review comment:
Is this `put` necessary?
##########
File path: server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java
##########
@@ -1560,29 +1551,68 @@ public TSStatus insertTablets(TSInsertTabletsReq req) {
}
}
- public List<TSStatus> insertTabletsInternal(TSInsertTabletsReq req) throws IllegalPathException {
- List<TSStatus> statusList = new ArrayList<>();
+ private InsertTabletPlan constructInsertTabletPlan(TSInsertTabletsReq req, int i)
+ throws IllegalPathException {
+ InsertTabletPlan insertTabletPlan = new InsertTabletPlan(
+ new PartialPath(req.deviceIds.get(i)),
+ req.measurementsList.get(i));
+ insertTabletPlan.setTimes(
+ QueryDataSetUtils.readTimesFromBuffer(req.timestampsList.get(i), req.sizeList.get(i)));
+ insertTabletPlan.setColumns(
+ QueryDataSetUtils.readValuesFromBuffer(
+ req.valuesList.get(i), req.typesList.get(i), req.measurementsList.get(i).size(),
+ req.sizeList.get(i)));
+ insertTabletPlan.setRowCount(req.sizeList.get(i));
+ insertTabletPlan.setDataTypes(req.typesList.get(i));
+ return insertTabletPlan;
+ }
+ /**
+ * process the TSInsertTabletsReq by Split it into many InsertTabletPlan
+ */
+ public TSStatus insertTabletsInternalV1(TSInsertTabletsReq req) throws IllegalPathException {
Review comment:
If v1 is used nowhere, it should be removed.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is -1;
+ * <p>
+ * suppose we original have 3 InsertTabletPlans contains 1 InsertMultiTabletPlan, then the initial
+ * InsertMultiTabletPlan would have the following two attributes:
Review comment:
original -> originally
contains -> in
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is -1;
+ * <p>
+ * suppose we original have 3 InsertTabletPlans contains 1 InsertMultiTabletPlan, then the initial
+ * InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={-1,-1,-1} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlan, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlan, InsertTabletPlan_3 is split into four sub InsertTabletPlan.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belongs to two different raft data group, so will generate two new
+ * InsertMultiTabletPlan
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsetTablePlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsetTablePlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsetTablePlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsetTablePlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsetTablePlanIndexList = parentInsetTablePlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsetTablePlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsetTablePlanIndexList() {
+ return parentInsetTablePlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMinTime()) {
+ maxTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return maxTime;
+ }
+
+ public int getTabletsSize() {
+ return insertTabletPlanList.size();
+ }
+
+ public Map<Integer, TSStatus> getResults() {
+ return results;
+ }
+
+ /**
+ * @return the total row of the whole InsertTabletPlan
+ */
+ public int getTotalRowCount() {
+ int rowCount = 0;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ rowCount += insertTabletPlan.getRowCount();
+ }
+ return rowCount;
+ }
+
+ /**
+ * Gets the number of rows in the InsertTabletPlan of the index
+ *
+ * @param index the index of the insertTabletPlanList
+ * @return the total row count of the insertTabletPlanList.get(i)
Review comment:
Supplement that 0 will be returned if the index is out of bound.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/applier/AsyncDataLogApplier.java
##########
@@ -131,7 +132,7 @@ private PartialPath getPlanKey(PhysicalPlan plan) throws StorageGroupNotSetExcep
private PartialPath getPlanSG(PhysicalPlan plan) throws StorageGroupNotSetException {
PartialPath sgPath = null;
- if (plan instanceof InsertPlan) {
+ if (plan instanceof InsertPlan && !(plan instanceof InsertMultiTabletPlan)) {
Review comment:
This forces InsertMultiTabletPlan to be executed serially with other plans, converts the applying process to a synchronized one, and within one InsertMultiTabletPlan, tablets of different storage groups are executed one by one, so the parallelism is significantly reduced.
I think we can add a limitation that each InsertMultiTabletPlan after splitting will only contain tablets of one storage group, so tablets of different storage groups can be inserted in parallel.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is -1;
+ * <p>
+ * suppose we original have 3 InsertTabletPlans contains 1 InsertMultiTabletPlan, then the initial
+ * InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={-1,-1,-1} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlan, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlan, InsertTabletPlan_3 is split into four sub InsertTabletPlan.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belongs to two different raft data group, so will generate two new
+ * InsertMultiTabletPlan
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsetTablePlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsetTablePlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsetTablePlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsetTablePlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsetTablePlanIndexList = parentInsetTablePlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsetTablePlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsetTablePlanIndexList() {
+ return parentInsetTablePlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMinTime()) {
+ maxTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return maxTime;
+ }
+
+ public int getTabletsSize() {
+ return insertTabletPlanList.size();
+ }
+
+ public Map<Integer, TSStatus> getResults() {
+ return results;
+ }
+
+ /**
+ * @return the total row of the whole InsertTabletPlan
+ */
+ public int getTotalRowCount() {
+ int rowCount = 0;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ rowCount += insertTabletPlan.getRowCount();
+ }
+ return rowCount;
+ }
+
+ /**
+ * Gets the number of rows in the InsertTabletPlan of the index
+ *
+ * @param index the index of the insertTabletPlanList
+ * @return the total row count of the insertTabletPlanList.get(i)
+ */
+ public int getRowCount(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return 0;
+ }
+ return insertTabletPlanList.get(index).getRowCount();
+ }
+
+ public PartialPath getFirstDeviceId() {
+ return insertTabletPlanList.get(0).getDeviceId();
+ }
+
+ public InsertTabletPlan getInsertTabletPlan(int index) {
+ if (index >= insertTabletPlanList.size() || index < 0) {
+ return null;
+ }
+ return insertTabletPlanList.get(index);
+ }
+
+ /**
+ * @param index the index of the sub plan in this InsertMultiTabletPlan
+ * @return the parent's index in the parent InsertMultiTabletPlan
+ */
+ public int getParentIndex(int index) {
+ if (index >= parentInsetTablePlanIndexList.size() || index < 0) {
+ return -1;
+ }
+ return parentInsetTablePlanIndexList.get(index);
+ }
+
+ @Override
+ public void checkIntegrity() throws QueryProcessException {
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ insertTabletPlan.checkIntegrity();
+ }
+ }
+
+ public void setParentInsetTablePlanIndexList(List<Integer> parentInsetTablePlanIndexList) {
Review comment:
parentInsetTablePlanIndexList -> parentInsertTablePlanIndexList
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is -1;
+ * <p>
+ * suppose we original have 3 InsertTabletPlans contains 1 InsertMultiTabletPlan, then the initial
+ * InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={-1,-1,-1} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlan, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlan, InsertTabletPlan_3 is split into four sub InsertTabletPlan.
Review comment:
InsertTabletPlan -> InsertTabletPlans
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is -1;
+ * <p>
+ * suppose we original have 3 InsertTabletPlans contains 1 InsertMultiTabletPlan, then the initial
+ * InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={-1,-1,-1} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlan, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlan, InsertTabletPlan_3 is split into four sub InsertTabletPlan.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belongs to two different raft data group, so will generate two new
+ * InsertMultiTabletPlan
+ * <p>
+ * InsertMultiTabletPlant1.insertTabletPlanList={InsertTabletPlan_1_subPlan1,
+ * InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * InsertMultiTabletPlant1.parentInsetTablePlanIndexList={0,2,2,2}
+ *
+ * <p>
+ * InsertMultiTabletPlant2.insertTabletPlanList={InsertTabletPlan_1_subPlan2,
+ * InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2, InsertTabletPlan_2_subPlan3,
+ * InsertTabletPlan_3_subPlan2}
+ * <p>
+ * InsertMultiTabletPlant2.parentInsetTablePlanIndexList={0,1,1,1,2}
+ *
+ * <p>
+ * this is usually used to back-propagate exceptions to the parent plan without losing their
+ * proper positions.
+ */
+ List<Integer> parentInsetTablePlanIndexList;
+
+ /**
+ * the InsertTabletPlan list
+ */
+ List<InsertTabletPlan> insertTabletPlanList;
+
+ /**
+ * record the result of creation of time series
+ */
+ private Map<Integer, TSStatus> results = new TreeMap<>();
+
+
+ public InsertMultiTabletPlan() {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = new ArrayList<>();
+ this.parentInsetTablePlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsetTablePlanIndexList = new ArrayList<>();
+ }
+
+ public InsertMultiTabletPlan(List<InsertTabletPlan> insertTabletPlanList,
+ List<Integer> parentInsetTablePlanIndexList) {
+ super(OperatorType.MULTI_BATCH_INSERT);
+ this.insertTabletPlanList = insertTabletPlanList;
+ this.parentInsetTablePlanIndexList = parentInsetTablePlanIndexList;
+ }
+
+ public void addInsertTabletPlan(InsertTabletPlan plan, Integer parentIndex) {
+ insertTabletPlanList.add(plan);
+ parentInsetTablePlanIndexList.add(parentIndex);
+ }
+
+ public List<InsertTabletPlan> getInsertTabletPlanList() {
+ return insertTabletPlanList;
+ }
+
+ public List<Integer> getParentInsetTablePlanIndexList() {
+ return parentInsetTablePlanIndexList;
+ }
+
+
+ @Override
+ public List<PartialPath> getPaths() {
+ List<PartialPath> result = new ArrayList<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ result.addAll(insertTabletPlan.getPaths());
+ }
+ return result;
+ }
+
+ @Override
+ public long getMinTime() {
+ long minTime = Long.MAX_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (minTime > insertTabletPlan.getMinTime()) {
+ minTime = insertTabletPlan.getMinTime();
+ }
+ }
+ return minTime;
+ }
+
+ public long getMaxTime() {
+ long maxTime = Long.MIN_VALUE;
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ if (maxTime < insertTabletPlan.getMinTime()) {
+ maxTime = insertTabletPlan.getMinTime();
Review comment:
insertTabletPlan.getMinTime() -> insertTabletPlan.getMaxTime()
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.qp.physical.crud;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator.OperatorType;
+import org.apache.iotdb.service.rpc.thrift.TSStatus;
+
+/**
+ * Mainly used in the distributed version, when multiple InsertTabletPlans belong to a raft
+ * replication group, we merge these InsertTabletPlans into one InsertMultiTabletPlan, which can
+ * reduce the number of raft logs. For details, please refer to https://issues.apache.org/jira/browse/IOTDB-1099
+ */
+public class InsertMultiTabletPlan extends InsertPlan {
+
+ /**
+ * the value is used to indict the parent InsertTabletPlan's index when the parent
+ * InsertTabletPlan is split to multi sub InsertTabletPlans. if the InsertTabletPlan have no
+ * parent plan, the value is -1;
+ * <p>
+ * suppose we original have 3 InsertTabletPlans contains 1 InsertMultiTabletPlan, then the initial
+ * InsertMultiTabletPlan would have the following two attributes:
+ * <p>
+ * insertTabletPlanList={InsertTabletPlan_1,InsertTabletPlan_2,InsertTabletPlan_3}
+ * <p>
+ * parentInsetTablePlanIndexList={-1,-1,-1} both have three values.
+ *
+ * <p>
+ * if the InsertTabletPlan_1 is split into two sub InsertTabletPlan, InsertTabletPlan_2 is split
+ * into three sub InsertTabletPlan, InsertTabletPlan_3 is split into four sub InsertTabletPlan.
+ * <p>
+ * InsertTabletPlan_1={InsertTabletPlan_1_subPlan1, InsertTabletPlan_1_subPlan2}
+ *
+ * <p>
+ * InsertTabletPlan_2={InsertTabletPlan_2_subPlan1, InsertTabletPlan_2_subPlan2,
+ * InsertTabletPlan_2_subPlan3}
+ *
+ * <p>
+ * InsertTabletPlan_3={InsertTabletPlan_3_subPlan1, InsertTabletPlan_3_subPlan2,
+ * InsertTabletPlan_3_subPlan3, InsertTabletPlan_3_subPlan4}
+ * <p>
+ * those sub plans belongs to two different raft data group, so will generate two new
+ * InsertMultiTabletPlan
Review comment:
belongs -> belong
group -> groups
InsertMultiTabletPlan -> InsertMultiTabletPlans
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759139196
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='1.5%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [1.5% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757933395
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-756623738
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C.png' alt='C' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-762230593
test code is the above, and when called `insertTabletsSecondFailed()`, the following error will be occurred because of datatype not match.
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
import org.apache.iotdb.session.SessionDataSet;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.utils.Binary;
import org.apache.iotdb.tsfile.write.record.Tablet;
import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
public class SessionExample {
private static Session session;
private static final String ROOT_SG1_D1 = "root.sg1.d1";
public static void main(String[] args)
throws IoTDBConnectionException, StatementExecutionException {
session = new Session("127.0.0.1", 55560, "root", "root");
session.open(false);
//set session fetchSize
session.setFetchSize(10000);
insertTabletsFirstSuccess();
insertTabletsSecondFailed();
query();
session.close();
}
private static void query() throws IoTDBConnectionException, StatementExecutionException {
SessionDataSet dataSet = session.executeQueryStatement("select * from root");
System.out.println(dataSet.getColumnNames());
dataSet.setFetchSize(1024); // default is 10000
while (dataSet.hasNext()) {
System.out.println(dataSet.next());
}
dataSet.closeOperationHandle();
}
private static void insertTabletsFirstSuccess()
throws IoTDBConnectionException, StatementExecutionException {
// The schema of measurements of one device
// only measurementId and data type in MeasurementSchema take effects in Tablet
List<MeasurementSchema> schemaList = new ArrayList<>();
schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s3", TSDataType.INT64));
Tablet tablet1 = new Tablet(ROOT_SG1_D1, schemaList, 100);
Tablet tablet2 = new Tablet("root.sg1.d2", schemaList, 100);
Tablet tablet3 = new Tablet("root.sg1.d3", schemaList, 100);
Map<String, Tablet> tabletMap = new HashMap<>();
tabletMap.put(ROOT_SG1_D1, tablet1);
tabletMap.put("root.sg1.d2", tablet2);
tabletMap.put("root.sg1.d3", tablet3);
//Method 1 to add tablet data
long timestamp = System.currentTimeMillis();
for (long row = 0; row < 100; row++) {
int row1 = tablet1.rowSize++;
int row2 = tablet2.rowSize++;
int row3 = tablet3.rowSize++;
tablet1.addTimestamp(row1, timestamp);
tablet2.addTimestamp(row2, timestamp);
tablet3.addTimestamp(row3, timestamp);
for (int i = 0; i < 3; i++) {
long value = new Random().nextLong();
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, value);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, value);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, value);
}
if (tablet1.rowSize == tablet1.getMaxRowNumber()) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
timestamp++;
}
if (tablet1.rowSize != 0) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
}
private static void insertTabletsSecondFailed()
throws IoTDBConnectionException, StatementExecutionException {
// The schema of measurements of one device
// only measurementId and data type in MeasurementSchema take effects in Tablet
List<MeasurementSchema> schemaList = new ArrayList<>();
schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s3", TSDataType.TEXT));
Tablet tablet1 = new Tablet(ROOT_SG1_D1, schemaList, 100);
Tablet tablet2 = new Tablet("root.sg1.d2", schemaList, 100);
Tablet tablet3 = new Tablet("root.sg1.d3", schemaList, 100);
Map<String, Tablet> tabletMap = new HashMap<>();
tabletMap.put(ROOT_SG1_D1, tablet1);
tabletMap.put("root.sg1.d2", tablet2);
tabletMap.put("root.sg1.d3", tablet3);
//Method 1 to add tablet data
long timestamp = System.currentTimeMillis();
for (long row = 0; row < 100; row++) {
int row1 = tablet1.rowSize++;
int row2 = tablet2.rowSize++;
int row3 = tablet3.rowSize++;
tablet1.addTimestamp(row1, timestamp);
tablet2.addTimestamp(row2, timestamp);
tablet3.addTimestamp(row3, timestamp);
for (int i = 0; i < 3; i++) {
long value = new Random().nextLong();
if (i == 2) {
Binary b = new Binary(value + "-str");
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, b);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, b);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, b);
} else {
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, value);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, value);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, value);
}
}
if (tablet1.rowSize == tablet1.getMaxRowNumber()) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
timestamp++;
}
if (tablet1.rowSize != 0) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
}
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan edited a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan edited a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-762230593
The test code is shown as below, and when called `insertTabletsSecondFailed()`, the following error will be occurred because of datatype not match.
`
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iotdb;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.apache.iotdb.rpc.IoTDBConnectionException;
import org.apache.iotdb.rpc.StatementExecutionException;
import org.apache.iotdb.session.Session;
import org.apache.iotdb.session.SessionDataSet;
import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
import org.apache.iotdb.tsfile.utils.Binary;
import org.apache.iotdb.tsfile.write.record.Tablet;
import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
public class SessionExample {
private static Session session;
private static final String ROOT_SG1_D1 = "root.sg1.d1";
public static void main(String[] args)
throws IoTDBConnectionException, StatementExecutionException {
session = new Session("127.0.0.1", 55560, "root", "root");
session.open(false);
//set session fetchSize
session.setFetchSize(10000);
insertTabletsFirstSuccess();
insertTabletsSecondFailed();
query();
session.close();
}
private static void query() throws IoTDBConnectionException, StatementExecutionException {
SessionDataSet dataSet = session.executeQueryStatement("select * from root");
System.out.println(dataSet.getColumnNames());
dataSet.setFetchSize(1024); // default is 10000
while (dataSet.hasNext()) {
System.out.println(dataSet.next());
}
dataSet.closeOperationHandle();
}
private static void insertTabletsFirstSuccess()
throws IoTDBConnectionException, StatementExecutionException {
// The schema of measurements of one device
// only measurementId and data type in MeasurementSchema take effects in Tablet
List<MeasurementSchema> schemaList = new ArrayList<>();
schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s3", TSDataType.INT64));
Tablet tablet1 = new Tablet(ROOT_SG1_D1, schemaList, 100);
Tablet tablet2 = new Tablet("root.sg1.d2", schemaList, 100);
Tablet tablet3 = new Tablet("root.sg1.d3", schemaList, 100);
Map<String, Tablet> tabletMap = new HashMap<>();
tabletMap.put(ROOT_SG1_D1, tablet1);
tabletMap.put("root.sg1.d2", tablet2);
tabletMap.put("root.sg1.d3", tablet3);
//Method 1 to add tablet data
long timestamp = System.currentTimeMillis();
for (long row = 0; row < 100; row++) {
int row1 = tablet1.rowSize++;
int row2 = tablet2.rowSize++;
int row3 = tablet3.rowSize++;
tablet1.addTimestamp(row1, timestamp);
tablet2.addTimestamp(row2, timestamp);
tablet3.addTimestamp(row3, timestamp);
for (int i = 0; i < 3; i++) {
long value = new Random().nextLong();
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, value);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, value);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, value);
}
if (tablet1.rowSize == tablet1.getMaxRowNumber()) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
timestamp++;
}
if (tablet1.rowSize != 0) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
}
private static void insertTabletsSecondFailed()
throws IoTDBConnectionException, StatementExecutionException {
// The schema of measurements of one device
// only measurementId and data type in MeasurementSchema take effects in Tablet
List<MeasurementSchema> schemaList = new ArrayList<>();
schemaList.add(new MeasurementSchema("s1", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s2", TSDataType.INT64));
schemaList.add(new MeasurementSchema("s3", TSDataType.TEXT));
Tablet tablet1 = new Tablet(ROOT_SG1_D1, schemaList, 100);
Tablet tablet2 = new Tablet("root.sg1.d2", schemaList, 100);
Tablet tablet3 = new Tablet("root.sg1.d3", schemaList, 100);
Map<String, Tablet> tabletMap = new HashMap<>();
tabletMap.put(ROOT_SG1_D1, tablet1);
tabletMap.put("root.sg1.d2", tablet2);
tabletMap.put("root.sg1.d3", tablet3);
//Method 1 to add tablet data
long timestamp = System.currentTimeMillis();
for (long row = 0; row < 100; row++) {
int row1 = tablet1.rowSize++;
int row2 = tablet2.rowSize++;
int row3 = tablet3.rowSize++;
tablet1.addTimestamp(row1, timestamp);
tablet2.addTimestamp(row2, timestamp);
tablet3.addTimestamp(row3, timestamp);
for (int i = 0; i < 3; i++) {
long value = new Random().nextLong();
if (i == 2) {
Binary b = new Binary(value + "-str");
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, b);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, b);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, b);
} else {
tablet1.addValue(schemaList.get(i).getMeasurementId(), row1, value);
tablet2.addValue(schemaList.get(i).getMeasurementId(), row2, value);
tablet3.addValue(schemaList.get(i).getMeasurementId(), row3, value);
}
}
if (tablet1.rowSize == tablet1.getMaxRowNumber()) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
timestamp++;
}
if (tablet1.rowSize != 0) {
session.insertTablets(tabletMap, true);
tablet1.reset();
tablet2.reset();
tablet3.reset();
}
}
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757592002
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-759892476
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#discussion_r557008867
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/query/ClusterPlanRouter.java
##########
@@ -155,6 +158,78 @@ private PartitionGroup routePlan(ShowChildPathsPlan plan) {
return Collections.singletonMap(plan, partitionGroup);
}
+ /**
+ * @param plan InsertMultiTabletPlan
+ * @return key is InsertMultiTabletPlan, value is the partition group the plan belongs to, all
+ * InsertTabletPlans in InsertMultiTabletPlan belongs to one same storage group.
+ */
+ private Map<PhysicalPlan, PartitionGroup> splitAndRoutePlan(InsertMultiTabletPlan plan)
+ throws MetadataException {
+ /*
+ * the key of pgSgPathPlanMap is the partition group; the value is one map,
+ * the key of the map is storage group, the value is the InsertMultiTabletPlan,
+ * all InsertTabletPlans in InsertMultiTabletPlan belongs to one same storage group.
+ */
+ Map<PartitionGroup, Map<PartialPath, InsertMultiTabletPlan>> pgSgPathPlanMap = new HashMap<>();
+ for (int i = 0; i < plan.getInsertTabletPlanList().size(); i++) {
+ InsertTabletPlan insertTabletPlan = plan.getInsertTabletPlanList().get(i);
+ Map<PhysicalPlan, PartitionGroup> tmpResult = splitAndRoutePlan(insertTabletPlan);
+ for (Map.Entry<PhysicalPlan, PartitionGroup> entry : tmpResult.entrySet()) {
+ //1. handle the value returned by call splitAndRoutePlan(InsertTabletPlan)
+ InsertTabletPlan tmpPlan = (InsertTabletPlan) entry.getKey();
+ PartitionGroup tmpPg = entry.getValue();
+ //1.1 the sg that the plan(actually calculated based on device) belongs to
+ PartialPath tmpSgPath = IoTDB.metaManager.getStorageGroupPath(tmpPlan.getDeviceId());
+ Map<PartialPath, InsertMultiTabletPlan> sgPathPlanMap = pgSgPathPlanMap.get(tmpPg);
+ if (sgPathPlanMap == null) {
+ // 2.1 construct the InsertMultiTabletPlan
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ InsertMultiTabletPlan insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+
+ // 2.2 construct the sgPathPlanMap
+ sgPathPlanMap = new HashMap<>();
+ sgPathPlanMap.put(tmpSgPath, insertMultiTabletPlan);
+
+ // 2.3 put the sgPathPlanMap to the pgSgPathPlanMap
+ pgSgPathPlanMap.put(tmpPg, sgPathPlanMap);
+ } else {
+ InsertMultiTabletPlan insertMultiTabletPlan = sgPathPlanMap.get(tmpSgPath);
+ if (insertMultiTabletPlan == null) {
+ List<InsertTabletPlan> insertTabletPlanList = new ArrayList<>();
+ List<Integer> parentInsetTablePlanIndexList = new ArrayList<>();
+ insertTabletPlanList.add(tmpPlan);
+ parentInsetTablePlanIndexList.add(i);
+ insertMultiTabletPlan = new InsertMultiTabletPlan(
+ insertTabletPlanList, parentInsetTablePlanIndexList);
+
+ // 2.4 put the sgPathPlanMap to the pgSgPathPlanMap
+ sgPathPlanMap.put(tmpSgPath, insertMultiTabletPlan);
+ pgSgPathPlanMap.put(tmpPg, sgPathPlanMap);
Review comment:
Thanks, have removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] removed a comment on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-757119784
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-758355971
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] sonarcloud[bot] commented on pull request #2454: [IOTDB-1099] Optimize insertablets logic in cluster module
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #2454:
URL: https://github.com/apache/iotdb/pull/2454#issuecomment-758360991
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=2454&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=2454&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org