You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@iotdb.apache.org by "HTHou (via GitHub)" <gi...@apache.org> on 2023/07/17 07:13:10 UTC

[GitHub] [iotdb] HTHou opened a new pull request, #10566: Optimize the speed of delete by pattern

HTHou opened a new pull request, #10566:
URL: https://github.com/apache/iotdb/pull/10566

   ## Description
   
   When IoTDB has data of 100,000 devices, 20k devices in each tsfile, 163 tsfiles total, execute a delete sql like `delete from root.db.** where time<1970-1-01T08:01:01.000+08:00`, while no data actually matches the time filter.  The execution will takes over 20 hours. 
   
   The reason of this issue is the process of matching DeviceId in TsFileResource and the pattern `root.db.**` takes too match time. For this scenario , it will match 20k * 163 = 3,260,000 times.
   
   This PR is fixing the above issue. We can use a hashset to record the match deviceId info for this delete operation. The maximum matching times will be reduced to the device number 100,000.
   
   Plus, we can use a file level time filter to skip the device check before the deviceId matching.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] xingtanzjr merged pull request #10566: Optimize the speed of delete by pattern

Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.

xingtanzjr merged PR #10566:
URL: https://github.com/apache/iotdb/pull/10566


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] coveralls commented on pull request #10566: Optimize the speed of delete by pattern

Posted by "coveralls (via GitHub)" <gi...@apache.org>.

coveralls commented on PR #10566:
URL: https://github.com/apache/iotdb/pull/10566#issuecomment-1637559252

   
   [![Coverage Status](https://coveralls.io/builds/61377802/badge)](https://coveralls.io/builds/61377802)
   
   coverage: 47.888% (-0.002%) from 47.89% when pulling **e7d7679b27ab1a597128c703106906b86b22f18a on optimize_delete_by_pattern** into **c8c545a8f0c146f707d0c2a0f5995f32df8693ed on master**.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [iotdb] sonarcloud[bot] commented on pull request #10566: Optimize the speed of delete by pattern

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #10566:
URL: https://github.com/apache/iotdb/pull/10566#issuecomment-1637686324

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=10566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=10566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=10566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=10566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=10566&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=10566&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=10566&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=10566&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=10566&metric=new_duplicated_lines_density&view=list)
   
   
   
   ![idea](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png 'idea') Catch issues before they fail your Quality Gate with our IDE extension ![sonarlint](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png 'sonarlint') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org