You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "rickchengx (via GitHub)" <gi...@apache.org> on 2023/05/05 06:01:30 UTC
[GitHub] [dolphinscheduler] rickchengx opened a new pull request, #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
rickchengx opened a new pull request, #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
* close: #14052
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
There is no need to perform 2 regular matches on the same line
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS merged pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS merged PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mergeable[bot] commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "mergeable[bot] (via GitHub)" <gi...@apache.org>.
mergeable[bot] commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535747184
#### :x: Error Occurred while executing an Action
If you believe this is an unexpected error, please report it on our issue tracker: https://github.com/mergeability/mergeable/issues/new
##### Error Details
--------------------
HttpError: Invalid request.
No subschema in "anyOf" matched.
For 'anyOf/0', {"labels"=>{"labels"=>["backend"]}} is not an array.
For 'anyOf/1', {"labels"=>{"labels"=>["backend"]}} is not an array.
For 'properties/labels', {"labels"=>["backend"]} is not an array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535887338
> I'm not sure whether filter first and then forEach aims to avoid branch prediction overhead, but it seems that two regular matches cost a lot, either. cc @ruanwenjun
Hi, @Radeity , thanks for you comment
The `filter()` also needs to make regular judgments and processing for each line. I am not sure what branch prediction you are referring to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535890444
> I am not sure what branch prediction you are referring to.
I refer to `if (matcher.find())` in forEach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535897464
> > I am not sure what branch prediction you are referring to.
>
> I refer to `if (matcher.find())` in forEach
If there are a total of `N` lines in the log, and `M` lines have yarn app id.
Then the previous complexity is `O(N+M)`, and now is `O(N)`, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535777444
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14053)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_coverage&view=list) [100.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535775151
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14053)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14053&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_coverage&view=list) [100.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14053&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535924713
> > If there are a total of `N` lines in the log, and `M` lines have yarn app id. Then the previous complexity is `O(N+M)`, and now is `O(N)`, WDYT?
>
> For such a streaming execution, i think complexity should be considered `O(N)`. Maybe we should compare the overhead brought from `N` more times regular matches with wrong branch prediction, i have no ideas which one is more efficient...
I don't think this question is about branch prediction.
Since in any case, there are a total of N+M times `matcher.find()` before, now it becomes N.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535770088
## [Codecov](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14053?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#14053](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14053?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5424c5e) into [dev](https://app.codecov.io/gh/apache/dolphinscheduler/commit/0a025224206916bb2fb80dfabe97f0e419cb886e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0a02522) will **decrease** coverage by `0.02%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 5424c5e differs from pull request most recent head 8496b46. Consider uploading reports for the commit 8496b46 to get more accurate results
```diff
@@ Coverage Diff @@
## dev #14053 +/- ##
============================================
- Coverage 38.20% 38.18% -0.02%
+ Complexity 4434 4428 -6
============================================
Files 1220 1220
Lines 42706 42655 -51
Branches 4735 4701 -34
============================================
- Hits 16314 16288 -26
+ Misses 24587 24565 -22
+ Partials 1805 1802 -3
```
| [Impacted Files](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14053?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...lphinscheduler/plugin/task/api/utils/LogUtils.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14053?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS91dGlscy9Mb2dVdGlscy5qYXZh) | `36.58% <100.00%> (-1.07%)` | :arrow_down: |
... and [39 files with indirect coverage changes](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14053/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535930140
> I don't think this question is about branch prediction.
> Since in any case, there are a total of N+M times `matcher.find()` before, now it becomes N.
If only consider about matching times, i agree that use `forEach` only is more efficient, my point is that maybe original way can bring some other benefits, such as avoid branch prediction overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on pull request #14053: [Improvement-14052][Log] Remove the useless filter in getAppIdsFromLogFile
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on PR #14053:
URL: https://github.com/apache/dolphinscheduler/pull/14053#issuecomment-1535916789
> If there are a total of `N` lines in the log, and `M` lines have yarn app id. Then the previous complexity is `O(N+M)`, and now is `O(N)`, WDYT?
For such a streaming execution, i think complexity should be considered `O(N)`. Maybe we should compare the overhead brought from `N` more times regular matches with wrong branch prediction, i have no ideas which one is more efficient...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org