You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "liuchunhua (via GitHub)" <gi...@apache.org> on 2024/03/05 03:20:27 UTC
[PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
liuchunhua opened a new pull request, #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
## Purpose of the pull request
<!--(For example: This pull request adds checkstyle plugin).-->
resoivres #15668
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)
If your pull request contain incompatible change, you should also add it to `docs/docs/en/guide/upgrede/incompatible.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#issuecomment-1980142276
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_coverage&view=list) (required ≥ 60%)
 [19.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#discussion_r1512099094
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/DataSourceServiceImpl.java:
##########
@@ -499,11 +499,18 @@
}
DatabaseMetaData metaData = connection.getMetaData();
-
- if (dataSource.getType() == DbType.ORACLE) {
- database = null;
+ String schema = null;
+ try {
+ schema = metaData.getConnection().getSchema();
+ } catch (SQLException e) {
+ log.error("Cant not get the schema, datasourceId:{}.", datasourceId, e);
+ throw new ServiceException(Status.GET_DATASOURCE_TABLES_ERROR);
}
- rs = metaData.getColumns(database, null, tableName, "%");
+
+ rs = metaData.getColumns(
+ getCatalog(dataSource.getType(), database),
+ getDbSchemaPattern(dataSource.getType(), database, schema, connectionParam),
+ tableName, "%");
Review Comment:
## Query built from user-controlled sources
This query depends on a [user-provided value](1).
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/3932)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#issuecomment-1977919682
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_coverage&view=list) (required ≥ 60%)
 [19.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#issuecomment-1977915393
## [Codecov](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `19 lines` in your changes are missing coverage. Please review.
> Project coverage is 38.59%. Comparing base [(`04a6b0d`)](https://app.codecov.io/gh/apache/dolphinscheduler/commit/04a6b0d2814c69b056639f65546ffd62ead22321?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`d493ee7`)](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1 commits behind head on dev.
> :exclamation: Current head d493ee7 differs from pull request most recent head 17c4ab1. Consider uploading reports for the commit 17c4ab1 to get more accurate results
| [Files](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...eduler/api/service/impl/DataSourceServiceImpl.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9EYXRhU291cmNlU2VydmljZUltcGwuamF2YQ==) | 0.00% | [18 Missing :warning: ](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../org/apache/dolphinscheduler/spi/enums/DbType.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc3BpL2VudW1zL0RiVHlwZS5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## dev #15671 +/- ##
============================================
- Coverage 38.63% 38.59% -0.04%
+ Complexity 4795 4790 -5
============================================
Files 1316 1316
Lines 44969 44948 -21
Branches 4817 4793 -24
============================================
- Hits 17373 17349 -24
- Misses 25713 25717 +4
+ Partials 1883 1882 -1
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15671?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "liuchunhua (via GitHub)" <gi...@apache.org>.
liuchunhua commented on code in PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#discussion_r1513864607
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/enums/DbType.java:
##########
@@ -88,7 +88,7 @@ public String getDescp() {
}
public boolean isHive() {
- return this == DbType.HIVE;
+ return this == DbType.HIVE || this == DbType.KYUUBI;
Review Comment:
KYUUBI use spark engine to query hive table by default, using HiveReader is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on code in PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#discussion_r1513849238
##########
dolphinscheduler-spi/src/main/java/org/apache/dolphinscheduler/spi/enums/DbType.java:
##########
@@ -88,7 +88,7 @@ public String getDescp() {
}
public boolean isHive() {
- return this == DbType.HIVE;
+ return this == DbType.HIVE || this == DbType.KYUUBI;
Review Comment:
May I ask why KYUUBI is also true in `isHive()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#issuecomment-1977919102
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_coverage&view=list) (required ≥ 60%)
 [19.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15668][Api]Fix data quality cannot get database/columns with kyuubi [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15671:
URL: https://github.com/apache/dolphinscheduler/pull/15671#issuecomment-1980141143
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_coverage&view=list) (required ≥ 60%)
 [19.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15671&metric=new_duplicated_lines_density&view=list) (required ≤ 10%)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15671)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org