You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/16 02:55:01 UTC

[GitHub] [dolphinscheduler] fuchanghai opened a new pull request, #11493: [feature-11378] add hive option

fuchanghai opened a new pull request, #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493

   this close 11378
   
   ## Purpose of the pull request
   Enables users to configure hive tasks in the form of non-JSON in DataX type nodes
   
   ## Brief change log
   
   Enables users to configure hive tasks in the form of non-JSON in DataX type nodes
   
   ## Verify this pull request
   
   
   This pull request is code cleanup without any test coverage.
   
   
   sorry ,i'll rebase after this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1217422292

   @SbloodyS  Because HiveStatementParser using druid uses 'SELECT t.* FROM ( %s ) t WHERE 0 = 1' to parse the column name with a temporary table （t.a,t.b）.so i need replace it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#discussion_r947405671


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java:
##########
@@ -565,7 +565,7 @@ public String[] tryExecuteSqlResolveColumnNames(DbType sourceType, BaseConnectio
             int num = md.getColumnCount();
             columnNames = new String[num];
             for (int i = 1; i <= num; i++) {
-                columnNames[i - 1] = md.getColumnName(i);
+                columnNames[i - 1] = md.getColumnName(i).replaceAll("t.","");

Review Comment:
   I'm not familiar with datax. Can you describe the usage of this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1216097030

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=11493)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#discussion_r949760285


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java:
##########
@@ -565,7 +565,7 @@ public String[] tryExecuteSqlResolveColumnNames(DbType sourceType, BaseConnectio
             int num = md.getColumnCount();
             columnNames = new String[num];
             for (int i = 1; i <= num; i++) {
-                columnNames[i - 1] = md.getColumnName(i);
+                columnNames[i - 1] = md.getColumnName(i).replaceAll("t.","");

Review Comment:
   cc @zhuangchong 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by "fuchanghai (via GitHub)" <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1432682378

   > So,does dolphin support ddl command such as 'create and insert overwrite'?As far as I tested,it failed.
   that is not caused by ds。 I change the codes of DataX。if  you want to know that how to do ,plz  call me by email
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] eshaoliu commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
eshaoliu commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1223708206

   So,does dolphin support ddl command such as 'create and insert overwrite'?As far as I tested,it failed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220274234

   > tryGrammaticalAnalysisSqlColumnNames
   
   <img width="636" alt="image" src="https://user-images.githubusercontent.com/33984497/185550624-a216b662-7f9c-4761-83d2-d64ef75c495a.png">
   
   sql statements is "select * from xxx",after execute the method tryGrammaticalAnalysisSqlColumnNames,the length of columnNames is zero


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1216098418

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=11493)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=11493&resolved=false&types=CODE_SMELL)
   
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=11493&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220285081

   > > @SbloodyS Because HiveStatementParser using druid uses 'SELECT t.* FROM ( %s ) t WHERE 0 = 1' to parse the column name with a temporary table （t.a,t.b）.so i need replace it
   > 
   > @fuchanghai The 'SELECT t.* FROM (%s) t WHERE 0 = 1' statement was executed in the method `tryExecuteSqlResolveColumnNames`, which did not resolve the column names using the HiveStatementParser in the `tryGrammaticalAnalysisSqlColumnNames` method in the previous step. Is there an exception?
   > 
   > https://github.com/apache/dolphinscheduler/blob/f0f9791503c3bdedb5273e9ef5c2896558fe786b/dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java#L464-L471
   
   It looks like I'll also test if 'select *,'1' as b from table' resolves the exception


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] davidzollo merged pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
davidzollo merged PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1216094079

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/11493?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#11493](https://codecov.io/gh/apache/dolphinscheduler/pull/11493?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2cceac0) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/68316f68fa9b90560846c3a12274edec2bf5d960?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (68316f6) will **decrease** coverage by `0.02%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head 2cceac0 differs from pull request most recent head f0f9791. Consider uploading reports for the commit f0f9791 to get more accurate results
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #11493      +/-   ##
   ============================================
   - Coverage     39.37%   39.34%   -0.03%     
   + Complexity     4631     4626       -5     
   ============================================
     Files           987      987              
     Lines         37566    37551      -15     
     Branches       4184     4176       -8     
   ============================================
   - Hits          14790    14775      -15     
   - Misses        21226    21227       +1     
   + Partials       1550     1549       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/11493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../dolphinscheduler/plugin/task/datax/DataxTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stZGF0YXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svZGF0YXgvRGF0YXhUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...dolphinscheduler/plugin/task/datax/DataxUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stZGF0YXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svZGF0YXgvRGF0YXhVdGlscy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...erver/master/processor/queue/TaskEventService.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9wcm9jZXNzb3IvcXVldWUvVGFza0V2ZW50U2VydmljZS5qYXZh) | `69.64% <0.00%> (-10.72%)` | :arrow_down: |
   | [...org/apache/dolphinscheduler/remote/utils/Host.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL3V0aWxzL0hvc3QuamF2YQ==) | `42.55% <0.00%> (-2.13%)` | :arrow_down: |
   | [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `81.96% <0.00%> (-1.64%)` | :arrow_down: |
   | [...e/dolphinscheduler/remote/NettyRemotingClient.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL05ldHR5UmVtb3RpbmdDbGllbnQuamF2YQ==) | `51.38% <0.00%> (-1.39%)` | :arrow_down: |
   | [...r/plugin/task/sqoop/parameter/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Fvb3Avc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3Fvb3AvcGFyYW1ldGVyL1Nxb29wUGFyYW1ldGVycy5qYXZh) | `55.12% <0.00%> (-1.29%)` | :arrow_down: |
   | [...pache/dolphinscheduler/common/utils/DateUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3V0aWxzL0RhdGVVdGlscy5qYXZh) | `74.37% <0.00%> (-0.63%)` | :arrow_down: |
   | [...rver/master/runner/task/BlockingTaskProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvdGFzay9CbG9ja2luZ1Rhc2tQcm9jZXNzb3IuamF2YQ==) | `75.86% <0.00%> (-0.55%)` | :arrow_down: |
   | [...eduler/api/service/impl/DataSourceServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9EYXRhU291cmNlU2VydmljZUltcGwuamF2YQ==) | `27.90% <0.00%> (ø)` | |
   | ... and [11 more](https://codecov.io/gh/apache/dolphinscheduler/pull/11493/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhuangchong commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
zhuangchong commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220267778

   > @SbloodyS Because HiveStatementParser using druid uses 'SELECT t.* FROM ( %s ) t WHERE 0 = 1' to parse the column name with a temporary table （t.a,t.b）.so i need replace it
   
   @fuchanghai 
   The 'SELECT t.* FROM ( %s ) t WHERE 0 = 1' statement is in the method `tryExecuteSqlResolveColumnNames`, In the previous step using HiveStatementParser in the `tryGrammaticalAnalysisSqlColumnNames` method did not resolve the column names. Is there an exception?
   
   https://github.com/apache/dolphinscheduler/blob/f0f9791503c3bdedb5273e9ef5c2896558fe786b/dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java#L464-L471
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] davidzollo commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
davidzollo commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1221186482

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhuangchong commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
zhuangchong commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220354609

   > @zhuangchong Yes, but you gave me some inspiration just now, mainly as follows: Yes, but you gave me some inspiration just now, mainly as follows: I want to try
   > 
   > ```
   > 'select *,'1' as b from table'
   > ```
   > 
   > to see if the exception will be parsed, if there is a value in ColumnNames after calling the method tryGrammaticalAnalysisSql, will it do not parse`“*”`?
   
   What did it parse out?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220431458

   > > @zhuangchong Yes, but you gave me some inspiration just now, mainly as follows: Yes, but you gave me some inspiration just now, mainly as follows: I want to try
   > > ```
   > > 'select *,'1' as b from table'
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > to see if the exception will be parsed, if there is a value in ColumnNames after calling the method tryGrammaticalAnalysisSql, will it do not parse`“*”`?
   > 
   > What did it parse out?
   
   after calling the method tryGrammaticalAnalysisSql，length of ColumnNames  is zero .  finally result is right
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11378] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1216088806

   > Is it the wrong issue? #11378
   
   sorry,I've already changed it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on pull request #11493: [feature-11378] add hive option

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1216087411

   Is it the wrong issue? #11378


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] Amy0104 commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
Amy0104 commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1217399138

   > Is the frontend part missing?
   
   <img width="496" alt="image" src="https://user-images.githubusercontent.com/97265214/185024535-6c0372f1-d2d6-4646-a026-e0beabd64fe8.png">
   The front end part has already supported this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhuangchong commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
zhuangchong commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220299338

   > > tryGrammaticalAnalysisSqlColumnNames
   > 
   > <img alt="image" width="636" src="https://user-images.githubusercontent.com/33984497/185550624-a216b662-7f9c-4761-83d2-d64ef75c495a.png">
   > 
   > sql statements is "select * from xxx",after execute the method tryGrammaticalAnalysisSqlColumnNames,the length of columnNames is zero
   
   
   
   > > tryGrammaticalAnalysisSqlColumnNames
   > 
   > <img alt="image" width="636" src="https://user-images.githubusercontent.com/33984497/185550624-a216b662-7f9c-4761-83d2-d64ef75c495a.png">
   > 
   > sql statements is "select * from xxx",after execute the method tryGrammaticalAnalysisSqlColumnNames,the length of columnNames is zero
   
   As far as I know `'SELECT t.* FROM (%s) t WHERE 0 = 1'` uses jdbc to connect to the postgresql library, and when you execute get `MetaData`, you get the real column name, not t.a; if Hive executes The obtained column name is t.a, and your changes are no problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1220305002

   @zhuangchong Yes, but you gave me some inspiration just now, mainly as follows:
    Yes, but you gave me some inspiration just now, mainly as follows: 
   I want to try 'select *,'1' as b from table' to see if the exception will be parsed, if there is a value in ColumnNames after calling the method tryGrammaticalAnalysisSql, will it do not parse '*'?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#issuecomment-1223912815

   that is not caused by ds。and I change the codes of DataX。
   
   
   
   ---Original---
   From: ***@***.***&gt;
   Date: Tue, Aug 23, 2022 16:06 PM
   To: ***@***.***&gt;;
   Cc: ***@***.******@***.***&gt;;
   Subject: Re: [apache/dolphinscheduler] [feature-11408] add hive option (PR#11493)
   
   
   
   
    
   So,does dolphin support ddl command such as 'create and insert overwrite'?As far as I tested,it failed.
    
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you were mentioned.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on a diff in pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on code in PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#discussion_r947421459


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java:
##########
@@ -565,7 +565,7 @@ public String[] tryExecuteSqlResolveColumnNames(DbType sourceType, BaseConnectio
             int num = md.getColumnCount();
             columnNames = new String[num];
             for (int i = 1; i <= num; i++) {
-                columnNames[i - 1] = md.getColumnName(i);
+                columnNames[i - 1] = md.getColumnName(i).replaceAll("t.","");

Review Comment:
   Because HiveStatementParser using druid uses 'SELECT t.* FROM ( %s ) t WHERE 0 = 1' to parse the column name with a temporary table （t.a,t.b）.so i need replace it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#discussion_r947425997


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java:
##########
@@ -565,7 +565,7 @@ public String[] tryExecuteSqlResolveColumnNames(DbType sourceType, BaseConnectio
             int num = md.getColumnCount();
             columnNames = new String[num];
             for (int i = 1; i <= num; i++) {
-                columnNames[i - 1] = md.getColumnName(i);
+                columnNames[i - 1] = md.getColumnName(i).replaceAll("t.","");

Review Comment:
   If so, we should judge whether is hive type before performing this replacement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] fuchanghai commented on a diff in pull request #11493: [feature-11408] add hive option

Posted by GitBox <gi...@apache.org>.
fuchanghai commented on code in PR #11493:
URL: https://github.com/apache/dolphinscheduler/pull/11493#discussion_r947448133


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java:
##########
@@ -565,7 +565,7 @@ public String[] tryExecuteSqlResolveColumnNames(DbType sourceType, BaseConnectio
             int num = md.getColumnCount();
             columnNames = new String[num];
             for (int i = 1; i <= num; i++) {
-                columnNames[i - 1] = md.getColumnName(i);
+                columnNames[i - 1] = md.getColumnName(i).replaceAll("t.","");

Review Comment:
   'SELECT t.* FROM ( %s ) t WHERE 0 = 1' This statement will execute all jdbc types, that's why I put the repalce statement here @SbloodyS



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org