You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "ZHCGitHub (via GitHub)" <gi...@apache.org> on 2025/04/11 02:32:23 UTC

[I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

ZHCGitHub opened a new issue, #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   ![Image](https://github.com/user-attachments/assets/2743f0b3-175b-4f9b-a8f6-eb53b82921a3)
   
   ![Image](https://github.com/user-attachments/assets/1c8839b9-cbbc-446e-bc89-0c22172d32d7)
   create sqlserver table error
   
   
   ### What you expected to happen
   
   test_20250411
   
   ### How to reproduce
   
   create table test_${dd}(
       id int null
   );
   
   Custom parameters: dd in integer 20250411
   
   ### Anything else
   
   I can't believe this is an Apache project with millions of downloads. There is no version where existing features can be used without bugs. Even the most basic features have bugs. Many bugs have been raised in Git, but they have not been resolved in many versions. For example, the most basic SQL function, which dynamically creates table names using global parameters, should be a very simple function. As a result, MySQL will pass a global parameter and create a table with an error. Only by creating local parameters in the current task and creating an SQL Server table in the same way, the concatenated parameters are directly added to the table name to generate a table name test_@p0 This table name, along with other miscellaneous bugs, I have tried SQL as a basic feature from 2.1.70 to the latest 3.2.2, and none of them are bug free. I think you should focus on doing the basic functions well first, at least being able to use them normally like a normal project, and then consider the 
 idea of adding other functions and upgrading, as well as the damn official documentation, which is simply unbearable. This makes me feel that this project does not deserve the honor of being a top-level Apache project
   
   ### Version
   
   3.2.x
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2796228249

   This is a history problem, probably caused by the initialized design of SqlTask. We have many datasources, and all of them are using the same task plugin instance, so there are existing some cases that is difficultly to adapt it. e.g. fetch log, split sql...
   
   The long term solution is to split the SqlTask on the backend to adapt different datasource.
   
   Let's back to this case, the main reason is due the the sql task don't directly replace the parameter in the sql, but will use `?` placeholder, we would better keep the simple logic here, directly replace the parameter in sql rather than use `?`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2870400468

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZHCGitHub (via GitHub)" <gi...@apache.org>.
ZHCGitHub commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2796262955

   Yes, you are right. i look forward to DS releasing a stable and bug free new version as soon as possible, and i also hope that DS can go higher and further. Thank you very much&nbsp;
   
   
   ZHC
   ***@***.***
   
   
   
   &nbsp;
   
   
   
   
   ------------------&nbsp;原始邮件&nbsp;------------------
   发件人: "Wenjun ***@***.***&gt;; 
   发送时间: 2025年4月11日(星期五) 下午4:36
   收件人: ***@***.***&gt;; 
   抄送: ***@***.***&gt;; ***@***.***&gt;; 
   主题: Re: [apache/dolphinscheduler] [Bug] [ SQL module] passed variable to create table, parameter passing error (Issue #17124)
   
   
   
   
   
   
   
   
   
   
   
   
      
   This is a history problem, probably caused by the initialized design of SqlTask. We have many datasources, and all of them are using the same task plugin instance, so there are existing some cases that is difficultly to adapt it. e.g. fetch log, split sql...
    
   The long term solution is to split the SqlTask on the backend to adapt different datasource.
    
   Let's back to this case, the main reason is due the the sql task don't directly replace the parameter in the sql, but will use ? placeholder, we would better keep the simple logic here, directly replace the parameter in sql rather than use ?.
   
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;
     ruanwenjun left a comment (apache/dolphinscheduler#17124)
    
   This is a history problem, probably caused by the initialized design of SqlTask. We have many datasources, and all of them are using the same task plugin instance, so there are existing some cases that is difficultly to adapt it. e.g. fetch log, split sql...
    
   The long term solution is to split the SqlTask on the backend to adapt different datasource.
    
   Let's back to this case, the main reason is due the the sql task don't directly replace the parameter in the sql, but will use ? placeholder, we would better keep the simple logic here, directly replace the parameter in sql rather than use ?.
    
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZaneYork (via GitHub)" <gi...@apache.org>.
ZaneYork commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-3560946809

   看了下SQL节点的处理代码,推测出有这两处模板预替换逻辑:
   [内置日期变量](https://github.com/apache/dolphinscheduler/blob/83ac81c59fb82dd31a5935c272122a9a1b327dbe/dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java#L419C18-L419C19)
   [局部变量预替换](https://github.com/apache/dolphinscheduler/blob/83ac81c59fb82dd31a5935c272122a9a1b327dbe/dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java#L439)
   不难得知,使用$[YYYYmmdd]可以直接实现功能,如果需要进行变量加工传递,使用!{dt}进行参数传递可以实现高级替换功能。经过验证,两种都能正确产生最终的sql。
   但是还是得diss一下sql节点的说明文档(或者参数传递那边的文档),通篇没看到有提这个感叹号替换的参数模式,也没提日期替换这种替换模式的特殊性


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2796249907

   > I really hope that DS can become a top-level project for Apache, but you should develop and iterate new features on the basis that there are no functional bugs in the current version. So far, neither 3. x nor 2. x is bug free, and the most basic features cannot be used properly. A project that cannot use existing features properly will not be a successful project after multiple iterations, but will only make later changes more difficult. You should at least come up with a stable version where the basic features can be used normally
   
   Some time ago, the project introduced too many untested and poorly designed features that were difficult to stabilize for a long time, many of which were not reviewed by most of the core maintainers, resulting in a messy codebase, and due to the fact that the core maintainers were all maintain it in their spare time, so the issue processing is slow.
   
   A good change is that code merge has become more strict, which might improve the situation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZHCGitHub (via GitHub)" <gi...@apache.org>.
ZHCGitHub commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2795702604

   <!-- Failed to upload "image.png" -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZHCGitHub (via GitHub)" <gi...@apache.org>.
ZHCGitHub commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2795703215

   ![Image](https://github.com/user-attachments/assets/bb04a42f-e47d-44c6-8b33-cbd189bfb6f6)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZHCGitHub (via GitHub)" <gi...@apache.org>.
ZHCGitHub commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2796196212

   yes,mysql is right,but sqlserver is wrong&nbsp;
   
   
   ZHC
   ***@***.***
   
   
   
   &nbsp;
   
   
   
   
   ------------------&nbsp;原始邮件&nbsp;------------------
   发件人: "Wenjun ***@***.***&gt;; 
   发送时间: 2025年4月11日(星期五) 下午4:17
   收件人: ***@***.***&gt;; 
   抄送: ***@***.***&gt;; ***@***.***&gt;; 
   主题: Re: [apache/dolphinscheduler] [Bug] [ SQL module] passed variable to create table, parameter passing error (Issue #17124)
   
   
   
   
   
   
   
   
   
   
   
   
      
   I use Mysql, this works well on dev branch, not sure if this due to some interesting things in SqlServer, need to test on SqlServer.
   
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;
     ruanwenjun left a comment (apache/dolphinscheduler#17124)
    
   I use Mysql, this works well on dev branch, not sure if this due to some interesting things in SqlServer, need to test on SqlServer.
    
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZHCGitHub (via GitHub)" <gi...@apache.org>.
ZHCGitHub commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2796219855

   I really hope that DS can become a top-level project for Apache, but you should develop and iterate new features on the basis that there are no functional bugs in the current version. So far, neither 3. x nor 2. x is bug free, and the most basic features cannot be used properly. A project that cannot use existing features properly will not be a successful project after multiple iterations, but will only make later changes more difficult. You should at least come up with a stable version where the basic features can be used normally
   
   
   
   
   ZHC
   ***@***.***
   
   
   
   &nbsp;
   
   
   
   
   ------------------&nbsp;原始邮件&nbsp;------------------
   发件人:                                                                                                                        "apache/dolphinscheduler"                                                                                    ***@***.***&gt;;
   发送时间:&nbsp;2025年4月11日(星期五) 下午4:17
   ***@***.***&gt;;
   ***@***.******@***.***&gt;;
   主题:&nbsp;Re: [apache/dolphinscheduler] [Bug] [ SQL module] passed variable to create table, parameter passing error (Issue #17124)
   
   
   
      
   I use Mysql, this works well on dev branch, not sure if this due to some interesting things in SqlServer, need to test on SqlServer.
   
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;
     ruanwenjun left a comment (apache/dolphinscheduler#17124)
    
   I use Mysql, this works well on dev branch, not sure if this due to some interesting things in SqlServer, need to test on SqlServer.
    
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2796181401

   I use Mysql, this works well on dev branch, not sure if this due to some interesting things in SqlServer, need to test on SqlServer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [ SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "ZHCGitHub (via GitHub)" <gi...@apache.org>.
ZHCGitHub commented on issue #17124:
URL: https://github.com/apache/dolphinscheduler/issues/17124#issuecomment-2795702871

   ![Image](https://github.com/user-attachments/assets/2855e51d-005e-4e9e-b8c3-7e8f5021872f)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [SQL module] passed variable to create table, parameter passing error [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #17124: [Bug] [SQL module] passed variable to create table, parameter passing error
URL: https://github.com/apache/dolphinscheduler/issues/17124


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org