You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/11/10 08:32:14 UTC

[GitHub] [dolphinscheduler] changebooks opened a new issue, #12858: [Improvement][Task] SqlTask 改变正则替换 ${} 和 !{} 的顺序

changebooks opened a new issue, #12858:
URL: https://github.com/apache/dolphinscheduler/issues/12858

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   SqlBinds org.apache.dolphinscheduler.plugin.task.sql.SqlTask::getSqlAndSqlParamsMap(String sql)
   
   简单描述修改内容：
   将 replace !{} 的代码：sql = replaceOriginalValue(sql, rgexo, paramsMap);
   移动到 replace ${} 的代码前面：setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap,taskExecutionContext.getTaskInstanceId());
   
   简单描述业务场景：
   SQL语句：INSERT OVERWRITE TABLE label_1 PARTITION (ds = '${date}') !{sql}
   参数：sql = "SELECT user_id FROM mbr_db.share_friend_num_dt_g WHERE ds = '\\${date}' AND group_id = 1 AND num > 0"
   注：参数 sql 中也有 ${date}
   
   先替换 !{sql} ，替换后SQL语句：INSERT OVERWRITE TABLE label_1 PARTITION (ds = '${date}') SELECT user_id FROM mbr_db.share_friend_num_dt_g WHERE ds = '${date}' AND group_id = 1 AND num > 0
   再分析${date}，可确保 sql 参数中的 ${date} 也能被处理到。
   
   反过来，先分析${date}，再替换!{sql}，导致sql中的${date}未被处理。
   
   已亲测
   
   老代码：
   `
   //new
   
   //replace variable TIME with $[YYYYmmddd...] in sql when history run job and batch complement job
   
   sql = ParameterUtils.replaceScheduleTime(sql, taskExecutionContext.getScheduleTime());
   
   // special characters need to be escaped, ${} needs to be escaped
   
   setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap,taskExecutionContext.getTaskInstanceId());
   
   //Replace the original value in sql ！{...} ，Does not participate in precompilation
   
   String rgexo = "['\"]*\\!\\{(.*?)\\}['\"]*";
   
   sql = replaceOriginalValue(sql, rgexo, paramsMap);
   
   // replace the ${} of the SQL statement with the Placeholder
   
   String formatSql = sql.replaceAll(rgex, "?");
   
   sqlBuilder.append(formatSql);
   `
   
   新代码：
   `
   //new
   
   //Replace the original value in sql ！{...} ，Does not participate in precompilation
   
   String rgexo = "['\"]*\\!\\{(.*?)\\}['\"]*";
   
   sql = replaceOriginalValue(sql, rgexo, paramsMap);
   
   //replace variable TIME with $[YYYYmmddd...] in sql when history run job and batch complement job
   
   sql = ParameterUtils.replaceScheduleTime(sql, taskExecutionContext.getScheduleTime());
   
   // special characters need to be escaped, ${} needs to be escaped
   
   setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap,taskExecutionContext.getTaskInstanceId());
   
   // replace the ${} of the SQL statement with the Placeholder
   
   String formatSql = sql.replaceAll(rgex, "?");
   
   sqlBuilder.append(formatSql);
   `
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12858:
URL: https://github.com/apache/dolphinscheduler/issues/12858#issuecomment-1363463369

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12858:
URL: https://github.com/apache/dolphinscheduler/issues/12858#issuecomment-1309941347

   Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
   * In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
   * If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12858:
URL: https://github.com/apache/dolphinscheduler/issues/12858#issuecomment-1309941083

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   SqlBinds org.apache.dolphinscheduler.plugin.task.sql.SqlTask::getSqlAndSqlParamsMap(String sql)
   
   Briefly describe the modification content:
   Code to replace !{}: sql = replaceOriginalValue(sql, rgexo, paramsMap);
   Move before the code that replaces ${}: setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap,taskExecutionContext.getTaskInstanceId());
   
   Briefly describe the business scenario:
   SQL statement: INSERT OVERWRITE TABLE label_1 PARTITION (ds = '${date}') !{sql}
   Parameters: sql = "SELECT user_id FROM mbr_db.share_friend_num_dt_g WHERE ds = '\\${date}' AND group_id = 1 AND num > 0"
   Note: There is also ${date} in the parameter sql
   
   Replace !{sql} first, then replace the SQL statement: INSERT OVERWRITE TABLE label_1 PARTITION (ds = '${date}') SELECT user_id FROM mbr_db.share_friend_num_dt_g WHERE ds = '${date}' AND group_id = 1 AND num > 0
   Re-analyze ${date} to ensure that ${date} in the sql parameter can also be processed.
   
   Conversely, analyze ${date} first, and then replace !{sql}, resulting in ${date} in sql not being processed.
   
   Tested
   
   Old code:
   `
   //new
   
   //replace variable TIME with $[YYYYmmddd...] in sql when history run job and batch complement job
   
   sql = ParameterUtils.replaceScheduleTime(sql, taskExecutionContext.getScheduleTime());
   
   // special characters need to be escaped, ${} needs to be escaped
   
   setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap, taskExecutionContext.getTaskInstanceId());
   
   //Replace the original value in sql ! {...} , Does not participate in precompilation
   
   String rgexo = "['\"]*\\!\\{(.*?)\\}['\"]*";
   
   sql = replaceOriginalValue(sql, rgexo, paramsMap);
   
   // replace the ${} of the SQL statement with the Placeholder
   
   String formatSql = sql.replaceAll(rgex, "?");
   
   sqlBuilder.append(formatSql);
   `
   
   New code:
   `
   //new
   
   //Replace the original value in sql ! {...} , Does not participate in precompilation
   
   String rgexo = "['\"]*\\!\\{(.*?)\\}['\"]*";
   
   sql = replaceOriginalValue(sql, rgexo, paramsMap);
   
   //replace variable TIME with $[YYYYmmddd...] in sql when history run job and batch complement job
   
   sql = ParameterUtils.replaceScheduleTime(sql, taskExecutionContext.getScheduleTime());
   
   // special characters need to be escaped, ${} needs to be escaped
   
   setSqlParamsMap(sql, rgex, sqlParamsMap, paramsMap, taskExecutionContext.getTaskInstanceId());
   
   // replace the ${} of the SQL statement with the Placeholder
   
   String formatSql = sql.replaceAll(rgex, "?");
   
   sqlBuilder.append(formatSql);
   `
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] github-actions[bot] closed issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}
URL: https://github.com/apache/dolphinscheduler/issues/12858


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] davidzollo commented on issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}

Posted by GitBox <gi...@apache.org>.
davidzollo commented on issue #12858:
URL: https://github.com/apache/dolphinscheduler/issues/12858#issuecomment-1312513767

   good job
   I am very impressed with your meticulous work, and please change the description into English, English is official language
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12858: [Improvement][Task] SqlTask ​​Change the order of regular replacement ${} and !{}

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12858:
URL: https://github.com/apache/dolphinscheduler/issues/12858#issuecomment-1352394739

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org