You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/11 12:16:30 UTC
[GitHub] [dolphinscheduler] songxiaohu123 opened a new issue, #11433: 3.0.0-release稳定版本，数据质量问题
songxiaohu123 opened a new issue, #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
3.0.0-release版本 数据质量，检测类型为空值过滤，得到的shell脚本中 sql中包含变量${comparison_name} AS comparison_value,这个导致sql无法正常运行，是bug还是我操作的问题？
### What you expected to happen
可以得到解决
### How to reproduce
1
### Anything else
_No response_
### Version
3.0.0
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11433: 3.0.0-release stable version, data quality issues
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433#issuecomment-1211928073
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
Version 3.0.0-release The data quality, the detection type is null value filtering, and the sql in the obtained shell script contains the variable ${comparison_name} AS comparison_value, which causes the sql to fail to run normally. Is it a bug or a problem with my operation?
### What you expected to happen
can be resolved
### How to reproduce
1
### Anything else
_No response_
### Version
3.0.0
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11433: 3.0.0-release stable version, data quality issues
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433#issuecomment-1247430818
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] songxiaohu123 commented on issue #11433: 3.0.0-release稳定版本，数据质量问题
Posted by GitBox <gi...@apache.org>.
songxiaohu123 commented on issue #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433#issuecomment-1211911523
who can help me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11433: 3.0.0-release stable version, data quality issues
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433#issuecomment-1211928259
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] songxiaohu123 commented on issue #11433: 3.0.0-release稳定版本，数据质量问题
Posted by GitBox <gi...@apache.org>.
songxiaohu123 commented on issue #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433#issuecomment-1211911158
完整的内容如下
/opt/module/spark-3.0.0/bin/spark-submit --master yarn --deploy-mode cluster --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default --conf spark.yarn.maxAppAttempts=1 /home/jar/dolphinscheduler-data-quality-3.0.1-alpha-SNAPSHOT.jar "{\"name\":\"$t(null_check)\",\"env\":{\"type\":\"batch\",\"config\":null},\"readers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"data_gateway\",\"password\":\"rootmysql\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"output_table\":\"data_gateway_dwg_center_channel\",\"table\":\"dwg_center_channel\",\"url\":\"jdbc:mysql://192.168.101.29:3306/data_gateway?allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\"} }],\"transformers\":[{\"type\":\"sql\",\"config\":{\"index\":1,\"output_table\":\"null_items\",\"sql\":\"SELECT * FROM data_gateway_dwg_center_channel WHERE (channel_name is null or channel_name = '') AND (center_name is not nu
ll)\"} },{\"type\":\"sql\",\"config\":{\"index\":2,\"output_table\":\"null_count\",\"sql\":\"SELECT COUNT(*) AS nulls FROM null_items\"} }],\"writers\":[{\"type\":\"JDBC\",\"config\":{\"database\":\"dolphinscheduler3\",\"password\":\"rootmysql\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"table\":\"t_ds_dq_execute_result\",\"url\":\"jdbc:mysql://192.168.101.29:3306/dolphinscheduler3?useUnicode=true&characterEncoding=UTF-8&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as rule_type,'$t(null_check)' as rule_name,0 as process_definition_id,2 as process_instance_id,2 as task_instance_id,null_count.nulls AS statistics_value,${comparison_name} AS comparison_value,1 AS comparison_type,0 as check_type,10 as threshold,0 as operator,0 as failure_strategy,'hdfs://mycluster:8020/user/ys/data_quality_error_data/0_2_dqtest' as error_output_path,'2022-08-11 17:23:25' as create_time,'2022-08-11 17:23:25' as up
date_time from null_count \"} },{\"type\":\"JDBC\",\"config\":{\"database\":\"dolphinscheduler3\",\"password\":\"rootmysql\",\"driver\":\"com.mysql.cj.jdbc.Driver\",\"user\":\"root\",\"table\":\"t_ds_dq_task_statistics_value\",\"url\":\"jdbc:mysql://192.168.101.29:3306/dolphinscheduler3?useUnicode=true&characterEncoding=UTF-8&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false\",\"sql\":\"select 0 as process_definition_id,2 as task_instance_id,1 as rule_id,'Q9BFKCMZRAA4YYPPEUG8WESGNDASRAVNILHQAW1OJUI=' as unique_code,'null_count.nulls'AS statistics_name,null_count.nulls AS statistics_value,'2022-08-11 17:23:25' as data_time,'2022-08-11 17:23:25' as create_time,'2022-08-11 17:23:25' as update_time from null_count\"} },{\"type\":\"hdfs_file\",\"config\":{\"path\":\"hdfs://mycluster:8020/user/ys/data_quality_error_data/0_2_dqtest\",\"input_table\":\"null_items\"} }]}"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] guodongym commented on issue #11433: 3.0.0-release stable version, data quality issues
Posted by GitBox <gi...@apache.org>.
guodongym commented on issue #11433:
URL: https://github.com/apache/dolphinscheduler/issues/11433#issuecomment-1215092142
@songxiaohu123 Please provide a screenshot of the interface configuration so that I can reproduce the problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org