You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "jeanleen (via GitHub)" <gi...@apache.org> on 2024/01/20 07:38:58 UTC
[I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]
jeanleen opened a new issue, #6260:
URL: https://github.com/apache/seatunnel/issues/6260
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
first i delete the data from mysql ,then write data ,so i set two jdbc configuration in sink module. when write data to mysql, the job did not end ,run every minute.
### SeaTunnel Version
2.3.3
### SeaTunnel Config
```conf
# Defining the runtime environment
env {
parallelism = 2
# spark.master = "yarn"
# spark.deploy-mode= "cluster"
job.name = ""
# spark.executor.instances = 2
# spark.executor.cores = 2
# spark.executor.memory = "4g"
job.mode = "BATCH"
}
source{
# Jdbc {
#
#
# query = """
# select * from test.users
#
# """
# # query = """SELECT PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD') as SYNC_DATETIME FROM ( SELECT PRO_ID,PRO_BOUGHT,CATE_1,SYNC_DATETIME FROM SMT_GET_PRODUCT ORDER BY UPDATE_TIME ) WHERE ROWNUM <= 100 """
# # query ="SELECT PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD') as SYNC_DATETIME FROM SMT_GET_PRODUCT where TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD')='2024-01-16'"
# }
HdfsFile {
# schema {
# fields {
# name = string
# address = string
# }
# }
path = "/user/hive/warehouse/test.db/users/part_ymd=jian"
file_format_type = "orc"
fs.defaultFS = "hdfs://hadoop121"
parse_partition_from_path = false
read_columns =[name,address]
result_table_name = "users"
# If you would like to get more information about how to configure seatunnel and see full list of source plugins,
# please go to https://seatunnel.apache.org/docs/category/source-v2
}
# #
# Hive {
# table_name = "test.users"
# metastore_uri = "thrift://hadoop121:9083"
# }
# Jdbc {
#
# query = """
# SELECT ID, CATEGORY_ID, PRODUCT_ID, RANK, AVG_RANK_7, AVG_RANK_7_UP,
# AVG_RANK_7_INCREASE, CREATE_USER_NAME,TO_CHAR(CREATE_DATETIME, 'YYYY-MM-DD')as CREATE_DATETIME,
# UPDATE_USER_NAME, UPDATE_DATETIME
# FROM product_selection_aliexpress_rank
#
# """
# # query = """SELECT PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD') as SYNC_DATETIME FROM ( SELECT PRO_ID,PRO_BOUGHT,CATE_1,SYNC_DATETIME FROM SMT_GET_PRODUCT ORDER BY UPDATE_TIME ) WHERE ROWNUM <= 100 """
# # query ="SELECT PRO_ID,PRO_BOUGHT,CATE_1,TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD') as SYNC_DATETIME FROM SMT_GET_PRODUCT where TO_CHAR(SYNC_DATETIME, 'YYYY-MM-DD')='2024-01-16'"
# }
}
transform{
Filter {
source_table_name = "users"
fields = [name]
result_table_name = "fake_name"
}
Filter {
source_table_name = "users"
fields = [name,address]
result_table_name = "users_all"
}
}
sink {
# 测试是否支持事务,初步测试不支持 20240120
# jdbc {
#
# auto_commit= false
# query = """
# start transaction ;
# delete from test.users where name = 'jian' ;
# insert into test.users(name,address) values(?,?) ;
# COMMIT;
# """
#
# }
# 分两步执行,第一步删除,第二步插入
jdbc {
query = """
delete from test.users where name = 'jian'
"""
}
# console {
# source_table_name = "users_all"
# }
#
# console {
# source_table_name = "fake_name"
# }
jdbc {
query = """
insert into test.users(name,address) values(?,?)
"""
}
}
# create table test.sink_user
# ( name string,age int)
# comment '测试seatunnel hive'
# ROW FORMAT DELIMITED FIELDS TERMINATED BY '#'
# STORED AS orc ;
```
### Running Command
```shell
/usr/local/apache-seatunnel-2.3.3/bin/seatunnel.sh --config ods_pdc_product_selection_aliexpress_rank.conf
```
### Error Exception
```log
2024-01-20 15:25:23,458 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 800989361205149698, job name: SeaTunnel
2024-01-20 15:25:23,466 WARN org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2024-01-20 15:26:23,624 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 800989361205149698
Read Count So Far : 1
Write Count So Far : 1
Average Read Count : 0/s
Average Write Count : 0/s
Last Statistic Time : 2024-01-20 15:25:23
Current Statistic Time : 2024-01-20 15:26:23
***********************************************
2024-01-20 15:27:23,593 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 800989361205149698
Read Count So Far : 2
Write Count So Far : 2
Average Read Count : 0/s
Average Write Count : 0/s
Last Statistic Time : 2024-01-20 15:26:23
Current Statistic Time : 2024-01-20 15:27:23
***********************************************
2024-01-20 15:28:23,560 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 800989361205149698
Read Count So Far : 2
Write Count So Far : 2
Average Read Count : 0/s
Average Write Count : 0/s
Last Statistic Time : 2024-01-20 15:27:23
Current Statistic Time : 2024-01-20 15:28:23
***********************************************
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #6260:
URL: https://github.com/apache/seatunnel/issues/6260#issuecomment-1903241729
set `parallelism = 1` and try again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6260:
URL: https://github.com/apache/seatunnel/issues/6260#issuecomment-1958439994
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6260:
URL: https://github.com/apache/seatunnel/issues/6260#issuecomment-1972206092
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [jdbc write] the job did not end, running every minutes [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6260: [Bug] [jdbc write] the job did not end, running every minutes
URL: https://github.com/apache/seatunnel/issues/6260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org