You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "PG-Young (via GitHub)" <gi...@apache.org> on 2023/03/06 02:25:38 UTC
[GitHub] [incubator-seatunnel] PG-Young opened a new issue, #4281: [Bug] [Clickhouse Sink] Sink to Clickhouse failed
PG-Young opened a new issue, #4281:
URL: https://github.com/apache/incubator-seatunnel/issues/4281
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
kafka Source Clickhouse Sink job.mode=Streaming Spark Jobs or stages had completed but clickhouse table not have data
but job.mode=BATCH it's ok
### SeaTunnel Version
2.3.0-dev
### SeaTunnel Config
```conf
env {
spark.app.name = "SeaTunnel"
spark.executor.instances =1
spark.executor.cores = 2
spark.executor.memory = "4g"
spark.executor.memoryOverhead="1g"
spark.streaming.batchDuration = 2
job.mode = "STREAMING"
#job.mode = "BATCH"
spark.streaming.kafka.maxRatePerPartition= 100
spark.streaming.batch.duration.seconds = 2
spark.streaming.backpressure.enabled=true
}
source {
kafka {
topic = "xxxxxxxxxxxx"
bootstrap.servers = "xxxxx:9092"
partition = 3
consumer.group = "seatunnel_group01"
format = json
kafka.request.timeout.ms = 60000
#kafka.max.poll.records = 200
kafka.fetch.max.bytes = 10485760
result_table_name = test
semantics = EXACTLY_ONCE
kafka.auto.offset.reset = latest
kafka.config = {
max.poll.records = 100
}
schema = {
fields {
xxx= double
xxxx= double
xxxx= double
}
}
}
}
transform{
}
sink {
Console {
}
Clickhouse {
source_table_name = test
host = "xx.xx.xx.xxx:8123"
clickhouse.socket_timeout = 50000
database = "xxxxxx"
table = "xxxxx"
fields = ["xxxx","xxxx","xxxx""]
username = "xxxxx"
password = "xxxxxx"
bulk_size = 20000
}
}
```
### Running Command
```shell
/start-seatunnel-spark-2-connector-v2.sh --master yarn --deploy-mode cluster --config ../config/examplespark01.conf
```
### Error Exception
```log
23/03/06 10:18:16 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:17 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:18 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:19 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:20 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:21 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:22 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:23 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:24 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:25 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:26 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:27 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:28 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:29 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:30 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:31 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:32 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:33 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:34 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
23/03/06 10:18:35 INFO yarn.Client: Application report for application_1677118697195_0046 (state: RUNNING)
```
### Flink or Spark Version
spark 2.4.4
### Java or Scala Version
jdk1.8
### Screenshots
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #4281: [Bug] [Clickhouse Sink] Sink to Clickhouse failed
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4281: [Bug] [Clickhouse Sink] Sink to Clickhouse failed
URL: https://github.com/apache/incubator-seatunnel/issues/4281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4281: [Bug] [Clickhouse Sink] Sink to Clickhouse failed
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4281:
URL: https://github.com/apache/incubator-seatunnel/issues/4281#issuecomment-1502508051
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhilinli123 commented on issue #4281: [Bug] [Clickhouse Sink] Sink to Clickhouse failed
Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on issue #4281:
URL: https://github.com/apache/incubator-seatunnel/issues/4281#issuecomment-1464865115
The clickhouse v2 connector is 20000 writes at a time. The Clickhouse V2 connector is 20000 writes at a time
@PG-Young
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4281: [Bug] [Clickhouse Sink] Sink to Clickhouse failed
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4281:
URL: https://github.com/apache/incubator-seatunnel/issues/4281#issuecomment-1520991239
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org