You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "lingtaolf (via GitHub)" <gi...@apache.org> on 2023/03/13 07:48:27 UTC
[GitHub] [incubator-seatunnel] lingtaolf opened a new issue, #4340: How to set multiple sources at same time in one task
lingtaolf opened a new issue, #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340
Hi all:
I use seatunnel to fetch data from multiple hive sources and write them to clickhouse. But the result is only the first table in source have been write to clickhouse. And I'm sure all other hive tables have data.
My config file is just like below:
```xml
env {
spark.app.name = "seatunnel_2023-02-01"
spark.driver.cores = 2
spark.rpc.message.maxSize = 2047
spark.executor.instances = 200
spark.driver.maxResultSize="5g"
spark.driver.memory = "20g"
spark.executor.cores = 5
spark.executor.memory = "10g"
spark.sql.catalogImplementation = "hive"
}
source {
hive {
pre_sql = "select .... from A'"
parallelism = 6
table_name = "A"
metastore_uri = "xxxxx"
result_table_name = "result_table_A"
}
hive {
pre_sql = "select .... from B"
parallelism = 6
table_name = "B"
metastore_uri = "xxxxx"
result_table_name = "result_table_B"
}
transform {}
sink {
clickhouse {
host = "1xxxx"
database = "xx"
table = "sink_table"
fields = [....]
username = "default"
password = "default"
bulk_size = 4000000
clickhouse.socket_timeout = 300000
clickhouse.connection_timeout = 300000
clickhouse.dataTransferTimeout = 300000
retry = 3
}
}
```
I don't know why. Could someone help me ?
Thx!
Btw, The seatunnel version I used is 2.1.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lingtaolf closed issue #4340: How to set multiple sources at same time in one task
Posted by "lingtaolf (via GitHub)" <gi...@apache.org>.
lingtaolf closed issue #4340: How to set multiple sources at same time in one task
URL: https://github.com/apache/incubator-seatunnel/issues/4340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] laglangyue commented on issue #4340: How to set multiple sources at same time in one task
Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340#issuecomment-1474769519
as zorrofox said，your source table is `result_table_A` and `result_table_B`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zorrofox commented on issue #4340: How to set multiple sources at same time in one task
Posted by "zorrofox (via GitHub)" <gi...@apache.org>.
zorrofox commented on issue #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340#issuecomment-1473997707
I think your sink part should be:
```
sink {
clickhouse {
host = "1xxxx"
database = "xx"
table = "sink_table"
fields = [....]
username = "default"
password = "default"
bulk_size = 4000000
clickhouse.socket_timeout = 300000
clickhouse.connection_timeout = 300000
clickhouse.dataTransferTimeout = 300000
retry = 3
source_table_name= result_table_A
}
clickhouse {
host = "1xxxx"
database = "xx"
table = "sink_table"
fields = [....]
username = "default"
password = "default"
bulk_size = 4000000
clickhouse.socket_timeout = 300000
clickhouse.connection_timeout = 300000
clickhouse.dataTransferTimeout = 300000
retry = 3
source_table_name= result_table_B
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lingtaolf commented on issue #4340: How to set multiple sources at same time in one task
Posted by "lingtaolf (via GitHub)" <gi...@apache.org>.
lingtaolf commented on issue #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340#issuecomment-1477202940
>
It works. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lingtaolf commented on issue #4340: How to set multiple sources at same time in one task
Posted by "lingtaolf (via GitHub)" <gi...@apache.org>.
lingtaolf commented on issue #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340#issuecomment-1475548512
@zorrofox @laglangyue ok, I will try. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] laglangyue commented on issue #4340: How to set multiple sources at same time in one task
Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340#issuecomment-1465680249
maybe it not supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] lingtaolf commented on issue #4340: How to set multiple sources at same time in one task
Posted by "lingtaolf (via GitHub)" <gi...@apache.org>.
lingtaolf commented on issue #4340:
URL: https://github.com/apache/incubator-seatunnel/issues/4340#issuecomment-1467263295
> maybe it not supported
@laglangyue But， I‘ve read the doc for 2.1.3, it says ' Multiple sources can be defined at the same time'. I'm not sure it can work correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org