You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/07/13 07:03:17 UTC
[GitHub] [incubator-seatunnel] CalvinKirs opened a new issue, #2171: [Feature]Connectors e2e test
CalvinKirs opened a new issue, #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171
SeaTunnel contains a large number of Connectors, and the basic UT cannot perform system testing in a real environment, so we introduced the `seatunnel-e2e` module for testing
e2e automated testing, we are based on `Testcontainers` to complete the testing of various Connectors.
About Testcontainers (https://www.testcontainers.org/)
Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
If you need to test the Connector based on the Flink engine, you need to inherit `org.apache.seatunnel.e2e.flink.FlinkContainer`, which is the Flink container we have implemented, which helps you to automatically complete the startup and operation of the Flink engine Execution, container shutdown and other operations, so that you can pay more attention to your own business implementation, if you need to test Kafka, then you need to manually introduce the Docker image of Hive,
Just like this, see https://www.testcontainers.org/
````
KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))
````
If you need to use a custom Docker image, this is also very easy, you can refer to https://www.testcontainers.org/features/creating_images/
For us, the main thing to test is end-to-end data synchronization, you just need to provide a simple configuration file like this, then create the container (maybe not necessary, such as the one below) and start it. Of course, after the execution is completed, there should be a process of data proofreading and verification, which you can complete by yourself.
````
env {
# You can set flink configuration here
execution.parallelism = 1
job.mode = "BATCH"
#execution.checkpoint.interval=10000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is an example source plugin **only for test and demonstrate the feature source plugin**
FakeSource {
result_table_name = "fake"
field_name = "name,age"
}
# If you would like to get more information about how to configure seatunnel and see full list of source plugins,
# please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake
}
transform {
sql {
sql = "select name, age from fake"
}
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
# please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql
}
sink {
Console {}
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
# please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
FWLamb commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1261869909
I plan to test the hive to clickhouse, but there is no hive related environment for testcontainers. How can I do this？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dinggege1024 commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
dinggege1024 commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1184454763
@CalvinKirs Hello, can you assign me the sink for Kafka? I want to try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed issue #2171: [Feature]Connectors e2e test
URL: https://github.com/apache/incubator-seatunnel/issues/2171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1192099340
> @CalvinKirs Please assign me the sink and the source of Hive,HdfsFile,LocalFile,Http,Socket. I can do these works.
BTW, also sink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1293285319
> I plan to test the hive to clickhouse, but there is no hive related environment for testcontainers. How can I do this？
We can use custom images, but the Hive docker images are more troublesome, you could try them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1261629609
> Hello, can you assign me the sink for Clickhouse? My current production environment wants to use new functions. I want to try it.
Sure, good job～
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FWLamb commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
FWLamb commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1261627301
Hello, can you assign me the sink for Clickhouse? My current production environment wants to use new functions. I want to try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1194893572
> @CalvinKirs Can you assign me the sink and source of clickhouse? I can do them.
yup, if you have any questions. please call me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] FlechazoW commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
FlechazoW commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1195420775
@CalvinKirs Can you assign me the sink and source of pulsar? I can do them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] bravekong commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
bravekong commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1293240590
@CalvinKirs Can you assign me the sink for Hudi? I'm syncing the data to hudi. I want to try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] cnwyc commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
cnwyc commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1298602802
@CalvinKirs Can you assign me the sink for Hudi? I'm syncing the data to hudi. I want to try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhangyuge1 commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
zhangyuge1 commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1183502197
@CalvinKirs Hello, I want to try the source and sink for jdbc. Can you assign them to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] eyys commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
eyys commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1183180239
@CalvinKirs Hello, can you assign me the source for Kafka? I want to try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1191724435
@CalvinKirs Please assign me the sink and the source of Hive,HdfsFile,LocalFile,Http,Socket. I can do these works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhangyuge1 commented on issue #2171: [Feature]Connectors e2e test
Posted by GitBox <gi...@apache.org>.
zhangyuge1 commented on issue #2171:
URL: https://github.com/apache/incubator-seatunnel/issues/2171#issuecomment-1194151562
@CalvinKirs Can you assign me the sink and source of clickhouse? I can do them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org