You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2023/01/02 08:42:48 UTC
[GitHub] [incubator-seatunnel] liugddx opened a new pull request, #3847: [Improve][Seatunnel-Engine]change queue to disruptor
liugddx opened a new pull request, #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/incubator-seatunnel/tree/dev/docs/en/connector-v2)
2. Update [plugin-mapping.properties](https://github.com/apache/incubator-seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
3. Update the pom file of [seatunnel-dist](https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-dist/pom.xml)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1387109836
> > Maybe you can use config to set some e2e test case use disruptor queue, to test disruptor queue in zeta.
>
> Done.
Not see e2e test cases. Do you test it in real env or linux server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1369324564
> Any discuss for this? and why change it?
Hmm, not discussed yet. Let me first compare whether the two solutions will improve performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1387322180
The following is the result of my test fake-console.
```
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######
###### This config file is a demonstration of streaming processing in seatunnel config
######
env {
# You can set engine configuration here
execution.parallelism = 1
job.mode = "BATCH"
checkpoint.interval = 5000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
FakeSource {
result_table_name = "fake"
row.num = 2500000
parallelism = 4
schema = {
fields {
name = "string"
age = "int"
}
}
}
}
transform {
}
sink {
console {
source_table_name="fake"
}
}
```
- disruptor
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-01-18 23:55:57
End Time : 2023-01-18 23:57:31
Total Time(s) : 94
Total Read Count : 10000000
Total Write Count : 10000000
Total Failed Count : 0
***********************************************
- blockingQueue
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-01-18 23:52:13
End Time : 2023-01-18 23:54:08
Total Time(s) : 114
Total Read Count : 10000000
Total Write Count : 10000000
Total Failed Count : 0
***********************************************
Efficiency increased by 17.5%. [（114 -94）%114 * 100% = 17.5%].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1387095823
> Maybe you can use config to set some e2e test case use disruptor queue, to test disruptor queue in zeta.
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1386755237
> Maybe you can use config to set some e2e test case use disruptor queue, to test disruptor queue in zeta.
BTW, could you please offer some performance reports to verify this pr is valid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1382826394
This is test case:
```
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######
###### This config file is a demonstration of streaming processing in seatunnel config
######
env {
# You can set engine configuration here
execution.parallelism = 1
job.mode = "BATCH"
checkpoint.interval = 5000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
FakeSource {
result_table_name = "fake"
row.num = 1000000
parallelism = 4
schema = {
fields {
name = "string"
age = "int"
}
}
}
}
transform {
}
sink {
console {
source_table_name="fake"
}
}
```
- BlockingQueue
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-01-14 23:30:58
End Time : 2023-01-14 23:35:07
Total Time(s) : 248
Total Read Count : 8000000
Total Write Count : 8000000
Total Failed Count : 0
***********************************************
- disruptor
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-01-14 23:40:07
End Time : 2023-01-14 23:43:47
Total Time(s) : 219
Total Read Count : 8000000
Total Write Count : 8000000
Total Failed Count : 0
***********************************************
The disruptor takes 30s less than BlockingQueue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1369309484
Any discuss for this? and why change it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1382826493
> This is test case:
>
> ```
> #
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #
> ######
> ###### This config file is a demonstration of streaming processing in seatunnel config
> ######
>
> env {
> # You can set engine configuration here
> execution.parallelism = 1
> job.mode = "BATCH"
> checkpoint.interval = 5000
> #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
> }
>
> source {
> # This is a example source plugin **only for test and demonstrate the feature source plugin**
> FakeSource {
> result_table_name = "fake"
> row.num = 1000000
> parallelism = 4
> schema = {
> fields {
> name = "string"
> age = "int"
> }
> }
> }
> }
>
> transform {
> }
>
> sink {
> console {
> source_table_name="fake"
> }
> }
> ```
>
> * BlockingQueue
>
> 
>
> ```
> Job Statistic Information
> ```
>
> Start Time : 2023-01-14 23:30:58 End Time : 2023-01-14 23:35:07 Total Time(s) : 248 Total Read Count : 8000000 Total Write Count : 8000000 Total Failed Count : 0
>
> * disruptor
>
> 
>
> ```
> Job Statistic Information
> ```
>
> Start Time : 2023-01-14 23:40:07 End Time : 2023-01-14 23:43:47 Total Time(s) : 219 Total Read Count : 8000000 Total Write Count : 8000000 Total Failed Count : 0
>
> The disruptor takes 30s less than BlockingQueue
@Hisoka-X
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1386382252
Maybe you can use config to set some e2e test case use disruptor queue, to test disruptor queue in zeta.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx closed pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx closed pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
URL: https://github.com/apache/incubator-seatunnel/pull/3847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#discussion_r1071038070
##########
seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java:
##########
@@ -49,6 +49,8 @@ public class ServerConfigOptions {
public static final Option<Integer> CHECKPOINT_STORAGE_MAX_RETAINED = Options.key("max-retained").intType().defaultValue(1).withDescription("The maximum number of retained checkpoints.");
+ public static final Option<String> QUEUE_TYPE = Options.key("queue_type").stringType().defaultValue("disruptor").withDescription("The internal data cache queue type.");
Review Comment:
The value should be enum, also please use `blockingQueue` at default value. Then change doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1387260955
> > > Maybe you can use config to set some e2e test case use disruptor queue, to test disruptor queue in zeta.
> >
> >
> > Done.
>
> Not see e2e test cases. Do you test it in real env or linux server?
As discussed earlier we need a benchmarking framework .So I only provide screenshots for test comparison.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by GitBox <gi...@apache.org>.
liugddx commented on PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847#issuecomment-1384863970
@Hisoka-X @EricJoy2048 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 merged pull request #3847: [Improve][Seatunnel-Engine]change queue to disruptor
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 merged PR #3847:
URL: https://github.com/apache/incubator-seatunnel/pull/3847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org