You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/10 07:25:33 UTC
[GitHub] [incubator-doris] liutang123 opened a new pull request #3294: Use
read lock when iterate tablet_map in TabletManager::start_trash_sweep
liutang123 opened a new pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294
In our cluster, there are many stream load jobs.
We found that these jobs slow down periodically.
By add logs, we found that `TabletManager::start_trash_sweep` will take write lock for a long time.
In this PR, we split `start_trash_sweep` in two phases:
First, iterate the tablet map with read lock.
Second, remove the empty tablet in map with write lock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r406765575
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
Is it safe to iterate a map without read lock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r406765575
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
Is it safe to iterate a map without read lock?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r406658150
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
You can put tablet->delete_expired_inc_rowsets() between the ReadLock and WriteLock.
This is bottleneck which consumes much resource.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3294: Use read lock when
iterate tablet_map in TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#issuecomment-611932036
> By add logs, we found that `TabletManager::start_trash_sweep` will take write lock for a long time.
How long? And after this PR applied, how long will it be?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #3294: Use read
lock when iterate tablet_map in TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#issuecomment-611930999
@liutang123 Please add a issue for this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r406765575
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
Can you give the example code?
Is it safe to iterate a map without read lock?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r407061670
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
Currently, there should be only one tablet in `tablet_arr`.
Only in the early Doris version, `tablet_arr` will appear multiple tablets.
And in `tablet->delete_expired_inc_rowsets();`, it will hold the `meta lock`, So need to be protected by `_tablet_map`.
So you can just by doing:
1. do the `tablets_to_clean.push_back(item.first);` in `ReadLock`
2. Unlock the `ReadLock`, and iterate the `tablets_to_clean` to do the `tablet->delete_expired_inc_rowsets();`
3. Do the following `tablet_map.erase(tablet_id_to_clean);` in `WriteLock`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r406765575
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
Can you give the example code?
I am not sure whether it is safe to iterate a map without read lock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 edited a comment on issue #3294: Use
read lock when iterate tablet_map in TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 edited a comment on issue #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#issuecomment-612041189
> > By add logs, we found that `TabletManager::start_trash_sweep` will take write lock for a long time.
>
> How long? And after this PR applied, how long will it be?
We set `tablet_map_shard_size` to 16, this function will holds each map lock about more than 1 second.
After this PR, this function holds write lock less than 100 ms.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on issue #3294: Use read
lock when iterate tablet_map in TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on issue #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#issuecomment-612041189
> > By add logs, we found that `TabletManager::start_trash_sweep` will take write lock for a long time.
>
> How long? And after this PR applied, how long will it be?
We set `tablet_map_shard_size` to 16, this function will holds each map lock about more than 1 second.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3294: Use read lock
when iterate tablet_map in TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
chaoyli merged pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r407078047
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
> 2. Unlock the `ReadLock`, and iterate the `tablets_to_clean` to do the `tablet->delete_expired_inc_rowsets();`
We should invoke `delete_expired_inc_rowsets` in all tablets but not only in tablets_to_clean. Isn't it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3294: Use read lock when iterate tablet_map in
TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#discussion_r407061670
##########
File path: be/src/olap/tablet_manager.cpp
##########
@@ -960,28 +960,36 @@ OLAPStatus TabletManager::start_trash_sweep() {
{
std::vector<int64_t> tablets_to_clean;
for (int32 i = 0; i < _tablet_map_lock_shard_size; i++) {
- WriteLock rlock(&_tablet_map_lock_array[i]);
tablet_map_t& tablet_map = _tablet_map_array[i];
- for (auto& item : tablet_map) {
- // try to clean empty item
- if (item.second.table_arr.empty()) {
- // try to get schema change lock if could get schema change lock, then nobody
- // own the lock could remove the item
- // it will core if schema change thread may hold the lock and this thread will deconstruct lock
- if (item.second.schema_change_lock.trylock() == OLAP_SUCCESS) {
- item.second.schema_change_lock.unlock();
+ {
+ ReadLock r_lock(&_tablet_map_lock_array[i]);
+ for (auto& item : tablet_map) {
+ // try to clean empty item
+ if (item.second.table_arr.empty()) {
tablets_to_clean.push_back(item.first);
}
- }
- for (TabletSharedPtr tablet : item.second.table_arr) {
- tablet->delete_expired_inc_rowsets();
+ for (TabletSharedPtr tablet : item.second.table_arr) {
Review comment:
Currently, there should be only one tablet in `tablet_arr`.
Only in the early Doris version, `tablet_arr` will appear multiple tablets.
And in `tablet->delete_expired_inc_rowsets();`, it will hold the `meta lock`, So no need to be protected by `_tablet_map`.
So you can just by doing:
1. do the `tablets_to_clean.push_back(item.first);` in `ReadLock`
2. Unlock the `ReadLock`, and iterate the `tablets_to_clean` to do the `tablet->delete_expired_inc_rowsets();`
3. Do the following `tablet_map.erase(tablet_id_to_clean);` in `WriteLock`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on issue #3294: Use read
lock when iterate tablet_map in TabletManager::start_trash_sweep
Posted by GitBox <gi...@apache.org>.
liutang123 commented on issue #3294: Use read lock when iterate tablet_map in TabletManager::start_trash_sweep
URL: https://github.com/apache/incubator-doris/pull/3294#issuecomment-612729982
@chaoyli Hi, do you have time to see this PR again.
This PR has been online in our cluster for two days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org