You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/11/28 13:39:50 UTC
[GitHub] [incubator-doris] xinyiZzz opened a new pull request #7234: [Memory] Add thread local variable TheadContext
xinyiZzz opened a new pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234
## Proposed changes
save query_id, fragment_instance_id, MemTracker, etc.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have created an issue on (Fix #7196) and described the bug/feature there in detail
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r822266596
##########
File path: be/src/runtime/threadlocal.cc
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
Review comment:
done, in threadlocal.h
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "common/logging.h"
+#include "gen_cpp/Types_types.h"
+#include "runtime/threadlocal.h"
+
+#define SCOPED_ATTACH_TASK_THREAD_4ARG(query_type, task_id, fragment_instance_id) \
+ auto VARNAME_LINENUM(attach_task_thread) = \
+ AttachTaskThread(query_type, task_id, fragment_instance_id)
+
+namespace doris {
+
+class TUniqueId;
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY = 1,
+ LOAD = 2,
+ COMPACTION = 3
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+
+ void attach(const TaskType& type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id) {
+ DCHECK(_type == TaskType::UNKNOWN && _task_id == "");
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type() const;
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() const { return _thread_id; }
+ const TUniqueId& fragment_instance_id() const { return _fragment_instance_id; }
+
+private:
+ std::thread::id _thread_id;
+ TaskType _type;
+ std::string _task_id;
+ TUniqueId _fragment_instance_id;
+};
+
+// Using gcc11 compiles thread_local variable on lower versions of GLIBC will report an error,
+// see https://github.com/apache/incubator-doris/pull/7911
+//
+// If we want to avoid this error,
+// 1. For non-trivial variables in thread_local, such as std::string, you need to store them as pointers to
+// ensure that thread_local is trivial, these non-trivial pointers will uniformly call destructors elsewhere.
+// 2. The default destructor of the thread_local variable cannot be overridden.
+//
+// This is difficult to implement. Because the destructor is not overwritten, it means that the outside cannot
+// be notified when the thread terminates, and the non-trivial pointers in thread_local cannot be released in time.
+// The func provided by pthread and std::thread doesn't help either.
+//
+// So, kudu Class-scoped static thread local implementation was introduced. Solve the above problem by
+// Thread-scopedthread local + Class-scoped thread local.
+//
+// This may look very track, but it's the best way I can find.
+//
+// refer to:
+// https://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Thread-Local.html
+// https://stackoverflow.com/questions/12049684/
+// https://sourceware.org/glibc/wiki/Destructor%20support%20for%20thread_local%20variables
+// https://www.jianshu.com/p/756240e837dd
+// https://man7.org/linux/man-pages/man3/pthread_tryjoin_np.3.html
+class ThreadContextPtr {
+public:
+ ThreadContextPtr();
+
+ ThreadContext* get();
+
+private:
+ DECLARE_STATIC_THREAD_LOCAL(ThreadContext, thread_local_ctx);
+};
+
+inline thread_local ThreadContextPtr thread_local_ctx;
+
+inline const std::string task_type_string(ThreadContext::TaskType type) {
+ switch (type) {
+ case ThreadContext::TaskType::QUERY:
+ return "QUERY";
+ case ThreadContext::TaskType::LOAD:
+ return "LOAD";
+ case ThreadContext::TaskType::COMPACTION:
+ return "COMPACTION";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+inline const std::string ThreadContext::type() const {
+ return task_type_string(_type);
+}
+
+class AttachTaskThread {
+public:
+ explicit AttachTaskThread(const ThreadContext::TaskType& type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id) {
+ DCHECK(task_id != "" && fragment_instance_id != TUniqueId());
Review comment:
done
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "common/logging.h"
+#include "gen_cpp/Types_types.h"
+#include "runtime/threadlocal.h"
+
+#define SCOPED_ATTACH_TASK_THREAD_4ARG(query_type, task_id, fragment_instance_id) \
+ auto VARNAME_LINENUM(attach_task_thread) = \
+ AttachTaskThread(query_type, task_id, fragment_instance_id)
+
+namespace doris {
+
+class TUniqueId;
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY = 1,
+ LOAD = 2,
+ COMPACTION = 3
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+
+ void attach(const TaskType& type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id) {
+ DCHECK(_type == TaskType::UNKNOWN && _task_id == "");
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type() const;
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() const { return _thread_id; }
+ const TUniqueId& fragment_instance_id() const { return _fragment_instance_id; }
+
+private:
+ std::thread::id _thread_id;
+ TaskType _type;
+ std::string _task_id;
+ TUniqueId _fragment_instance_id;
+};
+
+// Using gcc11 compiles thread_local variable on lower versions of GLIBC will report an error,
+// see https://github.com/apache/incubator-doris/pull/7911
+//
+// If we want to avoid this error,
+// 1. For non-trivial variables in thread_local, such as std::string, you need to store them as pointers to
+// ensure that thread_local is trivial, these non-trivial pointers will uniformly call destructors elsewhere.
+// 2. The default destructor of the thread_local variable cannot be overridden.
+//
+// This is difficult to implement. Because the destructor is not overwritten, it means that the outside cannot
+// be notified when the thread terminates, and the non-trivial pointers in thread_local cannot be released in time.
+// The func provided by pthread and std::thread doesn't help either.
+//
+// So, kudu Class-scoped static thread local implementation was introduced. Solve the above problem by
+// Thread-scopedthread local + Class-scoped thread local.
+//
+// This may look very track, but it's the best way I can find.
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r762703528
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+ ~ThreadContext() {}
+
+ void attach(TaskType type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id = TUniqueId()) {
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type();
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() { return _thread_id; }
+
+private:
+ std::thread::id _thread_id;
+ TaskType _type;
+ std::string _task_id;
+ TUniqueId _fragment_instance_id;
+};
+
+inline thread_local ThreadContext thread_local_ctx;
+
+inline const std::string TASK_TYPE_STRING(ThreadContext::TaskType type) {
Review comment:
why upper case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#issuecomment-1062604708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r762703203
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+ ~ThreadContext() {}
+
+ void attach(TaskType type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id = TUniqueId()) {
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type();
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() { return _thread_id; }
Review comment:
```suggestion
const std::thread::id& thread_id() const { return _thread_id; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r762718454
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
Review comment:
Thats great
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+ ~ThreadContext() {}
+
+ void attach(TaskType type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id = TUniqueId()) {
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type();
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() { return _thread_id; }
+
+private:
+ std::thread::id _thread_id;
+ TaskType _type;
+ std::string _task_id;
+ TUniqueId _fragment_instance_id;
+};
+
+inline thread_local ThreadContext thread_local_ctx;
+
+inline const std::string TASK_TYPE_STRING(ThreadContext::TaskType type) {
Review comment:
done
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+ ~ThreadContext() {}
+
+ void attach(TaskType type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id = TUniqueId()) {
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type();
Review comment:
done
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+ ~ThreadContext() {}
+
+ void attach(TaskType type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id = TUniqueId()) {
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type();
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() { return _thread_id; }
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r762703268
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+ ~ThreadContext() {}
+
+ void attach(TaskType type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id = TUniqueId()) {
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type();
Review comment:
```suggestion
const std::string type() const;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r762702762
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "gen_cpp/Types_types.h"
+
+namespace doris {
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY,
+ LOAD,
+ COMPACTION
+ // to be added ...
Review comment:
you`d bettor add index number to every enum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7234: [Thread] Add thread local variable ThreadContext
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #7234:
URL: https://github.com/apache/incubator-doris/pull/7234#discussion_r821752617
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "common/logging.h"
+#include "gen_cpp/Types_types.h"
+#include "runtime/threadlocal.h"
+
+#define SCOPED_ATTACH_TASK_THREAD_4ARG(query_type, task_id, fragment_instance_id) \
+ auto VARNAME_LINENUM(attach_task_thread) = \
+ AttachTaskThread(query_type, task_id, fragment_instance_id)
+
+namespace doris {
+
+class TUniqueId;
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY = 1,
+ LOAD = 2,
+ COMPACTION = 3
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+
+ void attach(const TaskType& type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id) {
+ DCHECK(_type == TaskType::UNKNOWN && _task_id == "");
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type() const;
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() const { return _thread_id; }
+ const TUniqueId& fragment_instance_id() const { return _fragment_instance_id; }
+
+private:
+ std::thread::id _thread_id;
+ TaskType _type;
+ std::string _task_id;
+ TUniqueId _fragment_instance_id;
+};
+
+// Using gcc11 compiles thread_local variable on lower versions of GLIBC will report an error,
+// see https://github.com/apache/incubator-doris/pull/7911
+//
+// If we want to avoid this error,
+// 1. For non-trivial variables in thread_local, such as std::string, you need to store them as pointers to
+// ensure that thread_local is trivial, these non-trivial pointers will uniformly call destructors elsewhere.
+// 2. The default destructor of the thread_local variable cannot be overridden.
+//
+// This is difficult to implement. Because the destructor is not overwritten, it means that the outside cannot
+// be notified when the thread terminates, and the non-trivial pointers in thread_local cannot be released in time.
+// The func provided by pthread and std::thread doesn't help either.
+//
+// So, kudu Class-scoped static thread local implementation was introduced. Solve the above problem by
+// Thread-scopedthread local + Class-scoped thread local.
+//
+// This may look very track, but it's the best way I can find.
Review comment:
```suggestion
// This may look very trick, but it's the best way I can find.
```
##########
File path: be/src/runtime/thread_context.h
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <string>
+#include <thread>
+
+#include "common/logging.h"
+#include "gen_cpp/Types_types.h"
+#include "runtime/threadlocal.h"
+
+#define SCOPED_ATTACH_TASK_THREAD_4ARG(query_type, task_id, fragment_instance_id) \
+ auto VARNAME_LINENUM(attach_task_thread) = \
+ AttachTaskThread(query_type, task_id, fragment_instance_id)
+
+namespace doris {
+
+class TUniqueId;
+
+// The thread context saves some info about a working thread.
+// 2 requried info:
+// 1. thread_id: Current thread id, Auto generated.
+// 2. type: The type is a enum value indicating which type of task current thread is running.
+// For example: QUERY, LOAD, COMPACTION, ...
+// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+//
+// There may be other optional info to be added later.
+class ThreadContext {
+public:
+ enum TaskType {
+ UNKNOWN = 0,
+ QUERY = 1,
+ LOAD = 2,
+ COMPACTION = 3
+ // to be added ...
+ };
+
+public:
+ ThreadContext() : _thread_id(std::this_thread::get_id()), _type(TaskType::UNKNOWN) {}
+
+ void attach(const TaskType& type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id) {
+ DCHECK(_type == TaskType::UNKNOWN && _task_id == "");
+ _type = type;
+ _task_id = task_id;
+ _fragment_instance_id = fragment_instance_id;
+ }
+
+ void detach() {
+ _type = TaskType::UNKNOWN;
+ _task_id = "";
+ _fragment_instance_id = TUniqueId();
+ }
+
+ const std::string type() const;
+ const std::string& task_id() const { return _task_id; }
+ const std::thread::id& thread_id() const { return _thread_id; }
+ const TUniqueId& fragment_instance_id() const { return _fragment_instance_id; }
+
+private:
+ std::thread::id _thread_id;
+ TaskType _type;
+ std::string _task_id;
+ TUniqueId _fragment_instance_id;
+};
+
+// Using gcc11 compiles thread_local variable on lower versions of GLIBC will report an error,
+// see https://github.com/apache/incubator-doris/pull/7911
+//
+// If we want to avoid this error,
+// 1. For non-trivial variables in thread_local, such as std::string, you need to store them as pointers to
+// ensure that thread_local is trivial, these non-trivial pointers will uniformly call destructors elsewhere.
+// 2. The default destructor of the thread_local variable cannot be overridden.
+//
+// This is difficult to implement. Because the destructor is not overwritten, it means that the outside cannot
+// be notified when the thread terminates, and the non-trivial pointers in thread_local cannot be released in time.
+// The func provided by pthread and std::thread doesn't help either.
+//
+// So, kudu Class-scoped static thread local implementation was introduced. Solve the above problem by
+// Thread-scopedthread local + Class-scoped thread local.
+//
+// This may look very track, but it's the best way I can find.
+//
+// refer to:
+// https://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Thread-Local.html
+// https://stackoverflow.com/questions/12049684/
+// https://sourceware.org/glibc/wiki/Destructor%20support%20for%20thread_local%20variables
+// https://www.jianshu.com/p/756240e837dd
+// https://man7.org/linux/man-pages/man3/pthread_tryjoin_np.3.html
+class ThreadContextPtr {
+public:
+ ThreadContextPtr();
+
+ ThreadContext* get();
+
+private:
+ DECLARE_STATIC_THREAD_LOCAL(ThreadContext, thread_local_ctx);
+};
+
+inline thread_local ThreadContextPtr thread_local_ctx;
+
+inline const std::string task_type_string(ThreadContext::TaskType type) {
+ switch (type) {
+ case ThreadContext::TaskType::QUERY:
+ return "QUERY";
+ case ThreadContext::TaskType::LOAD:
+ return "LOAD";
+ case ThreadContext::TaskType::COMPACTION:
+ return "COMPACTION";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+inline const std::string ThreadContext::type() const {
+ return task_type_string(_type);
+}
+
+class AttachTaskThread {
+public:
+ explicit AttachTaskThread(const ThreadContext::TaskType& type, const std::string& task_id,
+ const TUniqueId& fragment_instance_id) {
+ DCHECK(task_id != "" && fragment_instance_id != TUniqueId());
Review comment:
If this is not a query task , the `fragment_instance_id` maybe empty.
##########
File path: be/src/runtime/threadlocal.cc
##########
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
Review comment:
If this file is copied from other project, better add a comment.
Same as other file in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org