You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "weixiang1862 (via GitHub)" <gi...@apache.org> on 2025/12/11 08:29:35 UTC

[PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

weixiang1862 opened a new pull request, #13611:
URL: https://github.com/apache/skywalking/pull/13611

   <!--
       ⚠️ Please make sure to read this template first, pull requests that don't accord with this template
       maybe closed without notice.
       Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
       Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
   -->
   
   <!-- ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👇 ====
   ### Fix <bug description or the bug issue number or bug issue link>
   - [ ] Add a unit test to verify that the fix works.
   - [ ] Explain briefly why the bug exists and how to fix it.
        ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👆 ==== -->
   
   <!-- ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👇 ====
   ### Improve the performance of <class or module or ...>
   - [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
   - [ ] The benchmark result.
   ```text
   <Paste the benchmark results here>
   ```
   - [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
        ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👆 ==== -->
   
   <!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
   ### <Feature description>
   - [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
   - [ ] Update the documentation to include this new feature.
   - [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
   - [ ] If it's UI related, attach the screenshots below.
        ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
   
   - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
   - [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
   
   If multi DataCarriers share one `BulkConsumePool`, shuntdown channels should just remove their own workloads rather than shutdown all consumer threads. Because these threads are shared between all DataCarriers.
   
   I'm not sure if this change aligns with the original semantics of the method `DataCarrier#shutdownConsumers`. Could you help me confirm?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "weixiang1862 (via GitHub)" <gi...@apache.org>.
weixiang1862 closed pull request #13611: Shutdown DataCarrier own consume group if they share one BulkConsumePool
URL: https://github.com/apache/skywalking/pull/13611


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #13611:
URL: https://github.com/apache/skywalking/pull/13611#discussion_r2618140680


##########
oap-server/server-library/library-datacarrier-queue/src/main/java/org/apache/skywalking/oap/server/library/datacarrier/consumer/MultipleChannelsConsumer.java:
##########
@@ -133,6 +133,28 @@ void shutdown() {
         running = false;
     }
 
+    void close(Channels channels) {

Review Comment:
   Within `#addNewTarget`, for oap case, it is in the booting stage only. But here, you are really in the runtime. 
   As we don't have hot-reload in the upstream, we can't verify the result. 
   Also, as it is allocating the channels to other consumers(in your case), this is meaningless in upstream case, because we only call this in the shutting down process.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #13611:
URL: https://github.com/apache/skywalking/pull/13611#discussion_r2617140744


##########
oap-server/server-library/library-datacarrier-queue/src/main/java/org/apache/skywalking/oap/server/library/datacarrier/consumer/MultipleChannelsConsumer.java:
##########
@@ -133,6 +133,28 @@ void shutdown() {
         running = false;
     }
 
+    void close(Channels channels) {

Review Comment:
   If you are going to use this in a hot-update case, I have a concern whether this is thread safe. Because the reloading could happen in any time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #13611:
URL: https://github.com/apache/skywalking/pull/13611#issuecomment-3640878332

   I am not following. The `#shutdown` was just to make sure that the thread is stopped. Nothing else. What is the purpose of the new close?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "weixiang1862 (via GitHub)" <gi...@apache.org>.
weixiang1862 commented on code in PR #13611:
URL: https://github.com/apache/skywalking/pull/13611#discussion_r2618097603


##########
oap-server/server-library/library-datacarrier-queue/src/main/java/org/apache/skywalking/oap/server/library/datacarrier/consumer/MultipleChannelsConsumer.java:
##########
@@ -133,6 +133,28 @@ void shutdown() {
         running = false;
     }
 
+    void close(Channels channels) {

Review Comment:
   This method refers to `MultipleChannelsConsumer#addNewTarget`, it also recreate the new list to avoid change list while the list is used in consuming.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "weixiang1862 (via GitHub)" <gi...@apache.org>.
weixiang1862 commented on PR #13611:
URL: https://github.com/apache/skywalking/pull/13611#issuecomment-3644724842

   > I am not following. The `#shutdown` was just to make sure that the thread is stopped. Nothing else. What is the purpose of the new close?
   
   `DataCarrier` driven by `BulkConsumePool` is used in aggregate and persistent workers in skywalking.
   Let's imagine this scenario: 
   when a user modifies `MAL` and wants the changes to take effect immediately through hot reload, the corresponding aggregate worker and persistent worker need to be rebuilt. 
   In this process, one step is shutting down the current worker's datacarrier without affecting the datacarriers of other workers that are still running.
   
   Consider about this, I think this changes make more sense. But if it broken something, I will close this pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Shutdown DataCarrier own consume group if they share one BulkConsumePool [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #13611:
URL: https://github.com/apache/skywalking/pull/13611#issuecomment-3648794010

   I don't have much preference. @wankai123 any concern?
   
   From my side, we don't have this case from upstream. This seems to be your private fork feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org