You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/11/09 10:57:39 UTC
[GitHub] [skywalking-satellite] mrproliu opened a new pull request #82: Support partition queue
mrproliu opened a new pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82
* Support partition queue. Resolve https://github.com/apache/skywalking/issues/8077
* Add self-telemetry to the `ALS` protocol and `queue`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] mrproliu commented on a change in pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
mrproliu commented on a change in pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82#discussion_r745536655
##########
File path: docs/en/setup/plugins/queue_mmap-queue.md
##########
@@ -15,6 +15,9 @@ flush_period: 1000
flush_ceiling_num: 10000
# The max size of the input event. Default value is 20k.
max_event_size: 20480
+
+# The partition count of queue.
+partition_count: 1
Review comment:
Yes, It supports. I wrap a partitioned queue to manage all sub-queue(memory or mmap).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] mrproliu commented on a change in pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
mrproliu commented on a change in pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82#discussion_r745560008
##########
File path: plugins/queue/mmap/queue_operation.go
##########
@@ -20,11 +20,14 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
+//go:build !windows
Review comment:
They are the same, I'm using the go@1.17 version in my local machine. So It auto-generates by `go fmt`, being deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] kezhenxu94 commented on a change in pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82#discussion_r745538256
##########
File path: docs/en/setup/examples/high-throughput-queue/README.md
##########
@@ -0,0 +1,19 @@
+# High throughput queue
+
+High throughput queues can send messages to multiple channels using by `round-robin` policy for load-balance, and consume messages from each queue in parallel.
Review comment:
```suggestion
High throughput queues can send messages to multiple channels by using `round-robin` policy for load-balance, and consume messages from each queue in parallel.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] kezhenxu94 commented on a change in pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82#discussion_r745539610
##########
File path: plugins/queue/mmap/queue_operation.go
##########
@@ -20,11 +20,14 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
+//go:build !windows
Review comment:
What's the difference?
```
//go:build !windows
// +build !windows
```
##########
File path: docs/en/setup/examples/high-throughput-queue/README.md
##########
@@ -0,0 +1,19 @@
+# High throughput queue
+
+High throughput queues can send messages to multiple channels using by `round-robin` policy for load-balance, and consume messages from each queue in parallel.
+This prevents the single-threaded sender from performing network IO, blocking the `recieve`/`fetch`/`consumer` to the queue.
+
+## Config
+
+Both existing two queues(`memory`, `mmap`) support this future, add the `partition_count` to the queue config, default is `1`.
Review comment:
```suggestion
Both existing two queues(`memory`, `mmap`) support this feature, add the `partition_count` to the queue config, default is `1`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] wu-sheng commented on a change in pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82#discussion_r745532386
##########
File path: configs/satellite_config.yaml
##########
@@ -103,6 +105,8 @@ pipes:
plugin_name: "memory-queue"
# The maximum buffer event size.
event_buffer_size: ${SATELLITE_QUEUE_EVENT_BUFFER_SIZE:5000}
+ # The partition count of queue.
+ partition_count: ${SATELLITE_QUEUE_PARTITION_COUNT:1}
Review comment:
Remove count. Partition is enough.
##########
File path: docs/en/setup/examples/high-throughput-queue/README.md
##########
@@ -0,0 +1,19 @@
+# High throughput queue
+
+High throughput queues can send messages to multiple channels using by `round-robin` policy for load-balance, and consume messages from each queue in parallel.
+This prevents the single-threaded sender from performing network IO, blocking the `recieve`/`fetch`/`consumer` to the queue.
Review comment:
Receiver and fetcher.
##########
File path: docs/en/setup/plugins/queue_mmap-queue.md
##########
@@ -15,6 +15,9 @@ flush_period: 1000
flush_ceiling_num: 10000
# The max size of the input event. Default value is 20k.
max_event_size: 20480
+
+# The partition count of queue.
+partition_count: 1
Review comment:
mmap queue supports partitions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] wu-sheng commented on a change in pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82#discussion_r745602717
##########
File path: docs/menu.yml
##########
@@ -27,13 +27,13 @@ catalog:
path: /en/concepts-and-designs/project-goals
- name: Design
catalog:
- - name: Module Design
+ - name: Module design
Review comment:
I think we should keep upper cases for every noun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-satellite] wu-sheng merged pull request #82: Support partition queue
Posted by GitBox <gi...@apache.org>.
wu-sheng merged pull request #82:
URL: https://github.com/apache/skywalking-satellite/pull/82
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org