Re: [I] help request: skywalking-logger plugin throws "no memory" errors under high concurrency [apisix]

["juzhiyuan (via GitHub)" <gi...@apache.org>]

juzhiyuan commented on issue #12524:
URL: https://github.com/apache/apisix/issues/12524#issuecomment-4366374755

   In APISIX 3.13, the skywalking tracer pushes finished trace segments into ngx.shared.tracing_buffer. The skywalking-logger endpoint_addr only configures log reporting to /v3/logs; it does not configure where the skywalking tracer sends /v3/segments.
   
   From the screenshots, I can see tracing_buffer is already set to 500m, but I do not see the config.yaml plugin_attr.skywalking section. Please confirm whether you configured:
   
   ```yaml
   plugin_attr:
     skywalking:
       report_interval: 3
       endpoint_addr: http://<your-oap>:12800
       service_name: ...
       service_instance_name: ...
   ```
   
   The route-level skywalking plugin config should mainly contain sample_ratio. The report_interval and endpoint_addr for trace segment reporting should be in plugin_attr.skywalking.
   
   If _plugin_attr.skywalking.endpoint_addr_ is not set correctly, APISIX may still try to report trace segments to the default http://127.0.0.1:12800. Under _ab -c 2000 -t 20 -k_ with _sample_ratio=1_, trace segments can be produced much faster than they are drained, and tracing_buffer can eventually return no memory.
   
   Could you also check whether error.log contains Segment report fails, Instance report fails, or Agent ping fails during the test? It would help to test these three cases separately:
   
   1. only skywalking enabled
   2. only skywalking-logger enabled
   3. both enabled
   
   For skywalking-logger, batch_max_size=10000 plus timeout=100/max_retry_count=5 can also create large pending batches if OAP is slow or unreachable, but that is separate from the tracer.lua:145 error path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org