You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/12/02 17:02:48 UTC

[GitHub] [camel-k] haifzhan opened a new issue #2808: SEDA low performance issue

haifzhan opened a new issue #2808:
URL: https://github.com/apache/camel-k/issues/2808


   Hi Camel-K team,
   
   I have used SEDA in my route for processing high PPS(Packets per second) streaming.  Below is my code snippet, I use 5 consumers to consume events from in-memory queue. I tested PPS to 10,000 and my pod crashes 30 minutes later and complains the JVM heap space out of memory.
   
   ```
   from("netty:udp://" + HOST + ":" + PORT + "?sync=false&receiveBufferSize=16777216")
     // .log("in: ${body}")
     .to("seda:next?size=5000&timeout=0&blockWhenFull=true");
   
   
     from("seda:next?concurrentConsumers=5")
     .unmarshal(myDataFormat)
     .routePolicy(myPolicy)
     .process(myProcessor)
     .choice()
     .when(body().contains(MISSING_REQUIRED_FIELDS)).to("log:warning").otherwise()
     .log("log:${body}");
   ```
   
   Is there any way to tune performance? What I am planning to try is replace in-memory queue with Kafka, unfortunately I haven't found any resources to demo how to connect with Kafka for SEDA component.
   
   Same question had asked on [StackOverflow](https://stackoverflow.com/questions/70189210/camel-k-seda-component-get-out-of-memory-error) 
   Can you please have a look and help? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] haifzhan commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
haifzhan commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-984978201


   @oscerd Thanks for your reply. SEDA uses in-memory queue by default, I am trying to replace the in memory queue with Kafaka, is it possible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] haifzhan commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
haifzhan commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-984980495


   I don't know how to do that. My understanding is the 2nd chunk's ` from("seda:next?concurrentConsumers=5")` would contains both kafka and seda. Is there any tutorial or where I can find some sample code? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] haifzhan commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
haifzhan commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-985047550


   I want to substitute SEDA with Kafka on the consumer side here's the pseudo code
   ```
   from("Netty listening on UDP")
   .to("Kafa Topic");
   
   from("SEDA taking events from Kafka Topic with multiple Consumers")
   .process("My procssor")
   .log();
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] github-actions[bot] closed issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #2808:
URL: https://github.com/apache/camel-k/issues/2808


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] oscerd commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-985233576


   Well you can do something like
   
   from(netty).to(kafka)
   
   from(kafka).to(seda)
   
   from(seda).process().log()


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] oscerd commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-984978855


   Sure, it's possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] oscerd commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-985041872


   Do you want to substitute seda with Kafka on the consumer side or you want to do something else? It's not really clear. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] oscerd commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-984867565


   What do you mean with Kafka for SEDA component? Camel provides the JVM trait, with it you could set a bigger heap and at least try to fine tuning the configuration. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [camel-k] github-actions[bot] commented on issue #2808: SEDA low performance issue

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2808:
URL: https://github.com/apache/camel-k/issues/2808#issuecomment-1058691423


   This issue has been automatically marked as stale due to 90 days of inactivity. 
   It will be closed if no further activity occurs within 15 days.
   If you think that’s incorrect or the issue should never stale, please simply write any comment.
   Thanks for your contributions!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org