You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/02/02 09:33:53 UTC
[GitHub] [camel-k] lancerdima opened a new issue #1984: Options to share state between routes
lancerdima opened a new issue #1984:
URL: https://github.com/apache/camel-k/issues/1984
What are the options to share state between multiple flows/routes in the context of single integration?
My need basically boils down to this scenario:
* capture a property to something like exchangeProperty, with set-property step
* be able to use the same property, like ${exchangeProperty.prop} with simple language in another unrelated route in the same integration
**Currently exchangeProprty is not retained.**
It is expected that this can be done with pure Camel DSL - e.g. with YAML.
I understand that this functionality can be achieved with the help of custom java bean, but I would like to avoid use of custom code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima edited a comment on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima edited a comment on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-773227390
Leaving the snippets below with a solution to the shared state problem, as a contribution to the community.
Valid for Camel K 1.3 runtime.
In this case:
* primary route retrieves event from Salesforce (can be any) and captures the offset of the topic
* secondary route, every 10 seconds does something useful with the offset
YAML integration definition
```
- from:
uri: "salesforce:event/PlatformEvent__e"
steps:
- {some useful stuff}
- {some more useful stuff}
- to: bean:cache?method=set("offset", ${headers.CamelSalesforceReplayId}) # Capture the offset
- from:
uri: "timer:offsetHandler?fixedRate=true&period=10000&delay=10000"
steps:
- to: bean:cache?method=get("offset") # Retrieve the offset from cache
- log: "Offset Store: ${body}"
- beans:
- name: cache
type: mycache.InMemoryKeyValueStore
```
Bean to capture state
```
package mycache;
import org.apache.camel.Handler;
import org.apache.camel.builder.RouteBuilder;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class InMemoryKeyValueStore extends RouteBuilder {
@Override
public void configure() throws Exception {
}
private Map<String, Object> container;
private Map<String, Object> getContainer() {
if (this.container == null) {
this.container = new ConcurrentHashMap<>();
}
return this.container;
}
@Handler
public Object get(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
return container.get(key);
}
return null;
}
@Handler
public void remove(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
container.remove(key);
}
}
@Handler
public void set(String key, Object value) {
Map<String, Object> container = this.getContainer();
container.put(key, value);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima closed issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima closed issue #1984:
URL: https://github.com/apache/camel-k/issues/1984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-771505850
can you provide a more concrete example ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-773227390
Leaving the snippets below with a solution to the shared state problem, as a contribution to the community.
Valid for Camel K 1.3 runtime.
YAML integration definition
```
- from:
uri: "salesforce:event/PlatformEvent__e"
steps:
- {some useful stuff}
- {some more useful stuff}
- to: bean:cache?method=set("offset", ${headers.CamelSalesforceReplayId}) # Capture the offset
- from:
uri: "timer:offsetHandler?fixedRate=true&period=10000&delay=10000"
steps:
- to: bean:cache?method=get("offset") # Retrieve the offset from cache
- log: "Offset Store: ${body}"
- beans:
- name: cache
type: mycache.InMemoryKeyValueStore
```
Bean to capture state
```
package mycache;
import org.apache.camel.Handler;
import org.apache.camel.builder.RouteBuilder;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class InMemoryKeyValueStore extends RouteBuilder {
@Override
public void configure() throws Exception {
}
private Map<String, Object> container;
private Map<String, Object> getContainer() {
if (this.container == null) {
this.container = new ConcurrentHashMap<>();
}
return this.container;
}
@Handler
public Object get(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
return container.get(key);
}
return null;
}
@Handler
public void remove(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
container.remove(key);
}
}
@Handler
public void set(String key, Object value) {
Map<String, Object> container = this.getContainer();
container.put(key, value);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima closed issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima closed issue #1984:
URL: https://github.com/apache/camel-k/issues/1984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima edited a comment on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima edited a comment on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-773227390
Leaving the snippets below with a solution to the shared state problem, as a contribution to the community.
Valid for Camel K 1.3 runtime.
In this case:
* primary route retrieves event from Salesforce (can be any) and captures the offset of the topic
* secondary route, every 10 seconds does something useful with the offset
YAML integration definition
```
- from:
uri: "salesforce:event/PlatformEvent__e"
steps:
- {some useful stuff}
- {some more useful stuff}
- to: bean:cache?method=set("offset", ${headers.CamelSalesforceReplayId}) # Capture the offset
- from:
uri: "timer:offsetHandler?fixedRate=true&period=10000&delay=10000"
steps:
- to: bean:cache?method=get("offset") # Retrieve the offset from cache
- log: "Offset Store: ${body}"
- beans:
- name: cache
type: mycache.InMemoryKeyValueStore
```
Bean to capture state
```
package mycache;
import org.apache.camel.Handler;
import org.apache.camel.builder.RouteBuilder;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class InMemoryKeyValueStore extends RouteBuilder {
@Override
public void configure() throws Exception {
}
private Map<String, Object> container;
private Map<String, Object> getContainer() {
if (this.container == null) {
this.container = new ConcurrentHashMap<>();
}
return this.container;
}
@Handler
public Object get(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
return container.get(key);
}
return null;
}
@Handler
public void remove(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
container.remove(key);
}
}
@Handler
public void set(String key, Object value) {
Map<String, Object> container = this.getContainer();
container.put(key, value);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli edited a comment on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli edited a comment on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-772445419
share state between routes is not something specifically related to camel-k, and about DSL, well, it depends on the one that you are using.
To create a bean in the registry:
- use [properties](https://camel.apache.org/components/latest/others/main.html#_specifying_custom_beans)
- use [`beans`](https://github.com/apache/camel-k-runtime/blob/master/camel-k-loader-yaml/impl/src/test/resources/routes/RoutesTest_beans.yaml) inside YAML (not yet documented as it is largely experimental work in progress)
I do not know how easy it would be to create a caffeine cache but you can eventually use a ConcurrentHashMap and use simple|groovy language to access the value
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-773228549
@lancerdima wonder if you can just set the type of the beans to `java.util.concurrent.ConcurrentHashMap` so you could avoid the need of the java code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-773227390
Leaving the snippets below with a solution to the shared state problem, as a contribution to the community.
Valid for Camel K 1.3 runtime.
YAML integration definition
```
- from:
uri: "salesforce:event/PlatformEvent__e"
steps:
- {some useful stuff}
- {some more useful stuff}
- to: bean:cache?method=set("offset", ${headers.CamelSalesforceReplayId}) # Capture the offset
- from:
uri: "timer:offsetHandler?fixedRate=true&period=10000&delay=10000"
steps:
- to: bean:cache?method=get("offset") # Retrieve the offset from cache
- log: "Offset Store: ${body}"
- beans:
- name: cache
type: mycache.InMemoryKeyValueStore
```
Bean to capture state
```
package mycache;
import org.apache.camel.Handler;
import org.apache.camel.builder.RouteBuilder;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public class InMemoryKeyValueStore extends RouteBuilder {
@Override
public void configure() throws Exception {
}
private Map<String, Object> container;
private Map<String, Object> getContainer() {
if (this.container == null) {
this.container = new ConcurrentHashMap<>();
}
return this.container;
}
@Handler
public Object get(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
return container.get(key);
}
return null;
}
@Handler
public void remove(String key) {
Map<String, Object> container = this.getContainer();
if (container.containsKey(key)) {
container.remove(key);
}
}
@Handler
public void set(String key, Object value) {
Map<String, Object> container = this.getContainer();
container.put(key, value);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-773228549
@lancerdima wonder if you can just set the type of the beans to `java.util.concurrent.ConcurrentHashMap` so you could avoid the need of the java code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-771505850
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-771508723
> - from:
> uri: "salesforce:event/PlatformEvent__e"
> steps:
> - {some useful stuff}
> - {some more useful stuff}
> - set-property: # capture the shared state here
> name: commit_replay_id
> simple: "${headers.CamelSalesforceReplayId}"
> - from:
> uri: "timer:offsetHandler?fixedRate=true&period=10000&delay=10000"
> steps:
> - log: "${exchangeProperty.CamelSalesforceReplayId}" # use the shared state here
This is the yaml config I have tried.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-772440206
Tried that now - same negative result and behavior.
I believe that problem with camel-caffeine might be in [CaffeineCacheEndpoint.java, L67](https://github.com/apache/camel/blob/9529408743d72a591fea90250ea6b796524ae890/components/camel-caffeine/src/main/java/org/apache/camel/component/caffeine/cache/CaffeineCacheEndpoint.java) - cache object is retrieved from Camel Context, but I can't find any place where it is put into Camel Context in the first place.
Back to the original question. Do I understand correctly that with the current state of the Camel K, there are no Camel DSL constructs to interact with Context and share state across routes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] rzonta commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
rzonta commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-819373197
@lancerdima you need to bind a instance of caffeine cache into Camel Registry
```
import org.apache.camel.BindToRegistry;
import org.apache.camel.builder.RouteBuilder;
import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
public class CacheTest extends RouteBuilder {
@BindToRegistry
public static Cache<String, Object> cache() {
return Caffeine.newBuilder()
.build();
}
@Override
public void configure() throws Exception {
from("timer:offsetHandler?fixedRate=true&period=10000&delay=5000")
.to("caffeine-cache:cache?key=offset&action=GET")
.log("Old value: ${body}")
.setBody(simple("${headers.CamelTimerCounter}"))
.to("caffeine-cache:cache?action=PUT&key=offset")
.log("New value: ${body}");
}
}
```
Then the result will be what you expect:
```
[1] 2021-04-14 09:23:00,326 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value:
[1] 2021-04-14 09:23:00,330 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 1
[1] 2021-04-14 09:23:10,300 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value: 1
[1] 2021-04-14 09:23:10,301 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 2
[1] 2021-04-14 09:23:20,299 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value: 2
[1] 2021-04-14 09:23:20,300 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-772399778
Maybe I am missing the whole point of camel-caffeine, but it fails to cache value between multiple executions of the same flow.
Consider the snippet below. The point of the snippet, is to display "cached" value from previous execution first, and then update the cache with the new one.
```
import org.apache.camel.builder.RouteBuilder;
public class Caffeine extends RouteBuilder {
@Override
public void configure() throws Exception {
from("timer:offsetHandler?fixedRate=true&period=10000&delay=5000")
.to("caffeine-cache://cache?key=offset&action=GET")
.log("Old value: ${body}")
.setBody(simple("${headers.CamelTimerCounter}"))
.to("caffeine-cache://cache?action=PUT&key=offset")
.log("New value: ${body}");
}
}
```
Output log indicate that value from previous execution is lost:
```
[1] 2021-02-03 10:13:15,031 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value:
[1] 2021-02-03 10:13:15,033 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 1
[1] 2021-02-03 10:13:25,018 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value:
[1] 2021-02-03 10:13:25,021 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 2
[1] 2021-02-03 10:13:35,017 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value:
[1] 2021-02-03 10:13:35,018 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 3
[1] 2021-02-03 10:13:45,017 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) Old value:
[1] 2021-02-03 10:13:45,018 INFO [route1] (Camel (camel-1) thread #0 - timer://offsetHandler) New value: 4
```
Not sure whether it is camel-caffeine problem, or a discrepancy in how Camel K vs classic Camel behaves.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-772402694
Have you tried with "classic" camel ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-772445419
share state between routes is not something specifically related to camel-k, and about DSL, well, it depends on the one that you are using.
To create a bean in the registry:
- use properties https://camel.apache.org/components/latest/others/main.html#_specifying_custom_beans
- use [`beans`](https://github.com/apache/camel-k-runtime/blob/master/camel-k-loader-yaml/impl/src/test/resources/routes/RoutesTest_beans.yaml) inside YAML (not yet documented as it is largely experimental work in progress)
I do not know how easy it would be to create a caffeine cache but you can eventually use a ConcurrentHashMap and use simple|groovy language to access the value
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lancerdima commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lancerdima commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-771508723
> - from:
> uri: "salesforce:event/PlatformEvent__e"
> steps:
> - {some useful stuff}
> - {some more useful stuff}
> - set-property: # capture the shared state here
> name: commit_replay_id
> simple: "${headers.CamelSalesforceReplayId}"
> - from:
> uri: "timer:offsetHandler?fixedRate=true&period=10000&delay=10000"
> steps:
> - log: "${exchangeProperty.CamelSalesforceReplayId}" # use the shared state here
This is the yaml config I have tried.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli edited a comment on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli edited a comment on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-772445419
share state between routes is not something specifically related to camel-k, and about DSL, well, it depends on the one that you are using.
To create a bean in the registry:
- use [properties](https://camel.apache.org/components/latest/others/main.html#_specifying_custom_beans)
- use [`beans`](https://github.com/apache/camel-k-runtime/blob/master/camel-k-loader-yaml/impl/src/test/resources/routes/RoutesTest_beans.yaml) inside YAML (not yet documented as it is largely experimental work in progress)
- use a java/groovy/kotlin/js DSL and programmatically bind your beans to the context
I do not know how easy it would be to create a caffeine cache but you can eventually use a ConcurrentHashMap and use simple|groovy language to access the value
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1984: Options to share state between routes
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1984:
URL: https://github.com/apache/camel-k/issues/1984#issuecomment-771514619
You could use camel-caffeine so you have a shared cache to store your bits
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org