Communication between PE and external service

[Branislav Jovicic <br...@nissatech.com>]

Hi everyone,

Me and my colleagues are working on a series of pipeline elements that need to communicate with our external service (a service which is not related to the StreamPipes).
Basically, we imagined that each of these elements receives an Event, sends Event data to the external service, this service processes the data, sends it back to the pipeline element and, finally, pipeline element forwards this data to the next element in the pipeline. Ideally, the external service processing and data exchange, both between element & service and between elements, should be executed before new Event arrives. But we cannot be sure that this is always going to be the case.

Based on all of this, we singled out several ways of achieving mentioned communication:

  1.  The simplest way to achieve this would be to send an HTTP request to the external service and to wait for its response with the resulting data. This method raises following questions:
     *   If pipeline element waits for the response, that means that the main thread is blocked. If, for whatever reason, this element does not receive the response before new Event arrives, what would happen to this new Event?
     *   If element is not able to process Event as soon as it gets sent by the previous element, does this Event gets lost, or is there a retention period within used message broker for exchanging Events?
     *   If there is a retention period, how long is it? Is there an option to configure it?
     *   What message broker is used by default?
     *   We could use a separate thread which would send a request to the external service, wait for the response and forward it to the next pipeline element. In this case, the main thread does not get blocked and can receive new Events if necessary. Would this be a good solution?
  2.  Implement an active endpoint within each pipeline element to which the external service sends the results:
     *   What would be the best possible way to define an endpoint?
     *   For example, if we use Spring, I suppose that all of the initialization would go into the onInvocation method of "runtime" class. Would this block the main thread and disable it from calling onEvent method (because this thread needs to listen for new requests)? If so, what would the workaround?
     *   What framework would you recommend?
  3.  Implement separate component that will be started every time StreamPipes is started (StreamPipes service). This component would receive responses from the external service for all the pipeline elements and forward it to the appropriate one (publish/subscribe architecture between pipeline elements and said component).
     *   This way, external service always uses single endpoint to send back the results to the component.
     *   Pipeline elements would subscribe to appropriate topic of some message broker to which the component publishes the results from the external service.
     *   Would we be able to use one of the existing message broker services started by the StreamPipes? If yes, what would be the best way of doing this?

According to your experience, what would the best option be? Is there some other option that we possibly missed?
Any help is greatly appreciated.

Kind regards,
Branislav