You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "Sugandhab (via GitHub)" <gi...@apache.org> on 2025/03/24 11:01:35 UTC

[D] Is There an Existing Flow to Send Data to REST API and Process Response?

GitHub user Sugandhab created a discussion: Is There an Existing Flow to Send Data to REST API and Process Response?

Hi team,

Iโ€™m exploring a use case where I need to send incoming weather data to an external REST API (AI simulator) and use its response to continue processing within the same pipeline (e.g., store prediction in a database).

Before I go ahead and build a custom processor, I wanted to check:
๐Ÿ‘‰ Is there any existing processor or flow in StreamPipes that supports sending data to a REST API and handling the response as part of the event stream?

Would appreciate any pointers or examples if something similar exists.

Thanks in advance!

GitHub link: https://github.com/apache/streampipes/discussions/3538

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Is There an Existing Flow to Send Data to REST API and Process Response?

Posted by "tenthe (via GitHub)" <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Is There an Existing Flow to Send Data to REST API and Process Response?

Hi @Sugandhab,  

the function would be a separate service containing dedicated code, capable of subscribing to event streams and creating new ones.  
If you want to reuse your code in a pipeline, you need to implement a data processor.  

In general:  
- For complex tasks or tasks that have a fixed data stream, use a **Function**, because it provides a dedicated solution.
- For more flexibility and generic tasks, **Processors** work better, as they can be used in different pipelines and configured by user input.  

Hereโ€™s the documentation for Java Functions: [Link](https://streampipes.apache.org/docs/extend-sdk-functions/)

I hope this helps!

GitHub link: https://github.com/apache/streampipes/discussions/3538#discussioncomment-12611380

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Is There an Existing Flow to Send Data to REST API and Process Response?

Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer edited a comment on the discussion: Is There an Existing Flow to Send Data to REST API and Process Response?

Hi @Sugandhab if you want to create a pipeline element which calls a REST service and transforms the event stream with the response from the service, have a look at the SDK tutorial [1] and our own pipeline elements [2]. I recommend to start with the Maven archetype which generates the code skeleton for you.

[1] https://streampipes.apache.org/docs/extend-tutorial-data-processors/
[2] e.g., https://github.com/apache/streampipes/blob/dev/streampipes-extensions/streampipes-processors-enricher-jvm/src/main/java/org/apache/streampipes/processors/enricher/jvm/processor/limitsenrichment/QualityControlLimitsEnrichmentProcessor.java

GitHub link: https://github.com/apache/streampipes/discussions/3538#discussioncomment-12614509

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Is There an Existing Flow to Send Data to REST API and Process Response?

Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Is There an Existing Flow to Send Data to REST API and Process Response?

Hi @Sugandhab if you want to create a pipeline element which calls a REST service and transforms the event stream with the response from the service, have a look at the SDK tutorial [1] our own pipeline elements [2]. I recommend to start with the Maven archetype which generates the code skeleton for you.

[1] https://streampipes.apache.org/docs/extend-tutorial-data-processors/
[2] e.g., https://github.com/apache/streampipes/blob/dev/streampipes-extensions/streampipes-processors-enricher-jvm/src/main/java/org/apache/streampipes/processors/enricher/jvm/processor/limitsenrichment/QualityControlLimitsEnrichmentProcessor.java

GitHub link: https://github.com/apache/streampipes/discussions/3538#discussioncomment-12614509

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Is There an Existing Flow to Send Data to REST API and Process Response?

Posted by "Sugandhab (via GitHub)" <gi...@apache.org>.
GitHub user Sugandhab added a comment to the discussion: Is There an Existing Flow to Send Data to REST API and Process Response?

I understand that we can use functions for such scenarios but how do we connect these functions to pipelines?
also do we have any Java sdk examples available so that I can run through it and can understand end to end process.

GitHub link: https://github.com/apache/streampipes/discussions/3538#discussioncomment-12611123

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Is There an Existing Flow to Send Data to REST API and Process Response?

Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Is There an Existing Flow to Send Data to REST API and Process Response?

Hey @Sugandhab 

For your use case StreamPipes Functions is probably the way to go.
Here is [one example](https://streampipes.apache.org/docs/docs/python/latest/tutorials/3-getting-live-data-from-the-streampipes-data-stream/) from our Python SDK. Of course, functions are also available in the [Java SDK](https://streampipes.apache.org/docs/extend-sdk-functions/) if that suits you more. 

Let me here what you think about it and if you need some help to get it started ๐Ÿ™‚ 

GitHub link: https://github.com/apache/streampipes/discussions/3538#discussioncomment-12606246

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org