RE: [EXTERNAL]Re: API use for Kafka Streams in KSQLDB

["Greer, Andrew C" <An...@conocophillips.com>]

Thank you. Is Faust a recommended library for accomplishing this in python? 

Andrew 

-----Original Message-----
From: Matthias J. Sax <mj...@apache.org> 
Sent: Wednesday, May 5, 2021 2:25 PM
To: users@kafka.apache.org
Subject: [EXTERNAL]Re: API use for Kafka Streams in KSQLDB

CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



In general, it's not recommended to call external APIs. Instead, you should try to get the external data into a TABLE in ksqlDB and do a stream-table join. Not sure if this would be possible for you use case.

It might be possible to write a UDF though... But given that the external call might fail, and you should not "block" inside the UDF it's would be a brittle approach.

Using Kafka Streams would give you more flexibility to do an external API can and handle failed requests and retries.


-Matthias


On 5/5/21 9:53 AM, Greer, Andrew C wrote:
> Hello,
>
> I am looking for options to call an external API within KSQL. My goal is to connect a Stream to a topic containing message data, call an API using some of that data, then produce updated data to new topics. Is there a known way for doing this? If not, any alternatives?
>
> Thank you for any future help,
>
> Andrew
>
>

Re: [EXTERNAL]Re: API use for Kafka Streams in KSQLDB

["Matthias J. Sax" <mj...@apache.org>]

Faust is an independent project, and we neither recommend not discourage
you from using it.

I personally only know that it exists, but don't know any details.


-Matthias

On 5/10/21 5:44 AM, Greer, Andrew C wrote:
> Thank you. Is Faust a recommended library for accomplishing this in python? 
> 
> Andrew 
> 
> -----Original Message-----
> From: Matthias J. Sax <mj...@apache.org> 
> Sent: Wednesday, May 5, 2021 2:25 PM
> To: users@kafka.apache.org
> Subject: [EXTERNAL]Re: API use for Kafka Streams in KSQLDB
> 
> CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> 
> In general, it's not recommended to call external APIs. Instead, you should try to get the external data into a TABLE in ksqlDB and do a stream-table join. Not sure if this would be possible for you use case.
> 
> It might be possible to write a UDF though... But given that the external call might fail, and you should not "block" inside the UDF it's would be a brittle approach.
> 
> Using Kafka Streams would give you more flexibility to do an external API can and handle failed requests and retries.
> 
> 
> -Matthias
> 
> 
> On 5/5/21 9:53 AM, Greer, Andrew C wrote:
>> Hello,
>>
>> I am looking for options to call an external API within KSQL. My goal is to connect a Stream to a topic containing message data, call an API using some of that data, then produce updated data to new topics. Is there a known way for doing this? If not, any alternatives?
>>
>> Thank you for any future help,
>>
>> Andrew
>>
>>