You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Michael Hempel Jørgensen <mh...@trifork.com> on 2023/04/18 06:45:05 UTC
Opensearch connector and oauth
Hi,
we need to use OAuth2 (Client Credentials Flow) in Flink to authenticate and authorise against different services, initially Kafka and Opensearch. We have it working with Kafka using however it doesn't seem to be possible with the Opensearch Flink Connector (https://github.com/apache/flink-connector-opensearch).
It looks like the connector only supports basic auth (user/pass) at the moment, and doesn't have an option for manually setting custom headers either. Is this correct?
Is there any work in progress towards making this possible?
We are willing to help contribute to code/discussions if necessary to get this implemented somehow.
Regards,
Michael
--
Michael Hempel-Jørgensen
Software Pilot
www.trifork.com
[cid:989723fe-9102-42cb-b034-267b79b2ca29]
Re: Opensearch connector and oauth
Posted by Andriy Redko <dr...@gmail.com>.
Hi Michael,
You are using the right one, it just lacks the support for Opensearch REST client customization at the moment. It would make sense to provide this functionality.
Thank you.
Best Regards,
Andriy Redko
MHJ> Hi Andriy,
MHJ> we are currently use the OpensearchSink[1] connector, as we assumed that that was the one to use going forwards. We are not quite sure what the difference is and if any will become deprecated soon?
MHJ> Regards,
MHJ> Michael
MHJ> [1] https://github.com/apache/flink-connector-opensearch/blob/main/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/sink/OpensearchSink.java
MHJ> ________________________________
MHJ> From: Andriy Redko <dr...@gmail.com>
MHJ> Sent: Tuesday, 18 April 2023 4:16:45 PM
MHJ> To: Martijn Visser; Michael Hempel Jørgensen
MHJ> Cc: user@flink.apache.org
MHJ> Subject: Re: Opensearch connector and oauth
MHJ> Hi Michael,
MHJ> Indeed the Opensearch REST client supports only basic auth at the moment. I don't know which sink are you using, but the
MHJ> streaming/connectors/opensearch/OpensearchSink provides the capability to configure the underlying client using
MHJ> RestClientFactory [1] so you could add the request / response interceptors to propagate OAuth2 tokens etc. This
MHJ> is not possible with other sinks yet but I think it should not be difficult to implement. If this is the case
MHJ> for you, could you please open the JIRA ticket?
MHJ> Thank you.
MHJ> [1] https://github.com/apache/flink-connector-opensearch/blob/main/flink-connector-opensearch/src/main/java/org/apache/flink/streaming/connectors/opensearch/RestClientFactory.java
MHJ> Best Regards,
MHJ> Andriy Redko
MHJ> Tuesday, April 18, 2023, 3:47:22 AM, you wrote:
>> Hi Michael,
>> I'm looping in Andrey since he has worked a lot on the Opensearch connector. A contribution is very welcome in case this can be improved.
>> Best regards,
>> Martijn
>> On Tue, Apr 18, 2023 at 8:45 AM Michael Hempel Jørgensen <mh...@trifork.com> wrote:
>>> Hi,
>>> we need to use OAuth2 (Client Credentials Flow) in Flink to authenticate and authorise against different services, initially Kafka and Opensearch. We have it working with Kafka using however it doesn't seem to be possible with the Opensearch Flink Connector (https://github.com/apache/flink-connector-opensearch).
>>> It looks like the connector only supports basic auth (user/pass) at the moment, and doesn't have an option for manually setting custom headers either. Is this correct?
>>> Is there any work in progress towards making this possible?
>>> We are willing to help contribute to code/discussions if necessary to get this implemented somehow.
>>> Regards,
>>> Michael
Re: Opensearch connector and oauth
Posted by Michael Hempel Jørgensen <mh...@trifork.com>.
Hi Andriy,
we are currently use the OpensearchSink[1] connector, as we assumed that that was the one to use going forwards. We are not quite sure what the difference is and if any will become deprecated soon?
Regards,
Michael
[1] https://github.com/apache/flink-connector-opensearch/blob/main/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/sink/OpensearchSink.java
________________________________
From: Andriy Redko <dr...@gmail.com>
Sent: Tuesday, 18 April 2023 4:16:45 PM
To: Martijn Visser; Michael Hempel Jørgensen
Cc: user@flink.apache.org
Subject: Re: Opensearch connector and oauth
Hi Michael,
Indeed the Opensearch REST client supports only basic auth at the moment. I don't know which sink are you using, but the
streaming/connectors/opensearch/OpensearchSink provides the capability to configure the underlying client using
RestClientFactory [1] so you could add the request / response interceptors to propagate OAuth2 tokens etc. This
is not possible with other sinks yet but I think it should not be difficult to implement. If this is the case
for you, could you please open the JIRA ticket?
Thank you.
[1] https://github.com/apache/flink-connector-opensearch/blob/main/flink-connector-opensearch/src/main/java/org/apache/flink/streaming/connectors/opensearch/RestClientFactory.java
Best Regards,
Andriy Redko
Tuesday, April 18, 2023, 3:47:22 AM, you wrote:
> Hi Michael,
> I'm looping in Andrey since he has worked a lot on the Opensearch connector. A contribution is very welcome in case this can be improved.
> Best regards,
> Martijn
> On Tue, Apr 18, 2023 at 8:45 AM Michael Hempel Jørgensen <mh...@trifork.com> wrote:
>> Hi,
>> we need to use OAuth2 (Client Credentials Flow) in Flink to authenticate and authorise against different services, initially Kafka and Opensearch. We have it working with Kafka using however it doesn't seem to be possible with the Opensearch Flink Connector (https://github.com/apache/flink-connector-opensearch).
>> It looks like the connector only supports basic auth (user/pass) at the moment, and doesn't have an option for manually setting custom headers either. Is this correct?
>> Is there any work in progress towards making this possible?
>> We are willing to help contribute to code/discussions if necessary to get this implemented somehow.
>> Regards,
>> Michael
Re: Opensearch connector and oauth
Posted by Andriy Redko <dr...@gmail.com>.
Hi Michael,
Indeed the Opensearch REST client supports only basic auth at the moment. I don't know which sink are you using, but the
streaming/connectors/opensearch/OpensearchSink provides the capability to configure the underlying client using
RestClientFactory [1] so you could add the request / response interceptors to propagate OAuth2 tokens etc. This
is not possible with other sinks yet but I think it should not be difficult to implement. If this is the case
for you, could you please open the JIRA ticket?
Thank you.
[1] https://github.com/apache/flink-connector-opensearch/blob/main/flink-connector-opensearch/src/main/java/org/apache/flink/streaming/connectors/opensearch/RestClientFactory.java
Best Regards,
Andriy Redko
Tuesday, April 18, 2023, 3:47:22 AM, you wrote:
> Hi Michael,
> I'm looping in Andrey since he has worked a lot on the Opensearch connector. A contribution is very welcome in case this can be improved.
> Best regards,
> Martijn
> On Tue, Apr 18, 2023 at 8:45 AM Michael Hempel Jørgensen <mh...@trifork.com> wrote:
>> Hi,
>> we need to use OAuth2 (Client Credentials Flow) in Flink to authenticate and authorise against different services, initially Kafka and Opensearch. We have it working with Kafka using however it doesn't seem to be possible with the Opensearch Flink Connector (https://github.com/apache/flink-connector-opensearch).
>> It looks like the connector only supports basic auth (user/pass) at the moment, and doesn't have an option for manually setting custom headers either. Is this correct?
>> Is there any work in progress towards making this possible?
>> We are willing to help contribute to code/discussions if necessary to get this implemented somehow.
>> Regards,
>> Michael
Re: Opensearch connector and oauth
Posted by Martijn Visser <ma...@apache.org>.
Hi Michael,
I'm looping in Andrey since he has worked a lot on the Opensearch
connector. A contribution is very welcome in case this can be improved.
Best regards,
Martijn
On Tue, Apr 18, 2023 at 8:45 AM Michael Hempel Jørgensen <mh...@trifork.com>
wrote:
> Hi,
>
> we need to use OAuth2 (Client Credentials Flow) in Flink to authenticate
> and authorise against different services, initially Kafka and Opensearch.
> We have it working with Kafka using however it doesn't seem to be possible
> with the Opensearch Flink Connector (
> https://github.com/apache/flink-connector-opensearch).
>
> It looks like the connector only supports basic auth (user/pass) at the
> moment, and doesn't have an option for manually setting custom headers
> either. Is this correct?
>
> Is there any work in progress towards making this possible?
>
> We are willing to help contribute to code/discussions if necessary to get
> this implemented somehow.
>
> Regards,
> Michael
>
> --
> Michael Hempel-Jørgensen
> Software Pilot
> www.trifork.com
>
>
>