You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vijay Balakrishnan <bv...@gmail.com> on 2018/10/02 22:49:19 UTC
Using FlinkKinesisConsumer through a proxy
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ?
Getting a Connection issue through the proxy.
Works outside the proxy.
Properties kinesisConsumerConfig = new Properties();
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION,
region);
if (local) {
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID,
accessKey);
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY,
secretKey);
} else {
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER,
"AUTO");
}
//only for Consumer
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX,
"10000");
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
"2000");
FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>>
kinesisConsumer = new FlinkKinesisConsumer<>(
"kinesisTopicRead", new Tuple2KinesisSchema(),
kinesisConsumerConfig);
TIA
Re: Using FlinkKinesisConsumer through a proxy
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Good to hear that it's working, thanks for the update!
On Sat, Dec 1, 2018, 4:29 AM Vijay Balakrishnan <bvijaykr@gmail.com wrote:
> Hi Gordon,
> Finally figured out my issue.Do not need to add http:// in proxyHost name.
> String proxyHost= "proxy-chain....com";//not http://proxy-chain...com
> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "proxyHost", proxyHost);//<== mo http:// in proxyHost name
>
> TIA,
> Vijay
>
>
> On Wed, Nov 14, 2018 at 12:50 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi Vijay,
>>
>> I’m pretty sure that this should work with the properties that you
>> provided, unless the AWS Kinesis SDK isn’t working as expected.
>>
>> What I’ve tested is that with those properties, the ClientConfiguration
>> used to build the Kinesis client has the proxy domain / host / ports etc.
>> properly set.
>> And according to [1], this should be enough to configure the constructed
>> Kinesis client to connect via the proxy.
>>
>> Cheers,
>> Gordon
>>
>> [1]
>> https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html
>>
>>
>> On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan (bvijaykr@gmail.com)
>> wrote:
>>
>> Hi Gordon,
>> This still didn't work :(
>>
>> Tried a few combinations with:
>> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
>> "proxyDomain", "...");
>>
>> inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
>> "proxyHost", "http://.....com");
>>
>> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
>> "proxyPort", "911");
>>
>> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
>> "proxyUsername", "...");
>>
>> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
>> "proxyPassword", "..");
>>
>> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
>> "nonProxyHosts", "
>>
>>
>> How does the FlinkKinesisProducer work so seamlessly through a proxy ?
>> TIA,
>> Vijay
>>
>> On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> Since Flink 1.5, you should be able to set all available configurations
>>> on the ClientConfiguration through the consumer Properties (see FLINK-9188
>>> [1]).
>>>
>>> The way to do that would be to prefix the configuration you want to set
>>> with "aws.clientconfig" and add that to the properties, as such:
>>>
>>> ```
>>> Properties kinesisConsumerProps = new Properties();
>>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
>>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
>>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
>>> ...
>>> ```
>>>
>>> Could you try that out and see if it works for you?
>>>
>>> I've also realized that this feature isn't documented very well, and
>>> have opened a ticket for that [2].
>>>
>>> Cheers,
>>> Gordon
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-9188
>>> [2] https://issues.apache.org/jira/browse/FLINK-10492
>>>
>>> On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <al...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm looping in Gordon and Thomas, they might have some idea about how
>>>> to resolve this.
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bv...@gmail.com>
>>>> wrote:
>>>>
>>>> I have been trying with all variations to no avail of java
>>>> -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://...
>>>> -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=..
>>>> -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=...
>>>> -Dhttp.proxyPassword=... -jar .. after looking at the code in
>>>> com.amazonaws.ClientConfiguration
>>>>
>>>> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bv...@gmail.com>
>>>> wrote:
>>>>
>>>>> HI,
>>>>> How do I use FlinkKinesisConsumer using the Properties through a proxy
>>>>> ? Getting a Connection issue through the proxy.
>>>>> Works outside the proxy.
>>>>>
>>>>> Properties kinesisConsumerConfig = new Properties();
>>>>>
>>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);
>>>>>
>>>>> if (local) {
>>>>>
>>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID,
>>>>> accessKey);
>>>>>
>>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY,
>>>>> secretKey);
>>>>> } else {
>>>>>
>>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER,
>>>>> "AUTO");
>>>>> }
>>>>>
>>>>> //only for Consumer
>>>>>
>>>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX,
>>>>> "10000");
>>>>>
>>>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
>>>>> "2000");
>>>>> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>>
>>>>> kinesisConsumer = new FlinkKinesisConsumer<>(
>>>>> "kinesisTopicRead", new Tuple2KinesisSchema(),
>>>>> kinesisConsumerConfig);
>>>>> TIA
>>>>>
>>>>
>>>>
Re: Using FlinkKinesisConsumer through a proxy
Posted by Vijay Balakrishnan <bv...@gmail.com>.
Hi Gordon,
Finally figured out my issue.Do not need to add http:// in proxyHost name.
String proxyHost= "proxy-chain....com";//not http://proxy-chain...com
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"proxyHost", proxyHost);//<== mo http:// in proxyHost name
TIA,
Vijay
On Wed, Nov 14, 2018 at 12:50 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi Vijay,
>
> I’m pretty sure that this should work with the properties that you
> provided, unless the AWS Kinesis SDK isn’t working as expected.
>
> What I’ve tested is that with those properties, the ClientConfiguration
> used to build the Kinesis client has the proxy domain / host / ports etc.
> properly set.
> And according to [1], this should be enough to configure the constructed
> Kinesis client to connect via the proxy.
>
> Cheers,
> Gordon
>
> [1]
> https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html
>
>
> On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan (bvijaykr@gmail.com)
> wrote:
>
> Hi Gordon,
> This still didn't work :(
>
> Tried a few combinations with:
> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "proxyDomain", "...");
>
> inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "proxyHost", "http://.....com");
>
> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "proxyPort", "911");
>
> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "proxyUsername", "...");
>
> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "proxyPassword", "..");
>
> kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
> "nonProxyHosts", "
>
>
> How does the FlinkKinesisProducer work so seamlessly through a proxy ?
> TIA,
> Vijay
>
> On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi,
>>
>> Since Flink 1.5, you should be able to set all available configurations
>> on the ClientConfiguration through the consumer Properties (see FLINK-9188
>> [1]).
>>
>> The way to do that would be to prefix the configuration you want to set
>> with "aws.clientconfig" and add that to the properties, as such:
>>
>> ```
>> Properties kinesisConsumerProps = new Properties();
>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
>> kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
>> ...
>> ```
>>
>> Could you try that out and see if it works for you?
>>
>> I've also realized that this feature isn't documented very well, and have
>> opened a ticket for that [2].
>>
>> Cheers,
>> Gordon
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-9188
>> [2] https://issues.apache.org/jira/browse/FLINK-10492
>>
>> On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <al...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm looping in Gordon and Thomas, they might have some idea about how to
>>> resolve this.
>>>
>>> Best,
>>> Aljoscha
>>>
>>> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bv...@gmail.com> wrote:
>>>
>>> I have been trying with all variations to no avail of java
>>> -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://...
>>> -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=..
>>> -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=...
>>> -Dhttp.proxyPassword=... -jar .. after looking at the code in
>>> com.amazonaws.ClientConfiguration
>>>
>>> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bv...@gmail.com>
>>> wrote:
>>>
>>>> HI,
>>>> How do I use FlinkKinesisConsumer using the Properties through a proxy
>>>> ? Getting a Connection issue through the proxy.
>>>> Works outside the proxy.
>>>>
>>>> Properties kinesisConsumerConfig = new Properties();
>>>>
>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);
>>>>
>>>> if (local) {
>>>>
>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID,
>>>> accessKey);
>>>>
>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY,
>>>> secretKey);
>>>> } else {
>>>>
>>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER,
>>>> "AUTO");
>>>> }
>>>>
>>>> //only for Consumer
>>>>
>>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX,
>>>> "10000");
>>>>
>>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
>>>> "2000");
>>>> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>>
>>>> kinesisConsumer = new FlinkKinesisConsumer<>(
>>>> "kinesisTopicRead", new Tuple2KinesisSchema(),
>>>> kinesisConsumerConfig);
>>>> TIA
>>>>
>>>
>>>
Re: Using FlinkKinesisConsumer through a proxy
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Vijay,
I’m pretty sure that this should work with the properties that you provided, unless the AWS Kinesis SDK isn’t working as expected.
What I’ve tested is that with those properties, the ClientConfiguration used to build the Kinesis client has the proxy domain / host / ports etc. properly set.
And according to [1], this should be enough to configure the constructed Kinesis client to connect via the proxy.
Cheers,
Gordon
[1] https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html
On 7 November 2018 at 1:19:02 AM, Vijay Balakrishnan (bvijaykr@gmail.com) wrote:
Hi Gordon,
This still didn't work :(
Tried a few combinations with:
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyDomain", "...");
inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyHost", "http://.....com");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyPort", "911");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyUsername", "...");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "proxyPassword", "..");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX + "nonProxyHosts", "
How does the FlinkKinesisProducer work so seamlessly through a proxy ?
TIA,
Vijay
On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tz...@apache.org> wrote:
Hi,
Since Flink 1.5, you should be able to set all available configurations on the ClientConfiguration through the consumer Properties (see FLINK-9188 [1]).
The way to do that would be to prefix the configuration you want to set with "aws.clientconfig" and add that to the properties, as such:
```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```
Could you try that out and see if it works for you?
I've also realized that this feature isn't documented very well, and have opened a ticket for that [2].
Cheers,
Gordon
[1] https://issues.apache.org/jira/browse/FLINK-9188
[2] https://issues.apache.org/jira/browse/FLINK-10492
On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <al...@apache.org> wrote:
Hi,
I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.
Best,
Aljoscha
On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bv...@gmail.com> wrote:
I have been trying with all variations to no avail of java -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration
On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bv...@gmail.com> wrote:
HI,
How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy.
Works outside the proxy.
Properties kinesisConsumerConfig = new Properties();
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);
if (local) {
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
} else {
kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
}
//only for Consumer
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
"kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
TIA
Re: Using FlinkKinesisConsumer through a proxy
Posted by Vijay Balakrishnan <bv...@gmail.com>.
Hi Gordon,
This still didn't work :(
Tried a few combinations with:
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"proxyDomain", "...");
inesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"proxyHost", "http://.....com");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"proxyPort", "911");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"proxyUsername", "...");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"proxyPassword", "..");
kinesisConsumerConfig.setProperty(AWSUtil.AWS_CLIENT_CONFIG_PREFIX +
"nonProxyHosts", "
How does the FlinkKinesisProducer work so seamlessly through a proxy ?
TIA,
Vijay
On Thu, Oct 4, 2018 at 6:41 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi,
>
> Since Flink 1.5, you should be able to set all available configurations on
> the ClientConfiguration through the consumer Properties (see FLINK-9188
> [1]).
>
> The way to do that would be to prefix the configuration you want to set
> with "aws.clientconfig" and add that to the properties, as such:
>
> ```
> Properties kinesisConsumerProps = new Properties();
> kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
> kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
> kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
> ...
> ```
>
> Could you try that out and see if it works for you?
>
> I've also realized that this feature isn't documented very well, and have
> opened a ticket for that [2].
>
> Cheers,
> Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-9188
> [2] https://issues.apache.org/jira/browse/FLINK-10492
>
> On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <al...@apache.org> wrote:
>
>> Hi,
>>
>> I'm looping in Gordon and Thomas, they might have some idea about how to
>> resolve this.
>>
>> Best,
>> Aljoscha
>>
>> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bv...@gmail.com> wrote:
>>
>> I have been trying with all variations to no avail of java
>> -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://...
>> -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=..
>> -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=...
>> -Dhttp.proxyPassword=... -jar .. after looking at the code in
>> com.amazonaws.ClientConfiguration
>>
>> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bv...@gmail.com>
>> wrote:
>>
>>> HI,
>>> How do I use FlinkKinesisConsumer using the Properties through a proxy ?
>>> Getting a Connection issue through the proxy.
>>> Works outside the proxy.
>>>
>>> Properties kinesisConsumerConfig = new Properties();
>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION,
>>> region);
>>>
>>> if (local) {
>>>
>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID,
>>> accessKey);
>>>
>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY,
>>> secretKey);
>>> } else {
>>>
>>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER,
>>> "AUTO");
>>> }
>>>
>>> //only for Consumer
>>>
>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX,
>>> "10000");
>>>
>>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
>>> "2000");
>>> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>>
>>> kinesisConsumer = new FlinkKinesisConsumer<>(
>>> "kinesisTopicRead", new Tuple2KinesisSchema(),
>>> kinesisConsumerConfig);
>>> TIA
>>>
>>
>>
Re: Using FlinkKinesisConsumer through a proxy
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
Since Flink 1.5, you should be able to set all available configurations on
the ClientConfiguration through the consumer Properties (see FLINK-9188
[1]).
The way to do that would be to prefix the configuration you want to set
with "aws.clientconfig" and add that to the properties, as such:
```
Properties kinesisConsumerProps = new Properties();
kinesisConsumerProps.setProperty("aws.clientconfig.proxyHost", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyPort", ...);
kinesisConsumerProps.setProperty("aws.clientconfig.proxyUsert", ...);
...
```
Could you try that out and see if it works for you?
I've also realized that this feature isn't documented very well, and have
opened a ticket for that [2].
Cheers,
Gordon
[1] https://issues.apache.org/jira/browse/FLINK-9188
[2] https://issues.apache.org/jira/browse/FLINK-10492
On Thu, Oct 4, 2018, 7:57 PM Aljoscha Krettek <al...@apache.org> wrote:
> Hi,
>
> I'm looping in Gordon and Thomas, they might have some idea about how to
> resolve this.
>
> Best,
> Aljoscha
>
> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bv...@gmail.com> wrote:
>
> I have been trying with all variations to no avail of java
> -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://...
> -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=..
> -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=...
> -Dhttp.proxyPassword=... -jar .. after looking at the code in
> com.amazonaws.ClientConfiguration
>
> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bv...@gmail.com>
> wrote:
>
>> HI,
>> How do I use FlinkKinesisConsumer using the Properties through a proxy ?
>> Getting a Connection issue through the proxy.
>> Works outside the proxy.
>>
>> Properties kinesisConsumerConfig = new Properties();
>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION,
>> region);
>>
>> if (local) {
>>
>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID,
>> accessKey);
>>
>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY,
>> secretKey);
>> } else {
>>
>> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER,
>> "AUTO");
>> }
>>
>> //only for Consumer
>>
>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX,
>> "10000");
>>
>> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
>> "2000");
>> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>>
>> kinesisConsumer = new FlinkKinesisConsumer<>(
>> "kinesisTopicRead", new Tuple2KinesisSchema(),
>> kinesisConsumerConfig);
>> TIA
>>
>
>
Re: Using FlinkKinesisConsumer through a proxy
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I'm looping in Gordon and Thomas, they might have some idea about how to resolve this.
Best,
Aljoscha
> On 3. Oct 2018, at 17:29, Vijay Balakrishnan <bv...@gmail.com> wrote:
>
> I have been trying with all variations to no avail of java -Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911 -Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://.. -Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar .. after looking at the code in com.amazonaws.ClientConfiguration
>
> On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bvijaykr@gmail.com <ma...@gmail.com>> wrote:
> HI,
> How do I use FlinkKinesisConsumer using the Properties through a proxy ? Getting a Connection issue through the proxy.
> Works outside the proxy.
>
> Properties kinesisConsumerConfig = new Properties();
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION, region);
>
> if (local) {
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID, accessKey);
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY, secretKey);
> } else {
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER, "AUTO");
> }
>
> //only for Consumer
> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "10000");
> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "2000");
> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>> kinesisConsumer = new FlinkKinesisConsumer<>(
> "kinesisTopicRead", new Tuple2KinesisSchema(), kinesisConsumerConfig);
>
>
> TIA
Re: Using FlinkKinesisConsumer through a proxy
Posted by Vijay Balakrishnan <bv...@gmail.com>.
I have been trying with all variations to no avail of java
-Dhttp.nonProxyHosts=.. -Dhttps.proxyHost=http://... -Dhttps.proxyPort=911
-Dhttps.proxyUser= -Dhttps.proxyPassword=.. -Dhttp.proxyHost=http://..
-Dhttp.proxyPort=911 -Dhttp.proxyUser=... -Dhttp.proxyPassword=... -jar ..
after looking at the code in com.amazonaws.ClientConfiguration
On Tue, Oct 2, 2018 at 3:49 PM Vijay Balakrishnan <bv...@gmail.com>
wrote:
> HI,
> How do I use FlinkKinesisConsumer using the Properties through a proxy ?
> Getting a Connection issue through the proxy.
> Works outside the proxy.
>
> Properties kinesisConsumerConfig = new Properties();
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_REGION,
> region);
>
> if (local) {
>
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_ACCESS_KEY_ID,
> accessKey);
>
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_SECRET_ACCESS_KEY,
> secretKey);
> } else {
>
> kinesisConsumerConfig.setProperty(AWSConfigConstants.AWS_CREDENTIALS_PROVIDER,
> "AUTO");
> }
>
> //only for Consumer
>
> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_MAX,
> "10000");
>
> kinesisConsumerConfig.setProperty(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS,
> "2000");
> FlinkKinesisConsumer<Tuple2<InputMetadata, CameraWithCube>>
> kinesisConsumer = new FlinkKinesisConsumer<>(
> "kinesisTopicRead", new Tuple2KinesisSchema(),
> kinesisConsumerConfig);
> TIA
>