You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by David Magalhães <sp...@gmail.com> on 2020/02/20 12:16:42 UTC
AWS Client Builder with default credentials
I'm using
org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
to create a S3 client to copy objects and delete object inside
a TwoPhaseCommitSinkFunction.
Shouldn't be another way to set up configurations without put them
hardcoded ? Something like core-site.xml or flink-conf.yaml ?
Right now I need to have them hardcoded like this.
AmazonS3ClientBuilder.standard
.withPathStyleAccessEnabled(true)
.withEndpointConfiguration(
new EndpointConfiguration("http://minio:9000", "us-east-1")
)
.withCredentials(
new AWSStaticCredentialsProvider(new BasicAWSCredentials("minio",
"minio123"))
)
.build
Thanks
Re: AWS Client Builder with default credentials
Posted by Suneel Marthi <sm...@apache.org>.
Not sure if this helps - this is how I invoke a Sagemaker endpoint model
from a flink pipeline.
See
https://github.com/smarthi/NMT-Sagemaker-Inference/blob/master/src/main/java/de/dws/berlin/util/AwsUtil.java
On Mon, Feb 24, 2020 at 10:08 AM David Magalhães <sp...@gmail.com>
wrote:
> Hi Robert, thanks for your reply.
>
> GlobalConfiguration.loadConfiguration was useful to check if a
> flink-conf.yml file was on resources, for the integration tests that I'm
> doing. On the cluster I will use the default configurations.
>
> On Fri, Feb 21, 2020 at 10:58 AM Robert Metzger <rm...@apache.org>
> wrote:
>
>> There are multiple ways of passing configuration parameters to your user
>> defined code in Flink
>>
>> a) use getRuntimeContext().getUserCodeClassLoader().getResource() to
>> load a config file from your user code jar or the classpath.
>> b) use
>> getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
>> access a configuration object serialized from the main method.
>> you can pass a custom object to the job parameters, or use Flink's
>> "Configuration" object in your main method:
>>
>> final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
>>
>> Configuration config = new Configuration();
>> config.setString("foo", "bar");
>> env.getConfig().setGlobalJobParameters(config);
>>
>> c) Load the flink-conf.yaml:
>>
>> Configuration conf = GlobalConfiguration.loadConfiguration();
>>
>> I'm not 100% sure if this approach works, as it is not intended to be
>> used in user code (I believe).
>>
>>
>> Let me know if this helps!
>>
>> Best,
>> Robert
>>
>> On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> First things first, we do not intend for users to use anything in the S3
>>> filesystem modules except the filesystems itself,
>>> meaning that you're somewhat treading on unsupported ground here.
>>>
>>> Nevertheless, the S3 modules contain a large variety of AWS-provided
>>> CerentialsProvider implementations,
>>> that can derive credentials from environment variables, system
>>> properties, files on the classpath and many more.
>>>
>>> Ultimately though, you're kind of asking us how to use AWS APIs, for
>>> which I would direct you to the AWS documentation.
>>>
>>> On 20/02/2020 13:16, David Magalhães wrote:
>>>
>>> I'm using
>>> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
>>> to create a S3 client to copy objects and delete object inside
>>> a TwoPhaseCommitSinkFunction.
>>>
>>> Shouldn't be another way to set up configurations without put them
>>> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>>>
>>> Right now I need to have them hardcoded like this.
>>>
>>> AmazonS3ClientBuilder.standard
>>> .withPathStyleAccessEnabled(true)
>>> .withEndpointConfiguration(
>>> new EndpointConfiguration("http://minio:9000", "us-east-1")
>>> )
>>> .withCredentials(
>>> new AWSStaticCredentialsProvider(new
>>> BasicAWSCredentials("minio", "minio123"))
>>> )
>>> .build
>>>
>>> Thanks
>>>
>>>
>>>
Re: AWS Client Builder with default credentials
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
check this.
https://github.com/kali786516/FlinkStreamAndSql/blob/b8bcbadaa3cb6bfdae891f10ad1205e256adbc1e/src/main/scala/com/aws/examples/dynamodb/dynStreams/FlinkDynamoDBStreams.scala#L42
https://github.com/kali786516/FlinkStreamAndSql/blob/b8bcbadaa3cb6bfdae891f10ad1205e256adbc1e/src/main/scala/com/aws/examples/kinesis/consumer/transactionScalaWorkingExample/TransactionScalaConsumer.scala#L41
On Mon, Feb 24, 2020 at 9:08 AM David Magalhães <sp...@gmail.com>
wrote:
> Hi Robert, thanks for your reply.
>
> GlobalConfiguration.loadConfiguration was useful to check if a
> flink-conf.yml file was on resources, for the integration tests that I'm
> doing. On the cluster I will use the default configurations.
>
> On Fri, Feb 21, 2020 at 10:58 AM Robert Metzger <rm...@apache.org>
> wrote:
>
>> There are multiple ways of passing configuration parameters to your user
>> defined code in Flink
>>
>> a) use getRuntimeContext().getUserCodeClassLoader().getResource() to
>> load a config file from your user code jar or the classpath.
>> b) use
>> getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
>> access a configuration object serialized from the main method.
>> you can pass a custom object to the job parameters, or use Flink's
>> "Configuration" object in your main method:
>>
>> final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
>>
>> Configuration config = new Configuration();
>> config.setString("foo", "bar");
>> env.getConfig().setGlobalJobParameters(config);
>>
>> c) Load the flink-conf.yaml:
>>
>> Configuration conf = GlobalConfiguration.loadConfiguration();
>>
>> I'm not 100% sure if this approach works, as it is not intended to be
>> used in user code (I believe).
>>
>>
>> Let me know if this helps!
>>
>> Best,
>> Robert
>>
>> On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> First things first, we do not intend for users to use anything in the S3
>>> filesystem modules except the filesystems itself,
>>> meaning that you're somewhat treading on unsupported ground here.
>>>
>>> Nevertheless, the S3 modules contain a large variety of AWS-provided
>>> CerentialsProvider implementations,
>>> that can derive credentials from environment variables, system
>>> properties, files on the classpath and many more.
>>>
>>> Ultimately though, you're kind of asking us how to use AWS APIs, for
>>> which I would direct you to the AWS documentation.
>>>
>>> On 20/02/2020 13:16, David Magalhães wrote:
>>>
>>> I'm using
>>> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
>>> to create a S3 client to copy objects and delete object inside
>>> a TwoPhaseCommitSinkFunction.
>>>
>>> Shouldn't be another way to set up configurations without put them
>>> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>>>
>>> Right now I need to have them hardcoded like this.
>>>
>>> AmazonS3ClientBuilder.standard
>>> .withPathStyleAccessEnabled(true)
>>> .withEndpointConfiguration(
>>> new EndpointConfiguration("http://minio:9000", "us-east-1")
>>> )
>>> .withCredentials(
>>> new AWSStaticCredentialsProvider(new
>>> BasicAWSCredentials("minio", "minio123"))
>>> )
>>> .build
>>>
>>> Thanks
>>>
>>>
>>>
--
Thanks & Regards
Sri Tummala
Re: AWS Client Builder with default credentials
Posted by David Magalhães <sp...@gmail.com>.
Hi Robert, thanks for your reply.
GlobalConfiguration.loadConfiguration was useful to check if a
flink-conf.yml file was on resources, for the integration tests that I'm
doing. On the cluster I will use the default configurations.
On Fri, Feb 21, 2020 at 10:58 AM Robert Metzger <rm...@apache.org> wrote:
> There are multiple ways of passing configuration parameters to your user
> defined code in Flink
>
> a) use getRuntimeContext().getUserCodeClassLoader().getResource() to
> load a config file from your user code jar or the classpath.
> b) use getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
> access a configuration object serialized from the main method.
> you can pass a custom object to the job parameters, or use Flink's
> "Configuration" object in your main method:
>
> final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
>
> Configuration config = new Configuration();
> config.setString("foo", "bar");
> env.getConfig().setGlobalJobParameters(config);
>
> c) Load the flink-conf.yaml:
>
> Configuration conf = GlobalConfiguration.loadConfiguration();
>
> I'm not 100% sure if this approach works, as it is not intended to be used
> in user code (I believe).
>
>
> Let me know if this helps!
>
> Best,
> Robert
>
> On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> First things first, we do not intend for users to use anything in the S3
>> filesystem modules except the filesystems itself,
>> meaning that you're somewhat treading on unsupported ground here.
>>
>> Nevertheless, the S3 modules contain a large variety of AWS-provided
>> CerentialsProvider implementations,
>> that can derive credentials from environment variables, system
>> properties, files on the classpath and many more.
>>
>> Ultimately though, you're kind of asking us how to use AWS APIs, for
>> which I would direct you to the AWS documentation.
>>
>> On 20/02/2020 13:16, David Magalhães wrote:
>>
>> I'm using
>> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
>> to create a S3 client to copy objects and delete object inside
>> a TwoPhaseCommitSinkFunction.
>>
>> Shouldn't be another way to set up configurations without put them
>> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>>
>> Right now I need to have them hardcoded like this.
>>
>> AmazonS3ClientBuilder.standard
>> .withPathStyleAccessEnabled(true)
>> .withEndpointConfiguration(
>> new EndpointConfiguration("http://minio:9000", "us-east-1")
>> )
>> .withCredentials(
>> new AWSStaticCredentialsProvider(new BasicAWSCredentials("minio",
>> "minio123"))
>> )
>> .build
>>
>> Thanks
>>
>>
>>
Re: AWS Client Builder with default credentials
Posted by Robert Metzger <rm...@apache.org>.
There are multiple ways of passing configuration parameters to your user
defined code in Flink
a) use getRuntimeContext().getUserCodeClassLoader().getResource() to load
a config file from your user code jar or the classpath.
b) use getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
access a configuration object serialized from the main method.
you can pass a custom object to the job parameters, or use Flink's
"Configuration" object in your main method:
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
Configuration config = new Configuration();
config.setString("foo", "bar");
env.getConfig().setGlobalJobParameters(config);
c) Load the flink-conf.yaml:
Configuration conf = GlobalConfiguration.loadConfiguration();
I'm not 100% sure if this approach works, as it is not intended to be used
in user code (I believe).
Let me know if this helps!
Best,
Robert
On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <ch...@apache.org> wrote:
> First things first, we do not intend for users to use anything in the S3
> filesystem modules except the filesystems itself,
> meaning that you're somewhat treading on unsupported ground here.
>
> Nevertheless, the S3 modules contain a large variety of AWS-provided
> CerentialsProvider implementations,
> that can derive credentials from environment variables, system properties,
> files on the classpath and many more.
>
> Ultimately though, you're kind of asking us how to use AWS APIs, for which
> I would direct you to the AWS documentation.
>
> On 20/02/2020 13:16, David Magalhães wrote:
>
> I'm using
> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
> to create a S3 client to copy objects and delete object inside
> a TwoPhaseCommitSinkFunction.
>
> Shouldn't be another way to set up configurations without put them
> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>
> Right now I need to have them hardcoded like this.
>
> AmazonS3ClientBuilder.standard
> .withPathStyleAccessEnabled(true)
> .withEndpointConfiguration(
> new EndpointConfiguration("http://minio:9000", "us-east-1")
> )
> .withCredentials(
> new AWSStaticCredentialsProvider(new BasicAWSCredentials("minio",
> "minio123"))
> )
> .build
>
> Thanks
>
>
>
Re: AWS Client Builder with default credentials
Posted by Chesnay Schepler <ch...@apache.org>.
First things first, we do not intend for users to use anything in the S3
filesystem modules except the filesystems itself,
meaning that you're somewhat treading on unsupported ground here.
Nevertheless, the S3 modules contain a large variety of AWS-provided
CerentialsProvider implementations,
that can derive credentials from environment variables, system
properties, files on the classpath and many more.
Ultimately though, you're kind of asking us how to use AWS APIs, for
which I would direct you to the AWS documentation.
On 20/02/2020 13:16, David Magalhães wrote:
> I'm using
> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
> to create a S3 client to copy objects and delete object inside
> a TwoPhaseCommitSinkFunction.
>
> Shouldn't be another way to set up configurations without put them
> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
>
> Right now I need to have them hardcoded like this.
>
> AmazonS3ClientBuilder.standard
> .withPathStyleAccessEnabled(true)
> .withEndpointConfiguration(
> new EndpointConfiguration("http://minio:9000", "us-east-1")
> )
> .withCredentials(
> new AWSStaticCredentialsProvider(new
> BasicAWSCredentials("minio", "minio123"))
> )
> .build
>
> Thanks