You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Xinmin <my...@gmail.com> on 2023/12/01 05:11:08 UTC
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Hello Sachin,
You said that you were able to configure SnowflakeIO with S3 bucket. Can
you please share with me the steps to configure and test it? I would really
appreciate it. Thanks.
Regards,
Xinmin
On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
> Hi Sachin,
>
> Thanks for your information.
>
> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com> wrote:
>
>> I think AWS is supported and I was able to configure snowflake io with S3
>> buckets.
>>
>>
>>
>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> As per the javadoc below, only the GCP bucket is supported currently by
>>> SnowflakeIO connector? Can you please confirm that AWS and Azure are
>>> supported now? Thanks.
>>>
>>>
>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>> ,
>>>
>>
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Posted by Xinmin <my...@gmail.com>.
Hello Sachin,
Thanks for your suggestion, I really appreciate it.
Regards,
Xinmin
On Tue, Dec 5, 2023 at 1:17 AM Sachin Mittal <sj...@gmail.com> wrote:
> Hi,
> As far as I know, for "write" there is no difference between bounded and
> unbounded sources. I have never tried a bounded source.
>
> Thanks
> Sachin
>
>
> On Tue, Dec 5, 2023 at 10:48 AM Xinmin <my...@gmail.com> wrote:
>
>> Hello Sachin,
>>
>> Thanks for your information. The SnowPipe is used for unbounded sources.
>> I am sorry, but there is one last thing. Did you by any chance try with a
>> bounded source? If yes, could you please share some details? Anyway, I
>> really appreciate what you have provided already. :)
>>
>> [image: image.png]
>>
>> Regards,
>> Xinmin
>>
>>
>> On Mon, Dec 4, 2023 at 3:58 AM Sachin Mittal <sj...@gmail.com> wrote:
>>
>>> I think along with the storage integration name you also have to specify
>>> the snowpipe to use.
>>>
>>> SnowflakeIO.<>write()
>>> .withDataSourceConfiguration(dc)
>>> .to(TABLE_NAME)
>>> .withStagingBucketName(stagingBucketName)
>>> .withStorageIntegrationName(storageIntegrationName)
>>> .withDebugMode(StreamingLogLevel.INFO) // change this to
>>> get more information
>>> .withSnowPipe("pipe")
>>>
>>> However looking at your error I think there may be some other issue. It
>>> is mainly related to how you are authenticating for S3 and looks like what
>>> authenticator beam provides does not match with what is required by your
>>> aws account.
>>>
>>> What I can suggest is to just write to S3 using beam i/o.
>>>
>>> Then configure the snowpipe to read from that S3 bucket into a snowflake
>>> which will work independently from your beam application.
>>>
>>> In my case I have almost always just written to S3 and then used
>>> snowpipe to move data from S3 to snowflake and there is a standard way of
>>> doing that.
>>>
>>> Thanks
>>> Sachin
>>>
>>>
>>> On Sun, Dec 3, 2023 at 11:33 AM Xinmin <my...@gmail.com> wrote:
>>>
>>>> Hello Sachin,
>>>>
>>>> I am trying to write data to Snowflake using SnowflakeIO Connector. As
>>>> per https://docs.snowflake.com/en/user-guide/data-load-s3-config,
>>>> there are three options to configure the access to S3 buckets in Snowflake.
>>>> 1) Storage integration which avoids the need to provide access key and
>>>> secret; 2) IAM role; 3) IAM user which needs us to provide the access key
>>>> and secret. The option of IAM role is deprecated by Snowflake.
>>>>
>>>> As per https://beam.apache.org/documentation/io/built-in/snowflake/,
>>>> it mentions the usage of storage integration.
>>>>
>>>> [image: image.png]
>>>>
>>>> In the meantime, it also mentions that we need to provide both keys,
>>>> which should be related to the option of IAM user.
>>>>
>>>> [image: image.png]
>>>>
>>>> I googled the usage of SnowflakeIO connector and S3 bucket and then
>>>> found examples using Python with both keys (access key and secret key).
>>>>
>>>> https://github.com/dennisbachmann/beam-s3-reader/tree/master
>>>>
>>>> https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592
>>>>
>>>> https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14
>>>>
>>>> Since we are not allowed to create an IAM user in AWS due to security
>>>> policy, we would like to go with the option of storage integration, I did
>>>> the test with that and got the error below. Not sure if it was caused by
>>>> the environment or something else.
>>>>
>>>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>>>> java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The
>>>> provided token has expired. (Service: Amazon S3; Status Code: 400; Error
>>>> Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID:
>>>> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=;
>>>> Proxy: null), S3 Extended Request ID:
>>>> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=
>>>>
>>>> I would appreciate it if you could share with me the steps that you
>>>> followed to do the test using the SnowflakeIO connector and S3 bucket.
>>>> Thanks.
>>>>
>>>> Regards,
>>>> Xinmin
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <sj...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> What are you trying to do ?
>>>>> Write to snowflake or read from it ?
>>>>>
>>>>> Thanks
>>>>> Sachin
>>>>>
>>>>>
>>>>> On Fri, Dec 1, 2023 at 10:43 AM Xinmin <my...@gmail.com> wrote:
>>>>>
>>>>>> Hello Sachin,
>>>>>>
>>>>>> You said that you were able to configure SnowflakeIO with S3 bucket.
>>>>>> Can you please share with me the steps to configure and test it? I would
>>>>>> really appreciate it. Thanks.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Xinmin
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sachin,
>>>>>>>
>>>>>>> Thanks for your information.
>>>>>>>
>>>>>>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I think AWS is supported and I was able to configure snowflake io
>>>>>>>> with S3 buckets.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> As per the javadoc below, only the GCP bucket is supported
>>>>>>>>> currently by SnowflakeIO connector? Can you please confirm that AWS and
>>>>>>>>> Azure are supported now? Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>>>>>>> ,
>>>>>>>>>
>>>>>>>>
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
As far as I know, for "write" there is no difference between bounded and
unbounded sources. I have never tried a bounded source.
Thanks
Sachin
On Tue, Dec 5, 2023 at 10:48 AM Xinmin <my...@gmail.com> wrote:
> Hello Sachin,
>
> Thanks for your information. The SnowPipe is used for unbounded sources. I
> am sorry, but there is one last thing. Did you by any chance try with a
> bounded source? If yes, could you please share some details? Anyway, I
> really appreciate what you have provided already. :)
>
> [image: image.png]
>
> Regards,
> Xinmin
>
>
> On Mon, Dec 4, 2023 at 3:58 AM Sachin Mittal <sj...@gmail.com> wrote:
>
>> I think along with the storage integration name you also have to specify
>> the snowpipe to use.
>>
>> SnowflakeIO.<>write()
>> .withDataSourceConfiguration(dc)
>> .to(TABLE_NAME)
>> .withStagingBucketName(stagingBucketName)
>> .withStorageIntegrationName(storageIntegrationName)
>> .withDebugMode(StreamingLogLevel.INFO) // change this to
>> get more information
>> .withSnowPipe("pipe")
>>
>> However looking at your error I think there may be some other issue. It
>> is mainly related to how you are authenticating for S3 and looks like what
>> authenticator beam provides does not match with what is required by your
>> aws account.
>>
>> What I can suggest is to just write to S3 using beam i/o.
>>
>> Then configure the snowpipe to read from that S3 bucket into a snowflake
>> which will work independently from your beam application.
>>
>> In my case I have almost always just written to S3 and then used snowpipe
>> to move data from S3 to snowflake and there is a standard way of doing that.
>>
>> Thanks
>> Sachin
>>
>>
>> On Sun, Dec 3, 2023 at 11:33 AM Xinmin <my...@gmail.com> wrote:
>>
>>> Hello Sachin,
>>>
>>> I am trying to write data to Snowflake using SnowflakeIO Connector. As
>>> per https://docs.snowflake.com/en/user-guide/data-load-s3-config, there
>>> are three options to configure the access to S3 buckets in Snowflake. 1)
>>> Storage integration which avoids the need to provide access key and secret;
>>> 2) IAM role; 3) IAM user which needs us to provide the access key and
>>> secret. The option of IAM role is deprecated by Snowflake.
>>>
>>> As per https://beam.apache.org/documentation/io/built-in/snowflake/, it
>>> mentions the usage of storage integration.
>>>
>>> [image: image.png]
>>>
>>> In the meantime, it also mentions that we need to provide both keys,
>>> which should be related to the option of IAM user.
>>>
>>> [image: image.png]
>>>
>>> I googled the usage of SnowflakeIO connector and S3 bucket and then
>>> found examples using Python with both keys (access key and secret key).
>>>
>>> https://github.com/dennisbachmann/beam-s3-reader/tree/master
>>>
>>> https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592
>>>
>>> https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14
>>>
>>> Since we are not allowed to create an IAM user in AWS due to security
>>> policy, we would like to go with the option of storage integration, I did
>>> the test with that and got the error below. Not sure if it was caused by
>>> the environment or something else.
>>>
>>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>>> java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The
>>> provided token has expired. (Service: Amazon S3; Status Code: 400; Error
>>> Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID:
>>> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=;
>>> Proxy: null), S3 Extended Request ID:
>>> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=
>>>
>>> I would appreciate it if you could share with me the steps that you
>>> followed to do the test using the SnowflakeIO connector and S3 bucket.
>>> Thanks.
>>>
>>> Regards,
>>> Xinmin
>>>
>>>
>>>
>>>
>>> On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <sj...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> What are you trying to do ?
>>>> Write to snowflake or read from it ?
>>>>
>>>> Thanks
>>>> Sachin
>>>>
>>>>
>>>> On Fri, Dec 1, 2023 at 10:43 AM Xinmin <my...@gmail.com> wrote:
>>>>
>>>>> Hello Sachin,
>>>>>
>>>>> You said that you were able to configure SnowflakeIO with S3 bucket.
>>>>> Can you please share with me the steps to configure and test it? I would
>>>>> really appreciate it. Thanks.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Xinmin
>>>>>
>>>>>
>>>>> On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
>>>>>
>>>>>> Hi Sachin,
>>>>>>
>>>>>> Thanks for your information.
>>>>>>
>>>>>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I think AWS is supported and I was able to configure snowflake io
>>>>>>> with S3 buckets.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> As per the javadoc below, only the GCP bucket is supported
>>>>>>>> currently by SnowflakeIO connector? Can you please confirm that AWS and
>>>>>>>> Azure are supported now? Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>>>>>> ,
>>>>>>>>
>>>>>>>
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Posted by Xinmin <my...@gmail.com>.
Hello Sachin,
Thanks for your information. The SnowPipe is used for unbounded sources. I
am sorry, but there is one last thing. Did you by any chance try with a
bounded source? If yes, could you please share some details? Anyway, I
really appreciate what you have provided already. :)
[image: image.png]
Regards,
Xinmin
On Mon, Dec 4, 2023 at 3:58 AM Sachin Mittal <sj...@gmail.com> wrote:
> I think along with the storage integration name you also have to specify
> the snowpipe to use.
>
> SnowflakeIO.<>write()
> .withDataSourceConfiguration(dc)
> .to(TABLE_NAME)
> .withStagingBucketName(stagingBucketName)
> .withStorageIntegrationName(storageIntegrationName)
> .withDebugMode(StreamingLogLevel.INFO) // change this to
> get more information
> .withSnowPipe("pipe")
>
> However looking at your error I think there may be some other issue. It is
> mainly related to how you are authenticating for S3 and looks like what
> authenticator beam provides does not match with what is required by your
> aws account.
>
> What I can suggest is to just write to S3 using beam i/o.
>
> Then configure the snowpipe to read from that S3 bucket into a snowflake
> which will work independently from your beam application.
>
> In my case I have almost always just written to S3 and then used snowpipe
> to move data from S3 to snowflake and there is a standard way of doing that.
>
> Thanks
> Sachin
>
>
> On Sun, Dec 3, 2023 at 11:33 AM Xinmin <my...@gmail.com> wrote:
>
>> Hello Sachin,
>>
>> I am trying to write data to Snowflake using SnowflakeIO Connector. As
>> per https://docs.snowflake.com/en/user-guide/data-load-s3-config, there
>> are three options to configure the access to S3 buckets in Snowflake. 1)
>> Storage integration which avoids the need to provide access key and secret;
>> 2) IAM role; 3) IAM user which needs us to provide the access key and
>> secret. The option of IAM role is deprecated by Snowflake.
>>
>> As per https://beam.apache.org/documentation/io/built-in/snowflake/, it
>> mentions the usage of storage integration.
>>
>> [image: image.png]
>>
>> In the meantime, it also mentions that we need to provide both keys,
>> which should be related to the option of IAM user.
>>
>> [image: image.png]
>>
>> I googled the usage of SnowflakeIO connector and S3 bucket and then found
>> examples using Python with both keys (access key and secret key).
>>
>> https://github.com/dennisbachmann/beam-s3-reader/tree/master
>>
>> https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592
>>
>> https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14
>>
>> Since we are not allowed to create an IAM user in AWS due to security
>> policy, we would like to go with the option of storage integration, I did
>> the test with that and got the error below. Not sure if it was caused by
>> the environment or something else.
>>
>> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
>> java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The
>> provided token has expired. (Service: Amazon S3; Status Code: 400; Error
>> Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID:
>> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=;
>> Proxy: null), S3 Extended Request ID:
>> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=
>>
>> I would appreciate it if you could share with me the steps that you
>> followed to do the test using the SnowflakeIO connector and S3 bucket.
>> Thanks.
>>
>> Regards,
>> Xinmin
>>
>>
>>
>>
>> On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <sj...@gmail.com> wrote:
>>
>>> Hi,
>>> What are you trying to do ?
>>> Write to snowflake or read from it ?
>>>
>>> Thanks
>>> Sachin
>>>
>>>
>>> On Fri, Dec 1, 2023 at 10:43 AM Xinmin <my...@gmail.com> wrote:
>>>
>>>> Hello Sachin,
>>>>
>>>> You said that you were able to configure SnowflakeIO with S3 bucket.
>>>> Can you please share with me the steps to configure and test it? I would
>>>> really appreciate it. Thanks.
>>>>
>>>>
>>>> Regards,
>>>> Xinmin
>>>>
>>>>
>>>> On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
>>>>
>>>>> Hi Sachin,
>>>>>
>>>>> Thanks for your information.
>>>>>
>>>>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I think AWS is supported and I was able to configure snowflake io
>>>>>> with S3 buckets.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> As per the javadoc below, only the GCP bucket is supported currently
>>>>>>> by SnowflakeIO connector? Can you please confirm that AWS and Azure are
>>>>>>> supported now? Thanks.
>>>>>>>
>>>>>>>
>>>>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>>>>> ,
>>>>>>>
>>>>>>
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Posted by Sachin Mittal <sj...@gmail.com>.
I think along with the storage integration name you also have to specify
the snowpipe to use.
SnowflakeIO.<>write()
.withDataSourceConfiguration(dc)
.to(TABLE_NAME)
.withStagingBucketName(stagingBucketName)
.withStorageIntegrationName(storageIntegrationName)
.withDebugMode(StreamingLogLevel.INFO) // change this to
get more information
.withSnowPipe("pipe")
However looking at your error I think there may be some other issue. It is
mainly related to how you are authenticating for S3 and looks like what
authenticator beam provides does not match with what is required by your
aws account.
What I can suggest is to just write to S3 using beam i/o.
Then configure the snowpipe to read from that S3 bucket into a snowflake
which will work independently from your beam application.
In my case I have almost always just written to S3 and then used snowpipe
to move data from S3 to snowflake and there is a standard way of doing that.
Thanks
Sachin
On Sun, Dec 3, 2023 at 11:33 AM Xinmin <my...@gmail.com> wrote:
> Hello Sachin,
>
> I am trying to write data to Snowflake using SnowflakeIO Connector. As per
> https://docs.snowflake.com/en/user-guide/data-load-s3-config, there are
> three options to configure the access to S3 buckets in Snowflake. 1)
> Storage integration which avoids the need to provide access key and secret;
> 2) IAM role; 3) IAM user which needs us to provide the access key and
> secret. The option of IAM role is deprecated by Snowflake.
>
> As per https://beam.apache.org/documentation/io/built-in/snowflake/, it
> mentions the usage of storage integration.
>
> [image: image.png]
>
> In the meantime, it also mentions that we need to provide both keys, which
> should be related to the option of IAM user.
>
> [image: image.png]
>
> I googled the usage of SnowflakeIO connector and S3 bucket and then found
> examples using Python with both keys (access key and secret key).
>
> https://github.com/dennisbachmann/beam-s3-reader/tree/master
>
> https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592
>
> https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14
>
> Since we are not allowed to create an IAM user in AWS due to security
> policy, we would like to go with the option of storage integration, I did
> the test with that and got the error below. Not sure if it was caused by
> the environment or something else.
>
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The
> provided token has expired. (Service: Amazon S3; Status Code: 400; Error
> Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID:
> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=;
> Proxy: null), S3 Extended Request ID:
> foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=
>
> I would appreciate it if you could share with me the steps that you
> followed to do the test using the SnowflakeIO connector and S3 bucket.
> Thanks.
>
> Regards,
> Xinmin
>
>
>
>
> On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <sj...@gmail.com> wrote:
>
>> Hi,
>> What are you trying to do ?
>> Write to snowflake or read from it ?
>>
>> Thanks
>> Sachin
>>
>>
>> On Fri, Dec 1, 2023 at 10:43 AM Xinmin <my...@gmail.com> wrote:
>>
>>> Hello Sachin,
>>>
>>> You said that you were able to configure SnowflakeIO with S3 bucket. Can
>>> you please share with me the steps to configure and test it? I would really
>>> appreciate it. Thanks.
>>>
>>>
>>> Regards,
>>> Xinmin
>>>
>>>
>>> On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
>>>
>>>> Hi Sachin,
>>>>
>>>> Thanks for your information.
>>>>
>>>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com>
>>>> wrote:
>>>>
>>>>> I think AWS is supported and I was able to configure snowflake io with
>>>>> S3 buckets.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> As per the javadoc below, only the GCP bucket is supported currently
>>>>>> by SnowflakeIO connector? Can you please confirm that AWS and Azure are
>>>>>> supported now? Thanks.
>>>>>>
>>>>>>
>>>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>>>> ,
>>>>>>
>>>>>
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Posted by Xinmin <my...@gmail.com>.
Hello Sachin,
I am trying to write data to Snowflake using SnowflakeIO Connector. As per
https://docs.snowflake.com/en/user-guide/data-load-s3-config, there are
three options to configure the access to S3 buckets in Snowflake. 1)
Storage integration which avoids the need to provide access key and secret;
2) IAM role; 3) IAM user which needs us to provide the access key and
secret. The option of IAM role is deprecated by Snowflake.
As per https://beam.apache.org/documentation/io/built-in/snowflake/, it
mentions the usage of storage integration.
[image: image.png]
In the meantime, it also mentions that we need to provide both keys, which
should be related to the option of IAM user.
[image: image.png]
I googled the usage of SnowflakeIO connector and S3 bucket and then found
examples using Python with both keys (access key and secret key).
https://github.com/dennisbachmann/beam-s3-reader/tree/master
https://python.plainenglish.io/using-apache-beam-and-aws-s3-storage-i-o-transforms-in-python-6cabe2a8d592
https://medium.com/@varunkrishna97/apache-beam-reading-from-writing-to-s3-in-python-sdk-6f2ff6419a14
Since we are not allowed to create an IAM user in AWS due to security
policy, we would like to go with the option of storage integration, I did
the test with that and got the error below. Not sure if it was caused by
the environment or something else.
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The
provided token has expired. (Service: Amazon S3; Status Code: 400; Error
Code: ExpiredToken; Request ID: 1RCH191011YK9NEJ; S3 Extended Request ID:
foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=;
Proxy: null), S3 Extended Request ID:
foh3wogVMGT7+kQ+5sbnjPf7WA+9Vbz6zfwmnz8GDChNgkPhf0Wzk9MRBBNl9KqsjHDDB/Jt1r4=
I would appreciate it if you could share with me the steps that you
followed to do the test using the SnowflakeIO connector and S3 bucket.
Thanks.
Regards,
Xinmin
On Sat, Dec 2, 2023 at 11:46 AM Sachin Mittal <sj...@gmail.com> wrote:
> Hi,
> What are you trying to do ?
> Write to snowflake or read from it ?
>
> Thanks
> Sachin
>
>
> On Fri, Dec 1, 2023 at 10:43 AM Xinmin <my...@gmail.com> wrote:
>
>> Hello Sachin,
>>
>> You said that you were able to configure SnowflakeIO with S3 bucket. Can
>> you please share with me the steps to configure and test it? I would really
>> appreciate it. Thanks.
>>
>>
>> Regards,
>> Xinmin
>>
>>
>> On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
>>
>>> Hi Sachin,
>>>
>>> Thanks for your information.
>>>
>>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com>
>>> wrote:
>>>
>>>> I think AWS is supported and I was able to configure snowflake io with
>>>> S3 buckets.
>>>>
>>>>
>>>>
>>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> As per the javadoc below, only the GCP bucket is supported currently
>>>>> by SnowflakeIO connector? Can you please confirm that AWS and Azure are
>>>>> supported now? Thanks.
>>>>>
>>>>>
>>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>>> ,
>>>>>
>>>>
Re: [Question] Does SnowflakeIO connector support for AWS and Azure?
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
What are you trying to do ?
Write to snowflake or read from it ?
Thanks
Sachin
On Fri, Dec 1, 2023 at 10:43 AM Xinmin <my...@gmail.com> wrote:
> Hello Sachin,
>
> You said that you were able to configure SnowflakeIO with S3 bucket. Can
> you please share with me the steps to configure and test it? I would really
> appreciate it. Thanks.
>
>
> Regards,
> Xinmin
>
>
> On Thu, Oct 26, 2023 at 9:42 AM mybeam <my...@gmail.com> wrote:
>
>> Hi Sachin,
>>
>> Thanks for your information.
>>
>> On Tue, Oct 24, 2023 at 11:02 PM Sachin Mittal <sj...@gmail.com>
>> wrote:
>>
>>> I think AWS is supported and I was able to configure snowflake io with
>>> S3 buckets.
>>>
>>>
>>>
>>> On Wed, 25 Oct 2023 at 9:05 AM, mybeam <my...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> As per the javadoc below, only the GCP bucket is supported currently by
>>>> SnowflakeIO connector? Can you please confirm that AWS and Azure are
>>>> supported now? Thanks.
>>>>
>>>>
>>>> https://beam.apache.org/releases/javadoc/2.50.0/index.html?org/apache/beam/sdk/io/snowflake/SnowflakeIO.html
>>>> ,
>>>>
>>>