You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@paimon.apache.org by haishui <ha...@126.com> on 2023/08/17 07:06:00 UTC

Questions about using s3

Hi all,


I'm using paimon with s3  (minio) 



If you have already configured s3 access through Flink (Via Flink FileSystem), here you can skip the following configuration.[1]


Is this meant that `s3.access-key` and `s3.secret-key` can be set in flink-conf.yml? I have tried, but make no sense.


[1] https://paimon.apache.org/docs/master/filesystems/s3/

Re:Re: Questions about using s3

Posted by haishui <ha...@126.com>.
Hi,


Thanks for your replay! I find the reason is my secret-key contains #.


When the secret-key contains #, the sql-client throw exception:


UnsupportedSchemeException: Could not find a file io implementation for scheme 's3' in the classpath. FlinkFileIOLoader also cannot access this path. Hadoop FileSystem also cannot access this path 's3://warehouse/paimon'.

Suppressed: java.io.IOException: One or more required options are missing.

Missing required options are:

s3.access-key

s3.secret-key













在 2023-08-18 12:06:02，"yu zelin" <yu...@gmail.com> 写道：

Hi, 


Did you configure Flink under this guide [1]? Maybe you forgot to configure plugins and s3.path.style.access:true




Best,
Yu Zelin


[1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/




2023年8月17日 15:06，haishui <ha...@126.com> 写道：


Hi all,


I'm using paimon with s3  (minio) 



If you have already configured s3 access through Flink (Via Flink FileSystem), here you can skip the following configuration.[1]


Is this meant that `s3.access-key` and `s3.secret-key` can be set in flink-conf.yml? I have tried, but make no sense.


[1] https://paimon.apache.org/docs/master/filesystems/s3/

Re: Questions about using s3

Posted by yu zelin <yu...@gmail.com>.
Hi, 

Did you configure Flink under this guide [1]? Maybe you forgot to configure plugins and s3.path.style.access: true


Best,
Yu Zelin

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/


> 2023年8月17日 15:06，haishui <ha...@126.com> 写道：
> 
> Hi all,
> 
> I'm using paimon with s3  (minio) 
> 
> If you have already configured s3 access through Flink <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/filesystems/s3/> (Via Flink FileSystem), here you can skip the following configuration.[1]
> 
> Is this meant that `s3.access-key` and `s3.secret-key` can be set in flink-conf.yml? I have tried, but make no sense.
> 
> [1] https://paimon.apache.org/docs/master/filesystems/s3/