You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sriram Ganesh <sr...@gmail.com> on 2023/04/20 14:56:24 UTC

Facing issue when using S3 in Flink 1.17

Hi Team,

I am using S3 as FileSystem to write data from Flink. I am getting the
below error in Flink 1.17. The same code works in Flink 1.16. Could someone
please help?

Caused by: java.lang.IllegalStateException: Delegation token provider with
service name {} has multiple implementations [s3]

               │
│     at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
~[flink-dist-1.17.0.jar:1.17.0]

            │
│     at
org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.lambda$loadProviders$0(DefaultDelegationTokenManager.java:133)
~[flink-dist-1.17.0.jar:1.17.0]
                          │
│     at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]


                 │
│     at
org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.loadProviders(DefaultDelegationTokenManager.java:156)
~[flink-dist-1.17.0.jar:1.17.0]
                                   │
│     at
org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.<init>(DefaultDelegationTokenManager.java:111)
~[flink-dist-1.17.0.jar:1.17.0]
                                          │
│     at
org.apache.flink.runtime.security.token.DefaultDelegationTokenManagerFactory.create(DefaultDelegationTokenManagerFactory.java:50)
~[flink-dist-1.17.0.jar:1.17.0]
                             │
│     at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:392)
~[flink-dist-1.17.0.jar:1.17.0]
                                                          │
│     at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:282)
~[flink-dist-1.17.0.jar:1.17.0]
                                                                  │
│     at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:232)
~[flink-dist-1.17.0.jar:1.17.0]
                                                       │
│     at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
~[flink-dist-1.17.0.jar:1.17.0]
                                                        │
│     at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:229)
~[flink-dist-1.17.0.jar:1.17.0]
                                                                │
│     ... 2 more

Thanks,

-- 
*Sriram G*

Re: Facing issue when using S3 in Flink 1.17

Posted by Sriram Ganesh <sr...@gmail.com>.
Thanks a lot, Gabor. Do you know by any chance when this fix will be
released? Thanks in advance.

On Thu, Apr 20, 2023 at 9:06 PM Gabor Somogyi <ga...@gmail.com>
wrote:

> Hi Sriram,
>
> This has been fixed in https://issues.apache.org/jira/browse/FLINK-31839
>
> G
>
>
> On Thu, Apr 20, 2023 at 4:57 PM Sriram Ganesh <sr...@gmail.com> wrote:
>
>> Hi Team,
>>
>> I am using S3 as FileSystem to write data from Flink. I am getting the
>> below error in Flink 1.17. The same code works in Flink 1.16. Could someone
>> please help?
>>
>> Caused by: java.lang.IllegalStateException: Delegation token provider
>> with service name {} has multiple implementations [s3]
>>
>>                      │
>> │     at
>> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>
>>             │
>> │     at
>> org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.lambda$loadProviders$0(DefaultDelegationTokenManager.java:133)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                           │
>> │     at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
>>
>>
>>                    │
>> │     at
>> org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.loadProviders(DefaultDelegationTokenManager.java:156)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                    │
>> │     at
>> org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.<init>(DefaultDelegationTokenManager.java:111)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                           │
>> │     at
>> org.apache.flink.runtime.security.token.DefaultDelegationTokenManagerFactory.create(DefaultDelegationTokenManagerFactory.java:50)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                              │
>> │     at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:392)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                                           │
>> │     at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:282)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                                                   │
>> │     at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:232)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                                        │
>> │     at
>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                                         │
>> │     at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:229)
>> ~[flink-dist-1.17.0.jar:1.17.0]
>>                                                                 │
>> │     ... 2 more
>>
>> Thanks,
>>
>> --
>> *Sriram G*
>>
>>

-- 
*Sriram G*
*Tech*

Re: Facing issue when using S3 in Flink 1.17

Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Sriram,

This has been fixed in https://issues.apache.org/jira/browse/FLINK-31839

G


On Thu, Apr 20, 2023 at 4:57 PM Sriram Ganesh <sr...@gmail.com> wrote:

> Hi Team,
>
> I am using S3 as FileSystem to write data from Flink. I am getting the
> below error in Flink 1.17. The same code works in Flink 1.16. Could someone
> please help?
>
> Caused by: java.lang.IllegalStateException: Delegation token provider with
> service name {} has multiple implementations [s3]
>
>                │
> │     at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> ~[flink-dist-1.17.0.jar:1.17.0]
>
>             │
> │     at
> org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.lambda$loadProviders$0(DefaultDelegationTokenManager.java:133)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                           │
> │     at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
>
>
>                  │
> │     at
> org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.loadProviders(DefaultDelegationTokenManager.java:156)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                    │
> │     at
> org.apache.flink.runtime.security.token.DefaultDelegationTokenManager.<init>(DefaultDelegationTokenManager.java:111)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                           │
> │     at
> org.apache.flink.runtime.security.token.DefaultDelegationTokenManagerFactory.create(DefaultDelegationTokenManagerFactory.java:50)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                              │
> │     at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:392)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                                           │
> │     at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:282)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                                                   │
> │     at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:232)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                                        │
> │     at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                                         │
> │     at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:229)
> ~[flink-dist-1.17.0.jar:1.17.0]
>                                                                 │
> │     ... 2 more
>
> Thanks,
>
> --
> *Sriram G*
>
>