[jira] [Created] (FLINK-23231) The JM's pod was failed when the high-availability.storageDir which using the S3

["Levi Lv (Jira)" <ji...@apache.org>]

Levi Lv created FLINK-23231:
-------------------------------

             Summary: The JM's pod was failed when the high-availability.storageDir which using the S3
                 Key: FLINK-23231
                 URL: https://issues.apache.org/jira/browse/FLINK-23231
             Project: Flink
          Issue Type: Bug
         Environment: [^JM_log.txt]
            Reporter: Levi Lv
         Attachments: JM_log.txt

Hi Team,

I'm using the S3 to save the metadata for JM as flink-conf.yaml below

 
blob.server.port: 6124
high-availability: zookeeper
high-availability.zookeeper.quorum: zetcd:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /flink
*high-availability.storageDir: s3://dsw-dia-test/recovery*
high-availability.jobmanager.port: 50010
high-availability.zookeeper.client.acl: open
state.backend: filesystem
state.checkpoints.dir: s3://dsw-dia-test/checkpoints
state.backend.fs.checkpointdir: s3://dsw-dia-test/checkpoints
s3.path.style.access:true
fs.allowed-fallback-filesystems: s3
s3.endpoint: s3.us-south.cloud-object-storage.appdomain.cloud
s3.access-key: ***************
s3.secret-key:*************
 
I'm already building the new image for Flink 1.13 where it's used for the JM and TM's deployments. 
 
The Docker file is 
 
FROM flink:1.13.1-scala_2.11

#Add the hadoop jar file to folder /opt/flink/lib/
WORKDIR $FLINK_HOME
COPY flink-hadoop-compatibility_2.12-1.13.1.jar flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar $FLINK_HOME/lib/
RUN chown -R flink:flink .;

#Entropy injection for S3 file systems
RUN mkdir $FLINK_HOME/plugins/s3-fs-hadoop \
&&mkdir$FLINK_HOME/plugins/s3-fs-presto

COPY flink-s3-fs-hadoop-1.13.1.jar $FLINK_HOME/plugins/s3-fs-hadoop
COPY flink-s3-fs-presto-1.13.1.jar $FLINK_HOME/plugins/s3-fs-presto
RUN chown -R flink:flink .;
 
Have checked the jars inside TM pod 
 
the plugins for file system
drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-hadoop
drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-presto

flink@63f9c0076cfc:~/plugins/s3-fs-hadoop$ ls -lrt
total 19796
-rw-r--r-- 1 flink flink 20269950 Jul 2 02:25 flink-s3-fs-hadoop-1.13.1.jar

flink@63f9c0076cfc:~/plugins/s3-fs-presto$ ls -lrt
total 32692
-rw-r--r-- 1 flink flink 33474159 May 25 12:20 flink-s3-fs-presto-1.13.1.jar

 
The error for JM as attachments JM_log.txt.
 
If you need any more information, please let me know. 
Thanks In Advance!
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)