You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Levi Lv (Jira)" <ji...@apache.org> on 2021/07/05 02:33:00 UTC
[jira] [Updated] (FLINK-23231) The JM's pod was failed when the
high-availability.storageDir which using the S3
[ https://issues.apache.org/jira/browse/FLINK-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Levi Lv updated FLINK-23231:
----------------------------
Attachment: bucket_info.png
> The JM's pod was failed when the high-availability.storageDir which using the S3
> --------------------------------------------------------------------------------
>
> Key: FLINK-23231
> URL: https://issues.apache.org/jira/browse/FLINK-23231
> Project: Flink
> Issue Type: Bug
> Environment: [^JM_log.txt]
> Reporter: Levi Lv
> Priority: Major
> Attachments: JM_log.txt, bucket_info.png
>
>
> Hi Team,
> I'm using the S3 to save the metadata for JM as flink-conf.yaml below
>
> blob.server.port: 6124
> high-availability: zookeeper
> high-availability.zookeeper.quorum: zetcd:2181
> high-availability.zookeeper.path.root: /flink
> high-availability.cluster-id: /flink
> *high-availability.storageDir: s3://dsw-dia-test/recovery*
> high-availability.jobmanager.port: 50010
> high-availability.zookeeper.client.acl: open
> state.backend: filesystem
> state.checkpoints.dir: s3://dsw-dia-test/checkpoints
> state.backend.fs.checkpointdir: s3://dsw-dia-test/checkpoints
> s3.path.style.access:true
> fs.allowed-fallback-filesystems: s3
> s3.endpoint: s3.us-south.cloud-object-storage.appdomain.cloud
> s3.access-key: ***************
> s3.secret-key:*************
>
> I'm already building the new image for Flink 1.13 where it's used for the JM and TM's deployments.
>
> The Docker file is
>
> FROM flink:1.13.1-scala_2.11
> #Add the hadoop jar file to folder /opt/flink/lib/
> WORKDIR $FLINK_HOME
> COPY flink-hadoop-compatibility_2.12-1.13.1.jar flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar $FLINK_HOME/lib/
> RUN chown -R flink:flink .;
> #Entropy injection for S3 file systems
> RUN mkdir $FLINK_HOME/plugins/s3-fs-hadoop \
> &&mkdir$FLINK_HOME/plugins/s3-fs-presto
> COPY flink-s3-fs-hadoop-1.13.1.jar $FLINK_HOME/plugins/s3-fs-hadoop
> COPY flink-s3-fs-presto-1.13.1.jar $FLINK_HOME/plugins/s3-fs-presto
> RUN chown -R flink:flink .;
>
> Have checked the jars inside TM pod
>
> the plugins for file system
> drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-hadoop
> drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-presto
> flink@63f9c0076cfc:~/plugins/s3-fs-hadoop$ ls -lrt
> total 19796
> -rw-r--r-- 1 flink flink 20269950 Jul 2 02:25 flink-s3-fs-hadoop-1.13.1.jar
> flink@63f9c0076cfc:~/plugins/s3-fs-presto$ ls -lrt
> total 32692
> -rw-r--r-- 1 flink flink 33474159 May 25 12:20 flink-s3-fs-presto-1.13.1.jar
>
> The error for JM as attachments JM_log.txt.
>
> If you need any more information, please let me know.
> Thanks In Advance!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)