You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Levi Lv (Jira)" <ji...@apache.org> on 2021/07/05 02:40:00 UTC
[jira] [Commented] (FLINK-23231) The JM's pod was failed when the
high-availability.storageDir which using the S3
[ https://issues.apache.org/jira/browse/FLINK-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17374455#comment-17374455 ]
Levi Lv commented on FLINK-23231:
---------------------------------
BTW, I'm have checked the bucket did not have any file under recovery folder.
s3://dsw-dia-test/recovery !bucket_info.png!
> The JM's pod was failed when the high-availability.storageDir which using the S3
> --------------------------------------------------------------------------------
>
> Key: FLINK-23231
> URL: https://issues.apache.org/jira/browse/FLINK-23231
> Project: Flink
> Issue Type: Bug
> Environment: [^JM_log.txt]
> Reporter: Levi Lv
> Priority: Major
> Attachments: JM_log.txt, bucket_info.png
>
>
> Hi Team,
> Recently, I'm using S3 as Flink 1.13.1's HA storage directory as below. But I meet an error when deploying the JM and TM to k8s. the steps as below. Could you please have a check?
> 1. Fink-conf.yaml
> blob.server.port: 6124
> high-availability: zookeeper
> high-availability.zookeeper.quorum: zetcd:2181
> high-availability.zookeeper.path.root: /flink
> high-availability.cluster-id: /flink
> *high-availability.storageDir: s3://dsw-dia-test/recovery*
> high-availability.jobmanager.port: 50010
> high-availability.zookeeper.client.acl: open
> state.backend: filesystem
> state.checkpoints.dir: s3://dsw-dia-test/checkpoints
> state.backend.fs.checkpointdir: s3://dsw-dia-test/checkpoints
> s3.path.style.access:true
> fs.allowed-fallback-filesystems: s3
> s3.endpoint: s3.us-south.cloud-object-storage.appdomain.cloud
> s3.access-key: ***************
> s3.secret-key:*************
>
> I'm already building the new image for Flink 1.13 where it's used for the JM and TM's deployments.
>
> 2. The Docker file is
>
> FROM flink:1.13.1-scala_2.11
> #Add the hadoop jar file to folder /opt/flink/lib/
> WORKDIR $FLINK_HOME
> COPY flink-hadoop-compatibility_2.12-1.13.1.jar flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar $FLINK_HOME/lib/
> RUN chown -R flink:flink .;
> #Entropy injection for S3 file systems
> RUN mkdir $FLINK_HOME/plugins/s3-fs-hadoop \
> &&mkdir$FLINK_HOME/plugins/s3-fs-presto
> COPY flink-s3-fs-hadoop-1.13.1.jar $FLINK_HOME/plugins/s3-fs-hadoop
> COPY flink-s3-fs-presto-1.13.1.jar $FLINK_HOME/plugins/s3-fs-presto
> RUN chown -R flink:flink .;
>
> 3. Have checked the jars inside TM pod
>
> the plugins for file system
> drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-hadoop
> drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-presto
> flink@63f9c0076cfc:~/plugins/s3-fs-hadoop$ ls -lrt
> total 19796
> -rw-r--r-- 1 flink flink 20269950 Jul 2 02:25 flink-s3-fs-hadoop-1.13.1.jar
> flink@63f9c0076cfc:~/plugins/s3-fs-presto$ ls -lrt
> total 32692
> -rw-r--r-- 1 flink flink 33474159 May 25 12:20 flink-s3-fs-presto-1.13.1.jar
>
> 4. The error for JM as attachments JM_log.txt.
> 5. pod Status
> kubectl get pods
> NAME READY STATUS RESTARTS AGE
> flink-jobmanager-9f9664f87-4dwg5 0/1 CrashLoopBackOff 6 8m43s
> flink-taskmanager-d7f8b4f87-swl2q 0/1 CrashLoopBackOff 6 8m33s
> flink-taskmanager-d7f8b4f87-vs52c 0/1 Error 3 21m
> zetcd-6458cd87f8-67rxn 2/2 Running 0 3d17h
>
> If you need any more information, please let me know.
> Thanks In Advance!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)