You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Levi Lv (Jira)" <ji...@apache.org> on 2021/07/05 02:40:00 UTC

[jira] [Commented] (FLINK-23231) The JM's pod was failed when the high-availability.storageDir which using the S3

    [ https://issues.apache.org/jira/browse/FLINK-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17374455#comment-17374455 ] 

Levi Lv commented on FLINK-23231:
---------------------------------

BTW, I'm have checked the bucket did not have any file under recovery folder. 
s3://dsw-dia-test/recovery !bucket_info.png!
 

> The JM's pod was failed when the high-availability.storageDir which using the S3
> --------------------------------------------------------------------------------
>
>                 Key: FLINK-23231
>                 URL: https://issues.apache.org/jira/browse/FLINK-23231
>             Project: Flink
>          Issue Type: Bug
>         Environment: [^JM_log.txt]
>            Reporter: Levi Lv
>            Priority: Major
>         Attachments: JM_log.txt, bucket_info.png
>
>
> Hi Team,
> Recently, I'm using S3 as Flink 1.13.1's HA storage directory as below. But I meet an error when deploying the JM and TM to k8s.  the steps as below. Could you please have a check? 
>  1. Fink-conf.yaml 
>  blob.server.port: 6124
>  high-availability: zookeeper
>  high-availability.zookeeper.quorum: zetcd:2181
>  high-availability.zookeeper.path.root: /flink
>  high-availability.cluster-id: /flink
>  *high-availability.storageDir: s3://dsw-dia-test/recovery*
>  high-availability.jobmanager.port: 50010
>  high-availability.zookeeper.client.acl: open
>  state.backend: filesystem
>  state.checkpoints.dir: s3://dsw-dia-test/checkpoints
>  state.backend.fs.checkpointdir: s3://dsw-dia-test/checkpoints
>  s3.path.style.access:true
>  fs.allowed-fallback-filesystems: s3
>  s3.endpoint: s3.us-south.cloud-object-storage.appdomain.cloud
>  s3.access-key: ***************
>  s3.secret-key:*************
>   
>  I'm already building the new image for Flink 1.13 where it's used for the JM and TM's deployments. 
>   
> 2. The Docker file is 
>   
>  FROM flink:1.13.1-scala_2.11
> #Add the hadoop jar file to folder /opt/flink/lib/
>  WORKDIR $FLINK_HOME
>  COPY flink-hadoop-compatibility_2.12-1.13.1.jar flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar $FLINK_HOME/lib/
>  RUN chown -R flink:flink .;
> #Entropy injection for S3 file systems
>  RUN mkdir $FLINK_HOME/plugins/s3-fs-hadoop \
>  &&mkdir$FLINK_HOME/plugins/s3-fs-presto
> COPY flink-s3-fs-hadoop-1.13.1.jar $FLINK_HOME/plugins/s3-fs-hadoop
>  COPY flink-s3-fs-presto-1.13.1.jar $FLINK_HOME/plugins/s3-fs-presto
>  RUN chown -R flink:flink .;
>   
> 3. Have checked the jars inside TM pod 
>   
>  the plugins for file system
>  drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-hadoop
>  drwxr-xr-x 1 flink flink 4096 Jul 4 11:35 s3-fs-presto
> flink@63f9c0076cfc:~/plugins/s3-fs-hadoop$ ls -lrt
>  total 19796
>  -rw-r--r-- 1 flink flink 20269950 Jul 2 02:25 flink-s3-fs-hadoop-1.13.1.jar
> flink@63f9c0076cfc:~/plugins/s3-fs-presto$ ls -lrt
>  total 32692
>  -rw-r--r-- 1 flink flink 33474159 May 25 12:20 flink-s3-fs-presto-1.13.1.jar
>  
> 4. The error for JM as attachments JM_log.txt.
> 5.  pod Status
> kubectl get pods
> NAME READY STATUS RESTARTS AGE
> flink-jobmanager-9f9664f87-4dwg5 0/1 CrashLoopBackOff 6 8m43s
> flink-taskmanager-d7f8b4f87-swl2q 0/1 CrashLoopBackOff 6 8m33s
> flink-taskmanager-d7f8b4f87-vs52c 0/1 Error 3 21m
> zetcd-6458cd87f8-67rxn 2/2 Running 0 3d17h
>   
>  If you need any more information, please let me know. 
>  Thanks In Advance!
>   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)