You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yang Wang (Jira)" <ji...@apache.org> on 2021/03/04 07:53:00 UTC
[jira] [Commented] (FLINK-21591) Flink using OSS Backend on Native
K8S found some errors
[ https://issues.apache.org/jira/browse/FLINK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17295069#comment-17295069 ]
Yang Wang commented on FLINK-21591:
-----------------------------------
It seems to be a INFO log of aliyun oss. Could you please check whether the checkpoint has been stored successfully?
> Flink using OSS Backend on Native K8S found some errors
> -------------------------------------------------------
>
> Key: FLINK-21591
> URL: https://issues.apache.org/jira/browse/FLINK-21591
> Project: Flink
> Issue Type: Bug
> Reporter: yufan wang
> Priority: Blocker
>
> version: flink 1.12.0
> environment: Native Kubernetes
> mode: Application Mode
> backend: filesystem OSS
>
> Using OSS as the backend. When the job starts, I get some error logs with Jobmanager and Taskmanager.
> {code:java}
> 2021-03-04 02:55:02,634 WARN org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to find 'mime.types' file in classpath
> 2021-03-04 02:55:02,768 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C38387B9F21
> [HostId]: null
> {code}
> start command:
> {code:java}
> ./bin/flink run-application \
> --target kubernetes-application \
> -Dkubernetes.cluster-id=demo \
> -Dkubernetes.container.image=xx/xx/job:2.0.20 \
> -Dstate.backend=filesystem \
> -Dstate.checkpoints.dir=oss://xx/backend/ \
> -Dfs.oss.endpoint=oss-cn-beijing-internal.aliyuncs.com \
> -Dfs.oss.accessKeyId=xx \
> -Dfs.oss.accessKeySecret=xx \
> -Dhigh-availability.storageDir=oss://xx/flink-ha/ \
> -Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory \
> local:///opt/flink/usrlib/my-flink-job.jar
> {code}
> morelog:
> {code:java}
> Start command: $JAVA_HOME/bin/java -classpath $FLINK_CLASSPATH -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 -Dlog.file=/opt/flink/log/jobmanager.log -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=201326592b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=201326592b
> 2021-03-04 02:54:50,721 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
> 2021-03-04 02:54:50,723 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint (Version: 1.12.0, Scala: 2.12, Rev:fc00492, Date:2020-12-02T08:49:16+01:00)
> 2021-03-04 02:54:50,723 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink
> 2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: <no hadoop dependency found>
> 2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.275-b01
> 2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 989 MiBytes
> 2021-03-04 02:54:50,724 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-8
> 2021-03-04 02:54:50,773 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop Dependency available
> 2021-03-04 02:54:50,773 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/jobmanager.log
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
> 2021-03-04 02:54:50,774 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
> 2021-03-04 02:54:50,775 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,775 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.off-heap.size=134217728b
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.min=201326592b
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-metaspace.size=268435456b
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.heap.size=1073741824b
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.max=201326592b
> 2021-03-04 02:54:50,776 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.12.0.jar:/opt/flink/lib/flink-json-1.12.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.12-1.12.0.jar:/opt/flink/lib/flink-table_2.12-1.12.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/mysql-connector-java-5.1.49.jar:/opt/flink/lib/flink-dist_2.12-1.12.0.jar:::
> 2021-03-04 02:54:50,777 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
> 2021-03-04 02:54:50,777 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT]
> 2021-03-04 02:54:50,788 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
> 2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
> 2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.cluster-id, demo
> 2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
> 2021-03-04 02:54:50,789 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, demo
> 2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeySecret, ******
> 2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeyId, ***
> 2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.storageDir, oss://***/flink-ha/
> 2021-03-04 02:54:50,790 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.savepoint.ignore-unclaimed-state, false
> 2021-03-04 02:54:50,790 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
> 2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, **/**/job:2.0.20
> 2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
> 2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
> 2021-03-04 02:54:50,791 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
> 2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.jobmanager.port, 6123
> 2021-03-04 02:54:50,791 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
> 2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
> 2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-application
> 2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
> 2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
> 2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
> 2021-03-04 02:54:50,792 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.attached, true
> 2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
> 2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> 2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.shutdown-on-attached-exit, false
> 2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/my-flink-job.jar
> 2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
> 2021-03-04 02:54:50,793 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend, filesystem
> 2021-03-04 02:54:50,794 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, oss://***/backend/
> 2021-03-04 02:54:51,100 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting KubernetesApplicationClusterEntrypoint.
> 2021-03-04 02:54:51,108 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem.
> 2021-03-04 02:54:51,111 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
> 2021-03-04 02:54:51,188 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context.
> 2021-03-04 02:54:51,195 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
> 2021-03-04 02:54:51,198 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-5821322633800809072.conf.
> 2021-03-04 02:54:51,202 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
> 2021-03-04 02:54:51,203 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services.
> 2021-03-04 02:54:51,219 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.18.242.149:6123, bind address 0.0.0.0:6123.
> 2021-03-04 02:54:52,216 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
> 2021-03-04 02:54:52,278 INFO akka.remote.Remoting [] - Starting remoting
> 2021-03-04 02:54:52,407 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink@172.18.242.149:6123]
> 2021-03-04 02:54:52,519 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@172.18.242.149:6123
> 2021-03-04 02:54:53,499 INFO org.apache.flink.runtime.blob.FileSystemBlobStore [] - Creating highly available BLOB storage directory at oss://***/flink-ha/demo/blob
> 2021-03-04 02:54:53,855 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404BFDBD813C3838043521
> [HostId]: null
> 2021-03-04 02:54:53,866 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/blobStore-6d45b5b0-9271-4c3a-9b42-0cda22da25aa
> 2021-03-04 02:54:53,868 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
> 2021-03-04 02:54:53,877 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter configured, no metrics will be exposed/reported.
> 2021-03-04 02:54:53,879 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.18.242.149:0, bind address 0.0.0.0:0.
> 2021-03-04 02:54:53,895 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
> 2021-03-04 02:54:53,900 INFO akka.remote.Remoting [] - Starting remoting
> 2021-03-04 02:54:53,907 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.18.242.149:45382]
> 2021-03-04 02:54:53,979 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@172.18.242.149:45382
> 2021-03-04 02:54:53,990 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
> 2021-03-04 02:54:54,082 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload does not exist.
> 2021-03-04 02:54:54,083 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created directory /tmp/flink-web-eec17919-59bc-4700-af3b-0e1c24ca5d09/flink-web-upload for file uploads.
> 2021-03-04 02:54:54,094 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting rest endpoint.
> 2021-03-04 02:54:54,412 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/jobmanager.log
> 2021-03-04 02:54:54,412 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/jobmanager.out
> 2021-03-04 02:54:54,605 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest endpoint listening at 172.18.242.149:8081
> 2021-03-04 02:54:54,788 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector demo-restserver-leader with lock identity ecd54426-0378-4460-9a73-6fb2b239e76f.
> 2021-03-04 02:54:54,982 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='demo-restserver-leader'}.
> 2021-03-04 02:54:54,983 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend listening at http://172.18.242.149:8081.
> 2021-03-04 02:54:54,993 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
> 2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
> 2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
> 2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.cluster-id, demo
> 2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
> 2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, demo
> 2021-03-04 02:54:55,005 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeySecret, ******
> 2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.accessKeyId, ***
> 2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.storageDir, oss://***/flink-ha/
> 2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.savepoint.ignore-unclaimed-state, false
> 2021-03-04 02:54:55,006 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:10: "$internal.application.program-args: "
> 2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, **/**/job:2.0.20
> 2021-03-04 02:54:55,006 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
> 2021-03-04 02:54:55,073 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
> 2021-03-04 02:54:55,073 WARN org.apache.flink.configuration.GlobalConfiguration [] - Error while trying to split key and value in configuration file /opt/flink/conf/flink-conf.yaml:14: "pipeline.classpaths: "
> 2021-03-04 02:54:55,073 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.jobmanager.port, 6123
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-application
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.attached, true
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> 2021-03-04 02:54:55,074 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.shutdown-on-attached-exit, false
> 2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: pipeline.jars, local:///opt/flink/usrlib/my-flink-job.jar
> 2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.oss.endpoint, oss-cn-beijing-internal.aliyuncs.com
> 2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.backend, filesystem
> 2021-03-04 02:54:55,075 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, oss://***/backend/
> 2021-03-04 02:54:55,081 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at akka://flink/user/rpc/resourcemanager_0 .
> 2021-03-04 02:54:55,093 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector demo-dispatcher-leader with lock identity ecd54426-0378-4460-9a73-6fb2b239e76f.
> 2021-03-04 02:54:55,099 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='demo-dispatcher-leader'}.
> 2021-03-04 02:54:55,279 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='demo-resourcemanager-leader'}.
> 2021-03-04 02:54:55,285 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='demo-dispatcher-leader'}.
> 2021-03-04 02:54:55,492 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for demo-restserver-leader.
> 2021-03-04 02:54:55,493 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://172.18.242.149:8081 was granted leadership with leaderSessionID=77f241fd-89a6-4183-977d-4c4633f400fe
> 2021-03-04 02:54:55,494 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for demo-dispatcher-leader.
> 2021-03-04 02:54:55,578 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess.
> 2021-03-04 02:54:55,581 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs.
> 2021-03-04 02:54:55,588 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job ids [] from KubernetesStateHandleStore{configMapName='demo-dispatcher-leader'}
> 2021-03-04 02:54:55,588 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 0 persisted job graphs.
> 2021-03-04 02:54:55,595 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_1 .
> 2021-03-04 02:54:55,684 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: false)
> 2021-03-04 02:54:56,089 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0 pods from previous attempts, current attempt id is 1.
> 2021-03-04 02:54:56,090 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Recovered 0 workers from previous attempt.
> 2021-03-04 02:54:56,090 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector demo-resourcemanager-leader with lock identity ecd54426-0378-4460-9a73-6fb2b239e76f.
> 2021-03-04 02:54:56,097 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='demo-resourcemanager-leader'}.
> 2021-03-04 02:54:56,175 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected ecd54426-0378-4460-9a73-6fb2b239e76f for demo-resourcemanager-leader.
> 2021-03-04 02:54:56,274 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - ResourceManager akka.tcp://flink@172.18.242.149:6123/user/rpc/resourcemanager_0 was granted leadership with fencing token aae3a328d005a1fdb189546b6f3c46ba
> 2021-03-04 02:54:56,278 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Starting the SlotManager.
> 2021-03-04 02:54:56,512 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class com.**.entity.TransactionBase$Source is not public so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
> 2021-03-04 02:54:56,519 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class com.**.entity.TransactionBase$Source is not public so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
> 2021-03-04 02:54:56,575 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class com.**.entity.TransactionKeyBy is missing a default constructor so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
> 2021-03-04 02:54:56,614 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class com.**.entity.TransactionSum is missing a default constructor so it cannot be used as a POJO type and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
> 2021-03-04 02:54:56,806 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 00000000000000000000000000000000 is submitted.
> 2021-03-04 02:54:56,806 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=00000000000000000000000000000000.
> 2021-03-04 02:54:57,913 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C01BD813C38389D6321
> [HostId]: null
> 2021-03-04 02:54:57,915 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C01BD813C3838AA6321
> [HostId]: null
> 2021-03-04 02:54:58,902 WARN org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Unhandled exception
> java.io.IOException: Connection reset by peer // Ignore these exceptions, caused by cloud provider LoadBalance
> ...
> 2021-03-04 02:55:02,535 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838DC9C21
> [HostId]: null
> 2021-03-04 02:55:02,537 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838E69C21
> [HostId]: null
> 2021-03-04 02:55:02,634 WARN org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - Unable to find 'mime.types' file in classpath
> 2021-03-04 02:55:02,768 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C38387B9F21
> [HostId]: null
> 2021-03-04 02:55:02,770 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838869F21
> [HostId]: null
> 2021-03-04 02:55:02,888 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838E4A021
> [HostId]: null
> 2021-03-04 02:55:02,890 INFO org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss [] - [Server]Unable to execute HTTP request: Not Found
> [ErrorCode]: NoSuchKey
> [RequestId]: 60404C06BD813C3838ECA021
> [HostId]: null
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)