[jira] [Created] (FLINK-10928) Job unable to stabilise after restart

["Daniel Harper (JIRA)" <ji...@apache.org>]

Daniel Harper created FLINK-10928:
-------------------------------------

             Summary: Job unable to stabilise after restart 
                 Key: FLINK-10928
                 URL: https://issues.apache.org/jira/browse/FLINK-10928
             Project: Flink
          Issue Type: Bug
         Environment: AWS EMR 5.17.0
FLINK 1.5.2
BEAM 2.7.0
            Reporter: Daniel Harper


We've seen a few instances of this occurring in production now (it's difficult to reproduce) but essentially we've seen the following sequence of events: 

1. Job restarts due to exception
2. Job restores from a checkpoint but we get the exception

{code}
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
{code}
3. Job restarts
4. Job restores from a checkpoint but we get the exception

.... repeat a few times within 2-3 minutes....

5. YARN kills containers with out of memory

{code}
2018-11-14 00:16:04,430 INFO  org.apache.flink.yarn.YarnResourceManager                     - Closing TaskExecutor connection container_1541433014652_0001_01_000716 because: Container [pid=7725,containerID=container_1541433014652_0001_01_
000716] is running beyond physical memory limits. Current usage: 6.4 GB of 6.4 GB physical memory used; 8.4 GB of 31.9 GB virtual memory used. Killing container.
Dump of the process-tree for container_1541433014652_0001_01_000716 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 7725 7723 7725 7725 (bash) 0 0 115863552 696 /bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -Xms4995m -Xmx4995m -XX:MaxDirectMemorySize=1533m -Xloggc:/var/log/hadoop-yarn/flink_gc_container_1541433014652_0001_%p.log -XX:GCLogF
ileSize=200M -XX:NumberOfGCLogFiles=10 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCDateStamps -XX:+UseG1GC -Dlog.file=/var/log/hadoop-yarn/containers/application_1541433014652_00
01/container_1541433014652_0001_01_000716/taskmanager.log -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskExecutorRunner --configDir . 1> /var/log/hadoop-yarn/containers/application_1541433014652_0001/container
_1541433014652_0001_01_000716/taskmanager.out 2> /var/log/hadoop-yarn/containers/application_1541433014652_0001/container_1541433014652_0001_01_000716/taskmanager.err
        |- 7738 7725 7725 7725 (java) 6959576 976377 8904458240 1671684 /usr/lib/jvm/java-openjdk/bin/java -Xms4995m -Xmx4995m -XX:MaxDirectMemorySize=1533m -Xloggc:/var/log/hadoop-yarn/flink_gc_container_1541433014652_0001_%p.log -XX:GCL
ogFileSize=200M -XX:NumberOfGCLogFiles=10 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCDateStamps -XX:+UseG1GC -Dlog.file=/var/log/hadoop-yarn/containers/application_1541433014652
_0001/container_1541433014652_0001_01_000716/taskmanager.log -Dlog4j.configuration=file:./log4j.properties org.apache.flink.yarn.YarnTaskExecutorRunner --configDir .
 
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}

6. YARN allocates new containers but the job is never able to get back into a stable state, with constant restarts until eventually the job is cancelled 


We've seen this occurring too https://issues.apache.org/jira/browse/FLINK-10848 with some taskmanagers allocated but sitting 'idle' 








--
This message was sent by Atlassian JIRA
(v7.6.3#76005)