You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Amit Jain <aj...@gmail.com> on 2018/04/17 07:35:37 UTC
Testing on Flink 1.5
Hi,
We have started testing our workload on Flink 1.5.0 (Commit: 2af481a)
and faced few issues. Here, we are using YARN as cluster manager.
- Few of our jobs are stuck with logs [1], these jobs are only able
to allocate JM and couldn't get any TM, however, there are ample
resource on our cluster.
- All completed application on YARN are marked as KILLED, instead
of FINISHED. It creates problem on tracking jobs as we require to
navigate on each application logs and also history server is not
logging any completed application.
- We are running our system on AWS EMR and using CORE nodes at its
bare minimum, we see with new paradigm of application submission we
are hitting Out of space error in our HDFS Data Node. Do you think, we
should have configuration to automatically evict these .flink/ files?
- Lastly, we are also observing Akka.ask.timeout error for few of
our jobs (JM's logs[2]), we tried to increase this parameter to 2 min
from default 10 s and still observe the same.
Please help us guide through these issues.
--
Thanks,
Amit
[1] JM logs where jobs are not even started
2018-04-17 04:26:16,165 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-04-17 04:26:16,166 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a,
Date:02.04.2018 @ 15:17:35 UTC)
2018-04-17 04:26:16,166 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS
current user: yarn
2018-04-17 04:26:16,843 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Current Hadoop/Kerberos user: hadoop
2018-04-17 04:26:16,843 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM:
OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-04-17 04:26:16,843 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Maximum heap size: 406 MiBytes
2018-04-17 04:26:16,843 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-04-17 04:26:16,844 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Hadoop version: 2.8.3
2018-04-17 04:26:16,844 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM
Options:
2018-04-17 04:26:16,844 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Xmx424m
2018-04-17 04:26:16,845 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
2018-04-17 04:26:16,845 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlogback.configurationFile=file:logback.xml
2018-04-17 04:26:16,845 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog4j.configuration=file:log4j.properties
2018-04-17 04:26:16,845 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Program Arguments: (none)
2018-04-17 04:26:16,845 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
2018-04-17 04:26:16,846 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-04-17 04:26:16,848 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-17 04:26:16,851 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN
daemon is running as: hadoop Yarn client user obtainer: hadoop
2018-04-17 04:26:16,854 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.archive.fs.dir,
hdfs:///completed-jobs/
2018-04-17 04:26:16,854 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: fs.default-scheme, s3a
2018-04-17 04:26:16,854 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: classloader.resolve-order,
parent-first
2018-04-17 04:26:16,855 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: high-availability.cluster-id,
application_1523726493647_0503
2018-04-17 04:26:16,855 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.heap.mb, 1024
2018-04-17 04:26:16,855 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: yarn.heap-cutoff-ratio, 0.5
2018-04-17 04:26:16,855 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.memory.preallocate, true
2018-04-17 04:26:16,855 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: yarn.maximum-failed-containers, -1
2018-04-17 04:26:16,856 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: akka.watch.heartbeat.pause, 600 s
2018-04-17 04:26:16,856 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: internal.cluster.execution-mode,
NORMAL
2018-04-17 04:26:16,856 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.memory.off-heap, on
2018-04-17 04:26:16,856 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.memory.fraction, 0.1
2018-04-17 04:26:16,856 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-04-17 04:26:16,857 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
2018-04-17 04:26:16,857 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: akka.ask.timeout, 2 min
2018-04-17 04:26:16,857 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.heap.mb, 10772
2018-04-17 04:26:16,857 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: historyserver.archive.fs.dir,
hdfs:///completed-jobs/
2018-04-17 04:26:16,874 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Setting directories for temporary files to:
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503
2018-04-17 04:26:16,896 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Starting YarnSessionClusterEntrypoint.
2018-04-17 04:26:16,897 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Install default filesystem.
2018-04-17 04:26:17,014 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop
user set to hadoop (auth:SIMPLE)
2018-04-17 04:26:17,034 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Initializing cluster services.
2018-04-17 04:26:17,041 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying
to start actor system at
ip-10-0-22-186.ap-southeast-1.compute.internal:38669
2018-04-17 04:26:17,671 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2018-04-17 04:26:17,794 INFO akka.remote.Remoting
- Starting remoting
2018-04-17 04:26:17,937 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669]
2018-04-17 04:26:17,948 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor
system started at
akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669
2018-04-17 04:26:17,980 INFO org.apache.flink.runtime.blob.BlobServer
- Created BLOB server storage directory
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-bca451c7-977d-425a-9e8f-4fe324fa8623
2018-04-17 04:26:17,983 INFO org.apache.flink.runtime.blob.BlobServer
- Started BLOB server at 0.0.0.0:44217 - max
concurrent requests: 50 - max backlog: 1000
2018-04-17 04:26:18,009 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No
metrics reporter configured, no metrics will be exposed/reported.
2018-04-17 04:26:18,015 INFO
org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore -
Initializing FileArchivedExecutionGraphStore: Storage directory
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/executionGraphStore-28bf6537-0bbf-486c-a346-084bd7517d7d,
expiration time 3600000, maximum cache size 52428800 bytes.
2018-04-17 04:26:18,047 INFO
org.apache.flink.runtime.blob.TransientBlobCache -
Created BLOB cache storage directory
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-f1de0cad-77b0-4bbd-b218-63e32b10bf6a
2018-04-17 04:26:18,057 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload
directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload
does not exist, or has been deleted externally. Previously uploaded
files are no longer available.
2018-04-17 04:26:18,059 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
Created directory
/tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload
for file uploads.
2018-04-17 04:26:18,063 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
Starting rest endpoint.
2018-04-17 04:26:18,386 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils -
Determined location of main cluster component log file:
/var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
2018-04-17 04:26:18,386 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils -
Determined location of main cluster component stdout file:
/var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.out
2018-04-17 04:26:18,454 INFO org.apache.flink.runtime.rest.RestClient
- Rest client endpoint started.
2018-04-17 04:26:18,538 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest
endpoint listening at
ip-10-0-22-186.ap-southeast-1.compute.internal:44431
2018-04-17 04:26:18,542 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
http://ip-10-0-22-186.ap-southeast-1.compute.internal:44431 was
granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2018-04-17 04:26:18,558 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at
akka://flink/user/resourcemanager .
2018-04-17 04:26:18,713 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/dispatcher .
2018-04-17 04:26:18,739 INFO
org.apache.flink.yarn.YarnResourceManager -
ResourceManager
akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/resourcemanager
was granted leadership with fencing token
00000000000000000000000000000000
2018-04-17 04:26:18,740 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager -
Starting the SlotManager.
2018-04-17 04:26:18,769 INFO org.apache.hadoop.yarn.client.RMProxy
- Connecting to ResourceManager at
ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
2018-04-17 04:26:19,083 INFO
org.apache.flink.yarn.YarnResourceManager -
Recovered 0 containers from previous attempts ([]).
2018-04-17 04:26:19,086 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy
- yarn.client.max-cached-nodemanagers-proxies : 0
2018-04-17 04:26:19,101 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/dispatcher
was granted leadership with fencing token
00000000000000000000000000000000
2018-04-17 04:26:19,101 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Recovering all persisted jobs.
[2] JM's logs
2018-04-16 12:53:15,825 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-04-16 12:53:15,827 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a,
Date:02.04.2018 @ 15:17:35 UTC)
2018-04-16 12:53:15,827 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS
current user: yarn
2018-04-16 12:53:16,798 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Current Hadoop/Kerberos user: hadoop
2018-04-16 12:53:16,799 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM:
OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-04-16 12:53:16,799 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Maximum heap size: 406 MiBytes
2018-04-16 12:53:16,799 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-04-16 12:53:16,800 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Hadoop version: 2.8.3
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM
Options:
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Xmx424m
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlogback.configurationFile=file:logback.xml
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog4j.configuration=file:log4j.properties
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Program Arguments: (none)
2018-04-16 12:53:16,801 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
2018-04-16 12:53:16,803 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2018-04-16 12:53:16,805 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-16 12:53:16,809 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN
daemon is running as: hadoop Yarn client user obtainer: hadoop
2018-04-16 12:53:16,813 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.archive.fs.dir,
hdfs:///completed-jobs/
2018-04-16 12:53:16,814 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: fs.default-scheme, s3a
2018-04-16 12:53:16,814 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: classloader.resolve-order,
parent-first
2018-04-16 12:53:16,814 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: high-availability.cluster-id,
application_1523726493647_0302
2018-04-16 12:53:16,815 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: jobmanager.heap.mb, 1024
2018-04-16 12:53:16,815 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: yarn.heap-cutoff-ratio, 0.5
2018-04-16 12:53:16,815 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.memory.preallocate, true
2018-04-16 12:53:16,815 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: yarn.maximum-failed-containers, -1
2018-04-16 12:53:16,815 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: akka.watch.heartbeat.pause, 600 s
2018-04-16 12:53:16,815 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: internal.cluster.execution-mode,
NORMAL
2018-04-16 12:53:16,816 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.memory.off-heap, on
2018-04-16 12:53:16,816 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.memory.fraction, 0.1
2018-04-16 12:53:16,816 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-04-16 12:53:16,816 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
2018-04-16 12:53:16,816 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: taskmanager.heap.mb, 9216
2018-04-16 12:53:16,817 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: historyserver.archive.fs.dir,
hdfs:///completed-jobs/
2018-04-16 12:53:16,833 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Setting directories for temporary files to:
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302
2018-04-16 12:53:16,859 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Starting YarnSessionClusterEntrypoint.
2018-04-16 12:53:16,859 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Install default filesystem.
2018-04-16 12:53:17,036 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop
user set to hadoop (auth:SIMPLE)
2018-04-16 12:53:17,071 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Initializing cluster services.
2018-04-16 12:53:17,081 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying
to start actor system at
ip-10-0-22-186.ap-southeast-1.compute.internal:36411
2018-04-16 12:53:18,022 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2018-04-16 12:53:18,254 INFO akka.remote.Remoting
- Starting remoting
2018-04-16 12:53:18,461 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411]
2018-04-16 12:53:18,488 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor
system started at
akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411
2018-04-16 12:53:18,559 INFO org.apache.flink.runtime.blob.BlobServer
- Created BLOB server storage directory
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-9ec0a59f-c03a-446f-a841-60a65892b383
2018-04-16 12:53:18,566 INFO org.apache.flink.runtime.blob.BlobServer
- Started BLOB server at 0.0.0.0:46869 - max
concurrent requests: 50 - max backlog: 1000
2018-04-16 12:53:18,632 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No
metrics reporter configured, no metrics will be exposed/reported.
2018-04-16 12:53:18,638 INFO
org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore -
Initializing FileArchivedExecutionGraphStore: Storage directory
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/executionGraphStore-ee071b9a-ad86-4b36-860c-6501f509ecf9,
expiration time 3600000, maximum cache size 52428800 bytes.
2018-04-16 12:53:18,726 INFO
org.apache.flink.runtime.blob.TransientBlobCache -
Created BLOB cache storage directory
/mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-c15647ee-31c1-451d-84d2-729a67e065d8
2018-04-16 12:53:18,750 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload
directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload
does not exist, or has been deleted externally. Previously uploaded
files are no longer available.
2018-04-16 12:53:18,757 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
Created directory
/tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload
for file uploads.
2018-04-16 12:53:18,765 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
Starting rest endpoint.
2018-04-16 12:53:19,361 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils -
Determined location of main cluster component log file:
/var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
2018-04-16 12:53:19,362 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils -
Determined location of main cluster component stdout file:
/var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.out
2018-04-16 12:53:19,442 INFO org.apache.flink.runtime.rest.RestClient
- Rest client endpoint started.
2018-04-16 12:53:19,589 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest
endpoint listening at
ip-10-0-22-186.ap-southeast-1.compute.internal:45697
2018-04-16 12:53:19,589 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
http://ip-10-0-22-186.ap-southeast-1.compute.internal:45697 was
granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2018-04-16 12:53:19,614 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at
akka://flink/user/resourcemanager .
2018-04-16 12:53:19,979 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/dispatcher .
2018-04-16 12:53:20,049 INFO
org.apache.flink.yarn.YarnResourceManager -
ResourceManager
akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/resourcemanager
was granted leadership with fencing token
00000000000000000000000000000000
2018-04-16 12:53:20,050 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager -
Starting the SlotManager.
2018-04-16 12:53:20,101 INFO org.apache.hadoop.yarn.client.RMProxy
- Connecting to ResourceManager at
ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
2018-04-16 12:53:20,649 INFO
org.apache.flink.yarn.YarnResourceManager -
Recovered 0 containers from previous attempts ([]).
2018-04-16 12:53:20,655 INFO
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy
- yarn.client.max-cached-nodemanagers-proxies : 0
2018-04-16 12:53:20,706 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/dispatcher
was granted leadership with fencing token
00000000000000000000000000000000
2018-04-16 12:53:20,706 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Recovering all persisted jobs.
2018-04-16 12:53:26,207 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Submitting job 58a1b149b5b69de66bb931836c459d2b
(mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job).
2018-04-16 12:53:26,223 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster
at akka://flink/user/jobmanager_0 .
2018-04-16 12:53:26,237 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Initializing job
mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job
(58a1b149b5b69de66bb931836c459d2b).
2018-04-16 12:53:26,279 INFO
org.apache.flink.runtime.jobmaster.JobMaster - Using
restart strategy NoRestartStrategy for
mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job
(58a1b149b5b69de66bb931836c459d2b).
2018-04-16 12:53:26,285 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService -
Starting RPC endpoint for
org.apache.flink.runtime.jobmaster.slotpool.SlotPool at
akka://flink/user/820d207e-fddd-4e24-b327-1f3c1cb25024 .
2018-04-16 12:53:26,320 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job
recovers via failover strategy: full graph restart
2018-04-16 12:53:26,386 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Running initialization on master for job
mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job
(58a1b149b5b69de66bb931836c459d2b).
2018-04-16 12:53:26,671 INFO
org.apache.flink.runtime.jobmaster.JobMaster -
Successfully ran initialization on master in 284 ms.
2018-04-16 12:53:36,204 ERROR
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler -
Implementation error: Unhandled exception.
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#1710592138]] after [10000 ms].
Sender[null] sent message of type
"org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2018-04-16 12:53:36,300 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2018-04-16 12:53:36,311 INFO
org.apache.flink.runtime.blob.TransientBlobCache -
Shutting down BLOB cache
2018-04-16 12:53:36,314 INFO org.apache.flink.runtime.blob.BlobServer
- Stopped BLOB server at 0.0.0.0:46869
Re: Testing on Flink 1.5
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Amit,
web.timeout should only affect RPC calls originating from the REST API. In
FLIP-6, the submission of the job graph happens via HTTP. The value under
akka.ask.timeout is still used as the default timeout for RPC calls [1][2].
Since you also had custom heartbeats settings, you should consider setting
heartbeat.interval and heartbeat.timeout when using FLIP-6 mode [3]. AFAIK
Akka's DeathWatch is not used anymore to detect TaskManager failures. Hence,
akka.watch.heartbeat.interval should have no effect.
Best,
Gary
[1]
https://github.com/apache/flink/blob/fb254763c00df5d336c6defa1ae960e32c97b2ae/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java#L389
[2]
https://github.com/apache/flink/blob/fb254763c00df5d336c6defa1ae960e32c97b2ae/flink-runtime/src/main/scala/org/apache/flink/runtime/akka/AkkaUtils.scala#L606
[3]
https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#heartbeat-manager
On Fri, Apr 20, 2018 at 12:22 PM, Amit Jain <aj...@gmail.com> wrote:
> Hi Gary,
>
> This setting has resolved the issue. Does it increase timeout for all the
> RPC or specific components?
>
> We had following settings in Flink 1.3.2 and they did the job for us.
>
> akka.watch.heartbeat.pause: 600 s
> akka.client.timeout: 5 min
> akka.ask.timeout: 120 s
>
>
> --
> Thanks,
> Amit
>
Re: Testing on Flink 1.5
Posted by Amit Jain <aj...@gmail.com>.
Hi Gary,
This setting has resolved the issue. Does it increase timeout for all the
RPC or specific components?
We had following settings in Flink 1.3.2 and they did the job for us.
akka.watch.heartbeat.pause: 600 s
akka.client.timeout: 5 min
akka.ask.timeout: 120 s
--
Thanks,
Amit
Re: Testing on Flink 1.5
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Amit,
Can you try setting:
web.timeout: 120000
This will increase the timeout of the RPC call to 2 minutes. Did you need to
raise the timeouts prior to FLIP-6 as well?
Best,
Gary
On Wed, Apr 18, 2018 at 3:09 PM, Amit Jain <aj...@gmail.com> wrote:
> Hi Gary,
>
> Please find my reply in place. We have also logged https://issues.apache.
> org/jira/browse/FLINK-9207, please have a look on this.
>
>
> --
> Thanks,
> Amit
>
>
> On Tue, Apr 17, 2018 at 9:10 PM, Gary Yao <ga...@data-artisans.com> wrote:
>
>> Hi Amit,
>>
>> Thank you for testing the 1.5 Release Candidate. Find my answers below.
>>
>> > Few of our jobs are stuck with logs [1], these jobs are only able to
>> allocate JM and couldn't get any TM, however, there are ample resource on
>> our cluster.
>>
>> In the logs you attached I cannot see that a job was submitted. If a job
>> is
>> submitted, the dispatcher should log: "Submitting job <job_id>
>> (<job_name>)"
>> [1]. Could it be that the attached logs are incomplete, or the client
>> already
>> fails to submit the job? Can you also attach the client logs?
>>
>> I agree with you, we should have these logs however it is not happening.
> I'll work on it further and provide more info, right now, I can provide
> logs which got generated later.
>
> 2018-04-16 04:36:43,747 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-43.ap-southeast-1.compute.internal:39329/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
> 2018-04-16 04:36:43,747 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
> 2018-04-16 17:21:37,800 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:21:42,803 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:21:47,805 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:21:52,806 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:21:57,807 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:02,808 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:07,810 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:12,812 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:17,813 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:22,814 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:27,816 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:32,817 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:37,818 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:42,821 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:47,822 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:52,825 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:22:57,826 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:23:02,828 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:23:07,829 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
> 2018-04-16 17:23:12,832 INFO org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Updating with new AMRMToken
>
>
>
>> > All completed application on YARN are marked as KILLED, instead of
>> FINISHED. It creates problem on tracking jobs as we require to navigate on
>> each application logs and also history server is not logging any completed
>> application.
>>
>> I was able to reproduce this: In non-detached mode, the State and
>> FinalStatus
>> are set to KILLED. In detached mode, the Status and FinalStatus are set
>> correctly. I have re-opened this ticket:
>> https://issues.apache.org/jira/browse/FLINK-8900
>>
>> You are also right about the History Server, which is currently not
>> implemented
>> for FLIP-6. Here is the ticket for reference:
>> https://issues.apache.org/jira/browse/FLINK-9194
>>
>>
>> > We are running our system on AWS EMR and using CORE nodes at its bare
>> minimum, we see with new paradigm of application submission we are hitting
>> Out of space error in our HDFS Data Node. Do you think, we should have
>> configuration to automatically evict these .flink/ files?
>>
>> This seems to be a regression. In Flink 1.4., the files are cleaned up
>> automatically. I opened this ticket:
>> https://issues.apache.org/jira/browse/FLINK-9196
>>
>>
>> > Lastly, we are also observing Akka.ask.timeout error for few of our
>> jobs (JM's logs[2]), we tried to increase this parameter to 2 min from
>> default 10 s and still observe the same.
>>
>> Does this always happen for the same jobs? I cannot reproduce this.
>>
>> Yes, it happening for same set of jobs. Specifically, these jobs are
> having multiple S3 objects to work with. I tried increasing
> akka.ask.timeout which is not getting reflected on JM's log also and same
> has been experienced while running the jobs.
> I also tried one of these jobs on empty cluster(EMR TASK 15 node,
> r3.xlarge type) and still face same.
>
> Best,
>> Gary
>>
>> [1] https://github.com/apache/flink/blob/master/flink-runtime/
>> src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L235
>>
>>
>> On Tue, Apr 17, 2018 at 9:35 AM, Amit Jain <aj...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>
>>> We have started testing our workload on Flink 1.5.0 (Commit: 2af481a) and faced few issues. Here, we are using YARN as cluster manager.
>>>
>>>
>>> - Few of our jobs are stuck with logs [1], these jobs are only able to allocate JM and couldn't get any TM, however, there are ample resource on our cluster.
>>> - All completed application on YARN are marked as KILLED, instead of FINISHED. It creates problem on tracking jobs as we require to navigate on each application logs and also history server is not logging any completed application.
>>> - We are running our system on AWS EMR and using CORE nodes at its bare minimum, we see with new paradigm of application submission we are hitting Out of space error in our HDFS Data Node. Do you think, we should have configuration to automatically evict these .flink/ files?
>>> - Lastly, we are also observing Akka.ask.timeout error for few of our jobs (JM's logs[2]), we tried to increase this parameter to 2 min from default 10 s and still observe the same.
>>>
>>> Please help us guide through these issues.
>>>
>>> --
>>> Thanks,
>>> Amit
>>>
>>> [1] JM logs where jobs are not even started
>>>
>>> 2018-04-17 04:26:16,165 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>>> 2018-04-17 04:26:16,166 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a, Date:02.04.2018 @ 15:17:35 UTC)
>>> 2018-04-17 04:26:16,166 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
>>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hadoop
>>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
>>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
>>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-openjdk
>>> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
>>> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
>>> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
>>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
>>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml
>>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
>>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
>>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
>>> 2018-04-17 04:26:16,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>>> 2018-04-17 04:26:16,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
>>> 2018-04-17 04:26:16,851 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hadoop Yarn client user obtainer: hadoop
>>> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///completed-jobs/
>>> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, s3a
>>> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
>>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1523726493647_0503
>>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
>>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.5
>>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, true
>>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, -1
>>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.watch.heartbeat.pause, 600 s
>>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
>>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, on
>>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.fraction, 0.1
>>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
>>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
>>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 2 min
>>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 10772
>>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs:///completed-jobs/
>>> 2018-04-17 04:26:16,874 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503
>>> 2018-04-17 04:26:16,896 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint.
>>> 2018-04-17 04:26:16,897 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
>>> 2018-04-17 04:26:17,014 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hadoop (auth:SIMPLE)
>>> 2018-04-17 04:26:17,034 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
>>> 2018-04-17 04:26:17,041 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at ip-10-0-22-186.ap-southeast-1.compute.internal:38669
>>> 2018-04-17 04:26:17,671 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
>>> 2018-04-17 04:26:17,794 INFO akka.remote.Remoting - Starting remoting
>>> 2018-04-17 04:26:17,937 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669]
>>> 2018-04-17 04:26:17,948 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669
>>> 2018-04-17 04:26:17,980 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-bca451c7-977d-425a-9e8f-4fe324fa8623
>>> 2018-04-17 04:26:17,983 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:44217 - max concurrent requests: 50 - max backlog: 1000
>>> 2018-04-17 04:26:18,009 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
>>> 2018-04-17 04:26:18,015 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/executionGraphStore-28bf6537-0bbf-486c-a346-084bd7517d7d, expiration time 3600000, maximum cache size 52428800 bytes.
>>> 2018-04-17 04:26:18,047 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-f1de0cad-77b0-4bbd-b218-63e32b10bf6a
>>> 2018-04-17 04:26:18,057 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
>>> 2018-04-17 04:26:18,059 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload for file uploads.
>>> 2018-04-17 04:26:18,063 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
>>> 2018-04-17 04:26:18,386 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
>>> 2018-04-17 04:26:18,386 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.out
>>> 2018-04-17 04:26:18,454 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
>>> 2018-04-17 04:26:18,538 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at ip-10-0-22-186.ap-southeast-1.compute.internal:44431
>>> 2018-04-17 04:26:18,542 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://ip-10-0-22-186.ap-southeast-1.compute.internal:44431 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
>>> 2018-04-17 04:26:18,558 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
>>> 2018-04-17 04:26:18,713 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
>>> 2018-04-17 04:26:18,739 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
>>> 2018-04-17 04:26:18,740 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
>>> 2018-04-17 04:26:18,769 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
>>> 2018-04-17 04:26:19,083 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]).
>>> 2018-04-17 04:26:19,086 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
>>> 2018-04-17 04:26:19,101 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
>>> 2018-04-17 04:26:19,101 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
>>>
>>>
>>>
>>>
>>> [2] JM's logs
>>>
>>> 2018-04-16 12:53:15,825 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>>> 2018-04-16 12:53:15,827 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a, Date:02.04.2018 @ 15:17:35 UTC)
>>> 2018-04-16 12:53:15,827 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
>>> 2018-04-16 12:53:16,798 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hadoop
>>> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
>>> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
>>> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-openjdk
>>> 2018-04-16 12:53:16,800 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
>>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
>>> 2018-04-16 12:53:16,803 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>>> 2018-04-16 12:53:16,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
>>> 2018-04-16 12:53:16,809 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hadoop Yarn client user obtainer: hadoop
>>> 2018-04-16 12:53:16,813 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///completed-jobs/
>>> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, s3a
>>> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
>>> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1523726493647_0302
>>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
>>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.5
>>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, true
>>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, -1
>>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.watch.heartbeat.pause, 600 s
>>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
>>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, on
>>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.fraction, 0.1
>>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
>>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
>>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 9216
>>> 2018-04-16 12:53:16,817 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs:///completed-jobs/
>>> 2018-04-16 12:53:16,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302
>>> 2018-04-16 12:53:16,859 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint.
>>> 2018-04-16 12:53:16,859 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
>>> 2018-04-16 12:53:17,036 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hadoop (auth:SIMPLE)
>>> 2018-04-16 12:53:17,071 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
>>> 2018-04-16 12:53:17,081 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at ip-10-0-22-186.ap-southeast-1.compute.internal:36411
>>> 2018-04-16 12:53:18,022 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
>>> 2018-04-16 12:53:18,254 INFO akka.remote.Remoting - Starting remoting
>>> 2018-04-16 12:53:18,461 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411]
>>> 2018-04-16 12:53:18,488 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411
>>> 2018-04-16 12:53:18,559 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-9ec0a59f-c03a-446f-a841-60a65892b383
>>> 2018-04-16 12:53:18,566 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:46869 - max concurrent requests: 50 - max backlog: 1000
>>> 2018-04-16 12:53:18,632 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
>>> 2018-04-16 12:53:18,638 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/executionGraphStore-ee071b9a-ad86-4b36-860c-6501f509ecf9, expiration time 3600000, maximum cache size 52428800 bytes.
>>> 2018-04-16 12:53:18,726 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-c15647ee-31c1-451d-84d2-729a67e065d8
>>> 2018-04-16 12:53:18,750 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
>>> 2018-04-16 12:53:18,757 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload for file uploads.
>>> 2018-04-16 12:53:18,765 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
>>> 2018-04-16 12:53:19,361 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
>>> 2018-04-16 12:53:19,362 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.out
>>> 2018-04-16 12:53:19,442 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
>>> 2018-04-16 12:53:19,589 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at ip-10-0-22-186.ap-southeast-1.compute.internal:45697
>>> 2018-04-16 12:53:19,589 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://ip-10-0-22-186.ap-southeast-1.compute.internal:45697 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
>>> 2018-04-16 12:53:19,614 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
>>> 2018-04-16 12:53:19,979 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
>>> 2018-04-16 12:53:20,049 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
>>> 2018-04-16 12:53:20,050 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
>>> 2018-04-16 12:53:20,101 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
>>> 2018-04-16 12:53:20,649 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]).
>>> 2018-04-16 12:53:20,655 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
>>> 2018-04-16 12:53:20,706 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
>>> 2018-04-16 12:53:20,706 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
>>> 2018-04-16 12:53:26,207 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 58a1b149b5b69de66bb931836c459d2b (mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job).
>>> 2018-04-16 12:53:26,223 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
>>> 2018-04-16 12:53:26,237 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
>>> 2018-04-16 12:53:26,279 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy NoRestartStrategy for mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
>>> 2018-04-16 12:53:26,285 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/820d207e-fddd-4e24-b327-1f3c1cb25024 .
>>> 2018-04-16 12:53:26,320 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
>>> 2018-04-16 12:53:26,386 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
>>> 2018-04-16 12:53:26,671 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 284 ms.
>>> 2018-04-16 12:53:36,204 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Implementation error: Unhandled exception.
>>> akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#1710592138]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
>>> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
>>> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
>>> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>>> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>>> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>>> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
>>> at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
>>> at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
>>> at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
>>> at java.lang.Thread.run(Thread.java:748)
>>> 2018-04-16 12:53:36,300 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
>>> 2018-04-16 12:53:36,311 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
>>> 2018-04-16 12:53:36,314 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:46869
>>>
>>>
>>>
>>>
>>
>
Re: Testing on Flink 1.5
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Amit,
Thank you for the follow up. What you describe sounds like a bug but I am
not
able to reproduce it. Can you open an issue in Jira with an outline of your
code
and how you submit the job?
> Could you also recommend us the best practice in FLIP6, should we use
YARN session or submit jobs in non-detached mode?
This should be independent of FLIP-6. We are aiming at backwards
compatibility
but apparently there are still things that require polishing.
Best,
Gary
On Thu, Apr 19, 2018 at 12:49 PM, Amit Jain <aj...@gmail.com> wrote:
>
> Hi Gary,
>
> We found the underlying issue with the following problem.
> Few of our jobs are stuck with logs [1], these jobs are only able to
allocate JM and couldn't get any TM, however, there are ample resource on
our cluster.
>
> We are running ETL merge job here. In this job, we first find new deltas
and if there is no delta detected then we make exit without actually
executing the job. I think this is the reason we see no TM allocation is
happening.
>
> I believe in above case (non-detached mode) we should mark the submitted
application as complete compare to running. Please share your thoughts on
this.
> Should I log this improvement in JIRA?
>
> Could you also recommend us the best practice in FLIP6, should we use
YARN session or submit jobs in non-detached mode?
>
> --
> Thanks,
> Amit
>
Re: Testing on Flink 1.5
Posted by Amit Jain <aj...@gmail.com>.
Hi Gary,
We found the underlying issue with the following problem.
Few of our jobs are stuck with logs [1], these jobs are only able to
allocate JM and couldn't get any TM, however, there are ample resource on
our cluster.
We are running ETL merge job here. In this job, we first find new deltas
and if there is no delta detected then we make exit without actually
executing the job. I think this is the reason we see no TM allocation is
happening.
I believe in above case (non-detached mode) we should mark the submitted
application as complete compare to running. Please share your thoughts on
this.
Should I log this improvement in JIRA?
Could you also recommend us the best practice in FLIP6, should we use YARN
session or submit jobs in non-detached mode?
--
Thanks,
Amit
Re: Testing on Flink 1.5
Posted by Amit Jain <aj...@gmail.com>.
Hi Gary,
Please find my reply in place. We have also logged
https://issues.apache.org/jira/browse/FLINK-9207, please have a look on
this.
--
Thanks,
Amit
On Tue, Apr 17, 2018 at 9:10 PM, Gary Yao <ga...@data-artisans.com> wrote:
> Hi Amit,
>
> Thank you for testing the 1.5 Release Candidate. Find my answers below.
>
> > Few of our jobs are stuck with logs [1], these jobs are only able to
> allocate JM and couldn't get any TM, however, there are ample resource on
> our cluster.
>
> In the logs you attached I cannot see that a job was submitted. If a job is
> submitted, the dispatcher should log: "Submitting job <job_id>
> (<job_name>)"
> [1]. Could it be that the attached logs are incomplete, or the client
> already
> fails to submit the job? Can you also attach the client logs?
>
> I agree with you, we should have these logs however it is not happening.
I'll work on it further and provide more info, right now, I can provide
logs which got generated later.
2018-04-16 04:36:43,747 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Dispatcher akka.tcp://flink@ip-10-0-22-43.ap-southeast-1.compute.internal:39329/user/dispatcher
was granted leadership with fencing token
00000000000000000000000000000000
2018-04-16 04:36:43,747 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher -
Recovering all persisted jobs.
2018-04-16 17:21:37,800 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:21:42,803 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:21:47,805 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:21:52,806 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:21:57,807 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:02,808 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:07,810 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:12,812 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:17,813 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:22,814 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:27,816 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:32,817 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:37,818 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:42,821 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:47,822 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:52,825 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:22:57,826 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:23:02,828 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:23:07,829 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
2018-04-16 17:23:12,832 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl -
Updating with new AMRMToken
> > All completed application on YARN are marked as KILLED, instead of
> FINISHED. It creates problem on tracking jobs as we require to navigate on
> each application logs and also history server is not logging any completed
> application.
>
> I was able to reproduce this: In non-detached mode, the State and
> FinalStatus
> are set to KILLED. In detached mode, the Status and FinalStatus are set
> correctly. I have re-opened this ticket:
> https://issues.apache.org/jira/browse/FLINK-8900
>
> You are also right about the History Server, which is currently not
> implemented
> for FLIP-6. Here is the ticket for reference:
> https://issues.apache.org/jira/browse/FLINK-9194
>
>
> > We are running our system on AWS EMR and using CORE nodes at its bare
> minimum, we see with new paradigm of application submission we are hitting
> Out of space error in our HDFS Data Node. Do you think, we should have
> configuration to automatically evict these .flink/ files?
>
> This seems to be a regression. In Flink 1.4., the files are cleaned up
> automatically. I opened this ticket:
> https://issues.apache.org/jira/browse/FLINK-9196
>
>
> > Lastly, we are also observing Akka.ask.timeout error for few of our jobs
> (JM's logs[2]), we tried to increase this parameter to 2 min from default
> 10 s and still observe the same.
>
> Does this always happen for the same jobs? I cannot reproduce this.
>
> Yes, it happening for same set of jobs. Specifically, these jobs are
having multiple S3 objects to work with. I tried increasing
akka.ask.timeout which is not getting reflected on JM's log also and same
has been experienced while running the jobs.
I also tried one of these jobs on empty cluster(EMR TASK 15 node, r3.xlarge
type) and still face same.
Best,
> Gary
>
> [1] https://github.com/apache/flink/blob/master/flink-
> runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#
> L235
>
>
> On Tue, Apr 17, 2018 at 9:35 AM, Amit Jain <aj...@gmail.com> wrote:
>
>> Hi,
>>
>>
>> We have started testing our workload on Flink 1.5.0 (Commit: 2af481a) and faced few issues. Here, we are using YARN as cluster manager.
>>
>>
>> - Few of our jobs are stuck with logs [1], these jobs are only able to allocate JM and couldn't get any TM, however, there are ample resource on our cluster.
>> - All completed application on YARN are marked as KILLED, instead of FINISHED. It creates problem on tracking jobs as we require to navigate on each application logs and also history server is not logging any completed application.
>> - We are running our system on AWS EMR and using CORE nodes at its bare minimum, we see with new paradigm of application submission we are hitting Out of space error in our HDFS Data Node. Do you think, we should have configuration to automatically evict these .flink/ files?
>> - Lastly, we are also observing Akka.ask.timeout error for few of our jobs (JM's logs[2]), we tried to increase this parameter to 2 min from default 10 s and still observe the same.
>>
>> Please help us guide through these issues.
>>
>> --
>> Thanks,
>> Amit
>>
>> [1] JM logs where jobs are not even started
>>
>> 2018-04-17 04:26:16,165 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>> 2018-04-17 04:26:16,166 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a, Date:02.04.2018 @ 15:17:35 UTC)
>> 2018-04-17 04:26:16,166 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hadoop
>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
>> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-openjdk
>> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
>> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
>> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml
>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
>> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
>> 2018-04-17 04:26:16,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>> 2018-04-17 04:26:16,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
>> 2018-04-17 04:26:16,851 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hadoop Yarn client user obtainer: hadoop
>> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///completed-jobs/
>> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, s3a
>> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1523726493647_0503
>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.5
>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, true
>> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, -1
>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.watch.heartbeat.pause, 600 s
>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, on
>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.fraction, 0.1
>> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 2 min
>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 10772
>> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs:///completed-jobs/
>> 2018-04-17 04:26:16,874 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503
>> 2018-04-17 04:26:16,896 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint.
>> 2018-04-17 04:26:16,897 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
>> 2018-04-17 04:26:17,014 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hadoop (auth:SIMPLE)
>> 2018-04-17 04:26:17,034 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
>> 2018-04-17 04:26:17,041 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at ip-10-0-22-186.ap-southeast-1.compute.internal:38669
>> 2018-04-17 04:26:17,671 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
>> 2018-04-17 04:26:17,794 INFO akka.remote.Remoting - Starting remoting
>> 2018-04-17 04:26:17,937 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669]
>> 2018-04-17 04:26:17,948 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669
>> 2018-04-17 04:26:17,980 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-bca451c7-977d-425a-9e8f-4fe324fa8623
>> 2018-04-17 04:26:17,983 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:44217 - max concurrent requests: 50 - max backlog: 1000
>> 2018-04-17 04:26:18,009 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
>> 2018-04-17 04:26:18,015 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/executionGraphStore-28bf6537-0bbf-486c-a346-084bd7517d7d, expiration time 3600000, maximum cache size 52428800 bytes.
>> 2018-04-17 04:26:18,047 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-f1de0cad-77b0-4bbd-b218-63e32b10bf6a
>> 2018-04-17 04:26:18,057 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
>> 2018-04-17 04:26:18,059 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload for file uploads.
>> 2018-04-17 04:26:18,063 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
>> 2018-04-17 04:26:18,386 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
>> 2018-04-17 04:26:18,386 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.out
>> 2018-04-17 04:26:18,454 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
>> 2018-04-17 04:26:18,538 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at ip-10-0-22-186.ap-southeast-1.compute.internal:44431
>> 2018-04-17 04:26:18,542 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://ip-10-0-22-186.ap-southeast-1.compute.internal:44431 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
>> 2018-04-17 04:26:18,558 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
>> 2018-04-17 04:26:18,713 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
>> 2018-04-17 04:26:18,739 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
>> 2018-04-17 04:26:18,740 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
>> 2018-04-17 04:26:18,769 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
>> 2018-04-17 04:26:19,083 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]).
>> 2018-04-17 04:26:19,086 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
>> 2018-04-17 04:26:19,101 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
>> 2018-04-17 04:26:19,101 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
>>
>>
>>
>>
>> [2] JM's logs
>>
>> 2018-04-16 12:53:15,825 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>> 2018-04-16 12:53:15,827 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a, Date:02.04.2018 @ 15:17:35 UTC)
>> 2018-04-16 12:53:15,827 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
>> 2018-04-16 12:53:16,798 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hadoop
>> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
>> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
>> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-openjdk
>> 2018-04-16 12:53:16,800 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
>> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
>> 2018-04-16 12:53:16,803 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
>> 2018-04-16 12:53:16,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
>> 2018-04-16 12:53:16,809 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hadoop Yarn client user obtainer: hadoop
>> 2018-04-16 12:53:16,813 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///completed-jobs/
>> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, s3a
>> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
>> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1523726493647_0302
>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.5
>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, true
>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, -1
>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.watch.heartbeat.pause, 600 s
>> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, on
>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.fraction, 0.1
>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
>> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 9216
>> 2018-04-16 12:53:16,817 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs:///completed-jobs/
>> 2018-04-16 12:53:16,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302
>> 2018-04-16 12:53:16,859 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint.
>> 2018-04-16 12:53:16,859 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
>> 2018-04-16 12:53:17,036 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hadoop (auth:SIMPLE)
>> 2018-04-16 12:53:17,071 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
>> 2018-04-16 12:53:17,081 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at ip-10-0-22-186.ap-southeast-1.compute.internal:36411
>> 2018-04-16 12:53:18,022 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
>> 2018-04-16 12:53:18,254 INFO akka.remote.Remoting - Starting remoting
>> 2018-04-16 12:53:18,461 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411]
>> 2018-04-16 12:53:18,488 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411
>> 2018-04-16 12:53:18,559 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-9ec0a59f-c03a-446f-a841-60a65892b383
>> 2018-04-16 12:53:18,566 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:46869 - max concurrent requests: 50 - max backlog: 1000
>> 2018-04-16 12:53:18,632 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
>> 2018-04-16 12:53:18,638 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/executionGraphStore-ee071b9a-ad86-4b36-860c-6501f509ecf9, expiration time 3600000, maximum cache size 52428800 bytes.
>> 2018-04-16 12:53:18,726 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-c15647ee-31c1-451d-84d2-729a67e065d8
>> 2018-04-16 12:53:18,750 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
>> 2018-04-16 12:53:18,757 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload for file uploads.
>> 2018-04-16 12:53:18,765 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
>> 2018-04-16 12:53:19,361 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
>> 2018-04-16 12:53:19,362 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.out
>> 2018-04-16 12:53:19,442 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
>> 2018-04-16 12:53:19,589 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at ip-10-0-22-186.ap-southeast-1.compute.internal:45697
>> 2018-04-16 12:53:19,589 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://ip-10-0-22-186.ap-southeast-1.compute.internal:45697 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
>> 2018-04-16 12:53:19,614 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
>> 2018-04-16 12:53:19,979 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
>> 2018-04-16 12:53:20,049 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
>> 2018-04-16 12:53:20,050 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
>> 2018-04-16 12:53:20,101 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
>> 2018-04-16 12:53:20,649 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]).
>> 2018-04-16 12:53:20,655 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
>> 2018-04-16 12:53:20,706 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
>> 2018-04-16 12:53:20,706 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
>> 2018-04-16 12:53:26,207 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 58a1b149b5b69de66bb931836c459d2b (mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job).
>> 2018-04-16 12:53:26,223 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
>> 2018-04-16 12:53:26,237 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
>> 2018-04-16 12:53:26,279 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy NoRestartStrategy for mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
>> 2018-04-16 12:53:26,285 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/820d207e-fddd-4e24-b327-1f3c1cb25024 .
>> 2018-04-16 12:53:26,320 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
>> 2018-04-16 12:53:26,386 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
>> 2018-04-16 12:53:26,671 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 284 ms.
>> 2018-04-16 12:53:36,204 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Implementation error: Unhandled exception.
>> akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#1710592138]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
>> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
>> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
>> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
>> at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
>> at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
>> at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
>> at java.lang.Thread.run(Thread.java:748)
>> 2018-04-16 12:53:36,300 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
>> 2018-04-16 12:53:36,311 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
>> 2018-04-16 12:53:36,314 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:46869
>>
>>
>>
>>
>
Re: Testing on Flink 1.5
Posted by Gary Yao <ga...@data-artisans.com>.
Hi Amit,
Thank you for testing the 1.5 Release Candidate. Find my answers below.
> Few of our jobs are stuck with logs [1], these jobs are only able to
allocate JM and couldn't get any TM, however, there are ample resource on
our cluster.
In the logs you attached I cannot see that a job was submitted. If a job is
submitted, the dispatcher should log: "Submitting job <job_id> (<job_name>)"
[1]. Could it be that the attached logs are incomplete, or the client
already
fails to submit the job? Can you also attach the client logs?
> All completed application on YARN are marked as KILLED, instead of
FINISHED. It creates problem on tracking jobs as we require to navigate on
each application logs and also history server is not logging any completed
application.
I was able to reproduce this: In non-detached mode, the State and
FinalStatus
are set to KILLED. In detached mode, the Status and FinalStatus are set
correctly. I have re-opened this ticket:
https://issues.apache.org/jira/browse/FLINK-8900
You are also right about the History Server, which is currently not
implemented
for FLIP-6. Here is the ticket for reference:
https://issues.apache.org/jira/browse/FLINK-9194
> We are running our system on AWS EMR and using CORE nodes at its bare
minimum, we see with new paradigm of application submission we are hitting
Out of space error in our HDFS Data Node. Do you think, we should have
configuration to automatically evict these .flink/ files?
This seems to be a regression. In Flink 1.4., the files are cleaned up
automatically. I opened this ticket:
https://issues.apache.org/jira/browse/FLINK-9196
> Lastly, we are also observing Akka.ask.timeout error for few of our jobs
(JM's logs[2]), we tried to increase this parameter to 2 min from default
10 s and still observe the same.
Does this always happen for the same jobs? I cannot reproduce this.
Best,
Gary
[1]
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L235
On Tue, Apr 17, 2018 at 9:35 AM, Amit Jain <aj...@gmail.com> wrote:
> Hi,
>
>
> We have started testing our workload on Flink 1.5.0 (Commit: 2af481a) and faced few issues. Here, we are using YARN as cluster manager.
>
>
> - Few of our jobs are stuck with logs [1], these jobs are only able to allocate JM and couldn't get any TM, however, there are ample resource on our cluster.
> - All completed application on YARN are marked as KILLED, instead of FINISHED. It creates problem on tracking jobs as we require to navigate on each application logs and also history server is not logging any completed application.
> - We are running our system on AWS EMR and using CORE nodes at its bare minimum, we see with new paradigm of application submission we are hitting Out of space error in our HDFS Data Node. Do you think, we should have configuration to automatically evict these .flink/ files?
> - Lastly, we are also observing Akka.ask.timeout error for few of our jobs (JM's logs[2]), we tried to increase this parameter to 2 min from default 10 s and still observe the same.
>
> Please help us guide through these issues.
>
> --
> Thanks,
> Amit
>
> [1] JM logs where jobs are not even started
>
> 2018-04-17 04:26:16,165 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
> 2018-04-17 04:26:16,166 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a, Date:02.04.2018 @ 15:17:35 UTC)
> 2018-04-17 04:26:16,166 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hadoop
> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
> 2018-04-17 04:26:16,843 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
> 2018-04-17 04:26:16,844 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml
> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
> 2018-04-17 04:26:16,845 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
> 2018-04-17 04:26:16,846 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
> 2018-04-17 04:26:16,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
> 2018-04-17 04:26:16,851 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hadoop Yarn client user obtainer: hadoop
> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///completed-jobs/
> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, s3a
> 2018-04-17 04:26:16,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1523726493647_0503
> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.5
> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, true
> 2018-04-17 04:26:16,855 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, -1
> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.watch.heartbeat.pause, 600 s
> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, on
> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.fraction, 0.1
> 2018-04-17 04:26:16,856 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 2 min
> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 10772
> 2018-04-17 04:26:16,857 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs:///completed-jobs/
> 2018-04-17 04:26:16,874 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503
> 2018-04-17 04:26:16,896 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint.
> 2018-04-17 04:26:16,897 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
> 2018-04-17 04:26:17,014 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hadoop (auth:SIMPLE)
> 2018-04-17 04:26:17,034 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
> 2018-04-17 04:26:17,041 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at ip-10-0-22-186.ap-southeast-1.compute.internal:38669
> 2018-04-17 04:26:17,671 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
> 2018-04-17 04:26:17,794 INFO akka.remote.Remoting - Starting remoting
> 2018-04-17 04:26:17,937 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669]
> 2018-04-17 04:26:17,948 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669
> 2018-04-17 04:26:17,980 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-bca451c7-977d-425a-9e8f-4fe324fa8623
> 2018-04-17 04:26:17,983 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:44217 - max concurrent requests: 50 - max backlog: 1000
> 2018-04-17 04:26:18,009 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
> 2018-04-17 04:26:18,015 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/executionGraphStore-28bf6537-0bbf-486c-a346-084bd7517d7d, expiration time 3600000, maximum cache size 52428800 bytes.
> 2018-04-17 04:26:18,047 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0503/blobStore-f1de0cad-77b0-4bbd-b218-63e32b10bf6a
> 2018-04-17 04:26:18,057 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
> 2018-04-17 04:26:18,059 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-1fd19ea7-ae19-4880-8787-92c5ec6ee509/flink-web-upload for file uploads.
> 2018-04-17 04:26:18,063 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
> 2018-04-17 04:26:18,386 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.log
> 2018-04-17 04:26:18,386 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /var/log/hadoop-yarn/containers/application_1523726493647_0503/container_1523726493647_0503_01_000001/jobmanager.out
> 2018-04-17 04:26:18,454 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
> 2018-04-17 04:26:18,538 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at ip-10-0-22-186.ap-southeast-1.compute.internal:44431
> 2018-04-17 04:26:18,542 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://ip-10-0-22-186.ap-southeast-1.compute.internal:44431 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
> 2018-04-17 04:26:18,558 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
> 2018-04-17 04:26:18,713 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
> 2018-04-17 04:26:18,739 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
> 2018-04-17 04:26:18,740 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
> 2018-04-17 04:26:18,769 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
> 2018-04-17 04:26:19,083 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]).
> 2018-04-17 04:26:19,086 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
> 2018-04-17 04:26:19,101 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:38669/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
> 2018-04-17 04:26:19,101 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
>
>
>
>
> [2] JM's logs
>
> 2018-04-16 12:53:15,825 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
> 2018-04-16 12:53:15,827 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:2af481a, Date:02.04.2018 @ 15:17:35 UTC)
> 2018-04-16 12:53:15,827 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
> 2018-04-16 12:53:16,798 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hadoop
> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
> 2018-04-16 12:53:16,799 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-04-16 12:53:16,800 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
> 2018-04-16 12:53:16,801 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/flink-yarn_2.11-1.5-SNAPSHOT.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-annotations-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-streaming.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/hadoop/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archives.jar:/usr/lib/hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/hadoop/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-openstack.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-azure-datalake.jar:/usr/lib/hadoop/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop/hadoop-sls.jar:/usr/lib/hadoop/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/json-smart-1.1.1.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jsch-0.1.54.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-native-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.8.3-amzn-0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/json-smart-1.1.1.jar:/usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-core-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/jmespath-java-1.11.267.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.jar:/usr/lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/jcip-annotations-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-auth-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-ant-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/azure-data-lake-store-sdk-2.2.3.jar:/usr/lib/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/jsch-0.1.54.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-streaming-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-archives.jar:/usr/lib/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/jackson-databind-2.6.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-aws.jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-sls-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/nimbus-jose-jwt-3.9.jar:/usr/lib/hadoop-mapreduce/hadoop-aws-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/joda-time-2.9.4.jar:/usr/lib/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/htrace-core4-4.0.1-incubating.jar:/usr/lib/hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/hadoop-extras-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.8.3-amzn-0-tests.jar:/usr/lib/hadoop-mapreduce/hadoop-azure-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/okio-1.4.0.jar:/usr/lib/hadoop-mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/azure-storage-2.2.0.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-s3-1.11.267.jar:/usr/lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/okhttp-2.4.0.jar:/usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/lib/hadoop-mapreduce/hadoop-distcp-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/aws-java-sdk-kms-1.11.267.jar:/usr/lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.8.3-amzn-0.jar:/usr/lib/hadoop-mapreduce/jetty-sslengine-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-sharedcachemanager-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-proxy-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-registry-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.8.3-amzn-0.jar:/usr/lib/hadoop-yarn/lib/curator-test-2.7.1.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-math-2.2.jar:/usr/lib/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.21.0.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.267.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar:/usr/lib/spark/yarn/lib/datanucleus-core.jar:/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.1.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediaconvert-1.11.267.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appsync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastore-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-comprehend-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workmail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscalingplans-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediapackage-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-serverlessapplicationrepository-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mediastoredata-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloud9-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iotjobsdataplane-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-medialive-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pricing-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-guardduty-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicediscovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroups-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-translate-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-alexaforbusiness-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemaker-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costexplorer-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sagemakerruntime-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mq-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesisvideo-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.267.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.267.jar
> 2018-04-16 12:53:16,803 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
> 2018-04-16 12:53:16,805 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
> 2018-04-16 12:53:16,809 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: hadoop Yarn client user obtainer: hadoop
> 2018-04-16 12:53:16,813 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///completed-jobs/
> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, s3a
> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
> 2018-04-16 12:53:16,814 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1523726493647_0302
> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.heap-cutoff-ratio, 0.5
> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, true
> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.maximum-failed-containers, -1
> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.watch.heartbeat.pause, 600 s
> 2018-04-16 12:53:16,815 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, on
> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.fraction, 0.1
> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
> 2018-04-16 12:53:16,816 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 9216
> 2018-04-16 12:53:16,817 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs:///completed-jobs/
> 2018-04-16 12:53:16,833 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302
> 2018-04-16 12:53:16,859 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint.
> 2018-04-16 12:53:16,859 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
> 2018-04-16 12:53:17,036 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hadoop (auth:SIMPLE)
> 2018-04-16 12:53:17,071 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
> 2018-04-16 12:53:17,081 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at ip-10-0-22-186.ap-southeast-1.compute.internal:36411
> 2018-04-16 12:53:18,022 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
> 2018-04-16 12:53:18,254 INFO akka.remote.Remoting - Starting remoting
> 2018-04-16 12:53:18,461 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411]
> 2018-04-16 12:53:18,488 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411
> 2018-04-16 12:53:18,559 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-9ec0a59f-c03a-446f-a841-60a65892b383
> 2018-04-16 12:53:18,566 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:46869 - max concurrent requests: 50 - max backlog: 1000
> 2018-04-16 12:53:18,632 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
> 2018-04-16 12:53:18,638 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/executionGraphStore-ee071b9a-ad86-4b36-860c-6501f509ecf9, expiration time 3600000, maximum cache size 52428800 bytes.
> 2018-04-16 12:53:18,726 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /mnt/yarn/usercache/hadoop/appcache/application_1523726493647_0302/blobStore-c15647ee-31c1-451d-84d2-729a67e065d8
> 2018-04-16 12:53:18,750 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
> 2018-04-16 12:53:18,757 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-6d14c830-9074-4a2f-8e1e-7344545e34e5/flink-web-upload for file uploads.
> 2018-04-16 12:53:18,765 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.
> 2018-04-16 12:53:19,361 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.log
> 2018-04-16 12:53:19,362 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /var/log/hadoop-yarn/containers/application_1523726493647_0302/container_1523726493647_0302_01_000001/jobmanager.out
> 2018-04-16 12:53:19,442 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started.
> 2018-04-16 12:53:19,589 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at ip-10-0-22-186.ap-southeast-1.compute.internal:45697
> 2018-04-16 12:53:19,589 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://ip-10-0-22-186.ap-southeast-1.compute.internal:45697 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
> 2018-04-16 12:53:19,614 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
> 2018-04-16 12:53:19,979 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
> 2018-04-16 12:53:20,049 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
> 2018-04-16 12:53:20,050 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.
> 2018-04-16 12:53:20,101 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at ip-10-0-22-169.ap-southeast-1.compute.internal/10.0.22.169:8030
> 2018-04-16 12:53:20,649 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]).
> 2018-04-16 12:53:20,655 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
> 2018-04-16 12:53:20,706 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@ip-10-0-22-186.ap-southeast-1.compute.internal:36411/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000
> 2018-04-16 12:53:20,706 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
> 2018-04-16 12:53:26,207 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 58a1b149b5b69de66bb931836c459d2b (mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job).
> 2018-04-16 12:53:26,223 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
> 2018-04-16 12:53:26,237 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
> 2018-04-16 12:53:26,279 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy NoRestartStrategy for mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
> 2018-04-16 12:53:26,285 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/820d207e-fddd-4e24-b327-1f3c1cb25024 .
> 2018-04-16 12:53:26,320 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
> 2018-04-16 12:53:26,386 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job mongo_notificationDB.userdeviceidentifiers_userdeviceidentifiers_merge_job (58a1b149b5b69de66bb931836c459d2b).
> 2018-04-16 12:53:26,671 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 284 ms.
> 2018-04-16 12:53:36,204 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Implementation error: Unhandled exception.
> akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#1710592138]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-16 12:53:36,300 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2018-04-16 12:53:36,311 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
> 2018-04-16 12:53:36,314 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:46869
>
>
>
>