You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jeff Crane <je...@yahoo.com> on 2019/03/23 16:36:42 UTC
Install 1.7.2 on EC2 - No task slots - 2019
When downloading the latest 1.7.2 and extracting to (a free) AMZ EC2, the daemon (./bin/start-cluster.sh) reports 0 task managers, 0 task slots, and 0 avails, out of the box. All jobs fail. All ports are open to all traffic.
Can anyone tell me what I missed?
Re: Install 1.7.2 on EC2 - No task slots - 2019
Posted by Jeff Crane <je...@yahoo.com>.
There are 2 out files and a log file. I will describe the steps to take below, but I have not found anything to indicate there's a problem when it cannot allocate any resources (but otherwise runs).
----------start default free tier aws instanceedit security group to allow 8081 incomingsudo yum install java# wget http://ftp.naz.com/apache/flink/flink-1.7.2/flink-1.7.2-bin-scala_2.12.tgz# tar xzvf flink-1.7.2-bin-scala_2.12.tgz# cd flink-1.7.2/# ./bin/start-cluster.shWeb interface shows 0,0,0 for resources.While there are 2 other .out files, I dunno if those are important.Here is the log of the only .log -flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.log
2019-03-26 15:43:12,488 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------2019-03-26 15:43:12,489 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: 1.7.2, Rev:ceba8af, Date:11.02.2019 @ 14:17:09 UTC)2019-03-26 15:43:12,489 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: ec2-user2019-03-26 15:43:12,489 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found>2019-03-26 15:43:12,489 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.191-b122019-03-26 15:43:12,490 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 989 MiBytes2019-03-26 15:43:12,490 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: (not set)2019-03-26 15:43:12,490 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available2019-03-26 15:43:12,490 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:2019-03-26 15:43:12,490 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m2019-03-26 15:43:12,491 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m2019-03-26 15:43:12,491 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/home/ec2-user/flink-1.7.2/log/flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.log2019-03-26 15:43:12,491 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/home/ec2-user/flink-1.7.2/conf/log4j.properties2019-03-26 15:43:12,491 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/home/ec2-user/flink-1.7.2/conf/logback.xml2019-03-26 15:43:12,491 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments:2019-03-26 15:43:12,491 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir2019-03-26 15:43:12,492 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /home/ec2-user/flink-1.7.2/conf2019-03-26 15:43:12,492 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode2019-03-26 15:43:12,492 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster2019-03-26 15:43:12,492 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /home/ec2-user/flink-1.7.2/lib/flink-python_2.12-1.7.2.jar:/home/ec2-user/flink-1.7.2/lib/log4j-1.2.17.jar:/home/ec2-user/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/home/ec2-user/flink-1.7.2/lib/flink-dist_2.12-1.7.2.jar:::2019-03-26 15:43:12,492 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------2019-03-26 15:43:12,495 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]2019-03-26 15:43:12,531 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost2019-03-26 15:43:12,533 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 61232019-03-26 15:43:12,533 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m2019-03-26 15:43:12,534 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m2019-03-26 15:43:12,534 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 12019-03-26 15:43:12,534 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 12019-03-26 15:43:12,535 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 80812019-03-26 15:43:12,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint.2019-03-26 15:43:12,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.2019-03-26 15:43:12,714 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.2019-03-26 15:43:12,730 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context.2019-03-26 15:43:12,742 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.2019-03-26 15:43:12,761 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.2019-03-26 15:43:12,762 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.2019-03-26 15:43:13,432 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at localhost:61232019-03-26 15:43:14,281 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started2019-03-26 15:43:14,418 INFO akka.remote.Remoting - Starting remoting2019-03-26 15:43:14,669 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@localhost:6123]2019-03-26 15:43:14,692 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@localhost:61232019-03-26 15:43:14,711 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'2019-03-26 15:43:14,724 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-cbac7ece-9cb3-4bf8-a225-26fe4136067d2019-03-26 15:43:14,728 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:45861 - max concurrent requests: 50 - max backlog: 10002019-03-26 15:43:14,760 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.2019-03-26 15:43:14,762 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at localhost:02019-03-26 15:43:14,813 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started2019-03-26 15:43:14,829 INFO akka.remote.Remoting - Starting remoting2019-03-26 15:43:14,849 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink-metrics@localhost:381812019-03-26 15:43:14,853 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@localhost:38181]2019-03-26 15:43:14,861 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-99cbce6e-06d9-4159-b68e-e461b31e41cf, expiration time 3600000, maximum cache size 52428800 bytes.2019-03-26 15:43:14,898 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-d8d4eb58-74fd-4f6f-9b9b-e71a492286522019-03-26 15:43:14,915 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'2019-03-26 15:43:14,917 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-027518bf-4744-4f65-9bc0-c0d8e23853b0/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.2019-03-26 15:43:14,918 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-027518bf-4744-4f65-9bc0-c0d8e23853b0/flink-web-upload for file uploads.2019-03-26 15:43:14,926 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint.2019-03-26 15:43:15,286 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /home/ec2-user/flink-1.7.2/log/flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.log2019-03-26 15:43:15,286 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /home/ec2-user/flink-1.7.2/log/flink-ec2-user-standalonesession-0-ip-10-0-0-204.us-west-1.compute.internal.out2019-03-26 15:43:15,504 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at localhost:80812019-03-26 15:43:15,504 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://localhost:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-0000000000002019-03-26 15:43:15,504 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://localhost:8081.2019-03-26 15:43:15,670 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .2019-03-26 15:43:15,713 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .2019-03-26 15:43:15,744 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was granted leadership with fencing token 000000000000000000000000000000002019-03-26 15:43:15,745 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager.2019-03-26 15:43:15,781 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-0000000000002019-03-26 15:43:15,783 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs.
----------On Monday, March 25, 2019, 1:23:45 AM PDT, Konstantin Knauf <ko...@ververica.com> wrote:
Hi Jeff,
do you see any log files in the log directory of your Flink installation directory? If so, please share them.
Cheers,
Konstantin
Re: Install 1.7.2 on EC2 - No task slots - 2019
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Jeff,
do you see any log files in the log directory of your Flink installation
directory? If so, please share them.
Cheers,
Konstantin
On Sat, Mar 23, 2019 at 5:36 PM Jeff Crane <je...@yahoo.com> wrote:
> When downloading the latest 1.7.2 and extracting to (a free) AMZ EC2, the
> daemon (./bin/start-cluster.sh) reports 0 task managers, 0 task slots, and
> 0 avails, out of the box. All jobs fail. All ports are open to all traffic.
>
> Can anyone tell me what I missed?
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen