You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "r. r." <ro...@abv.bg> on 2017/10/02 12:10:35 UTC
Re: how many 'run -c' commands to start?
Thanks, Chesnay, that was indeed the problem.
It also explains why -p5 was not working for me from the cmdline
Best regards
Robert
>-------- Оригинално писмо --------
>От: Chesnay Schepler chesnay@apache.org
>Относно: Re: how many 'run -c' commands to start?
>До: "r. r." <ro...@abv.bg>
>Изпратено на: 01.10.2017 00:58
> The order in which you pass program arguments is quite important. Can
> you try again putting --detached directly after run?
> I.e. flink run --detached -c ...
>
> The reason being that arguments after the jar are treated as arguments
> for the user-program, and not as arguments for the job execution.
>
> On 29.09.2017 17:23, r. r. wrote:
> > Sure, here is the cmdline output:
> >
> > flink-1.3.2/bin/flink run -c com.corp.flink.KafkaJob quickstart.jar --topic InputQ --bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181 --group.id Consumers -p 5 --detached
> > Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
> > Using address localhost:6123 to connect to JobManager.
> > JobManager web interface address http://localhost:8081
> > Starting execution of program
> > Submitting job with JobID: 5cc74547361cec2ff9874764dac9ee91. Waiting for job completion.
> > Connected to JobManager at Actor[akka.tcp://flink@localhost:6123/user/jobmanager#-100905913] with leader session id 00000000-0000-0000-0000-000000000000.
> > 09/28/2017 16:30:40 Job execution switched to status RUNNING.
> > 09/28/2017 16:30:40 Source: Custom Source -> Map(1/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Source: Custom Source -> Map(2/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Source: Custom Source -> Map(3/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Source: Custom Source -> Map(4/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Source: Custom Source -> Map(5/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Sink: Unnamed(1/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Sink: Unnamed(2/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Sink: Unnamed(3/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Sink: Unnamed(4/5) switched to SCHEDULED
> > 09/28/2017 16:30:40 Sink: Unnamed(5/5) switched to SCHEDULED
> > ...
> >
> > I thought --detach will put the process in the background, and give me back the cmdline, but maybe I got the meaning behind this option wrong?
> >
> > Thank you!
> >
> >
> >
> >
> >
> >
> >
> > >-------- Оригинално писмо --------
> >
> > >От: Chesnay Schepler chesnay@apache.org
> >
> > >Относно: Re: how many 'run -c' commands to start?
> >
> > >До: user@flink.apache.org
> >
> > >Изпратено на: 29.09.2017 18:01
> >
> >
> >
> >
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> The only nodes that matter are those on which the Flink processes, i.e
> >
> >> Job- and TaskManagers, are being run.
> >
> >>
> >
> >> To prevent a JobManager node failure from causing the job to fail you have to look into an
> >
> >> HA setup.
> >
> >> (The jobmanager is responsible for distributing/coordinating work)
> >
> >>
> >
> >> In case of a TaskManager node failure the job will fail and
> >
> >> restart, provided that enough TaskManagers are
> >
> >> still alive to satisfy the resource requirements of the job.
> >
> >>
> >
> >>
> >
> >> Can you elaborate a bit more what happened when you used the --detached param?
> >
> >>
> >
> >> On 28.09.2017 16:33, r. r. wrote:
> >
> >>
> >
> >>
> >
> >>
> >
> >> Thank you, Chesnay
> >
> >> to make sure - should the node where the job has been submitted goes down, the processing will continue, I hope?
> >
> >> Do I need to ensure this by configuration?
> >
> >
> >> btw I added --detached param to the run cmd, but it didn't go into background process as I would've expected. Am I guessing wrong?
> >
> >
> >> Thanks!
> >
> >> Rob
> >
> >
> >> >-------- Оригинално писмо --------
> >
> >
> >> >От: Chesnay Schepler chesnay@apache.org
> >
> >
> >> >Относно: Re: how many 'run -c' commands to start?
> >
> >
> >> >До: user@flink.apache.org
> >
> >
> >> >Изпратено на: 28.09.2017 15:05
> >
> >
> >>
> >
> >>
> >
> >>
> >
> >> Hi!
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> Given a Flink cluster, you would only call `flink run ...` to submit a
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> job once; for simplicity i would submit it on the node where you started
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> the cluster. Flink will automatically distribute job across the cluster,
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> in smaller independent parts known as Tasks.
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> Regards,
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> Chesnay
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> On 28.09.2017 08:31, r. r. wrote:
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> Hello
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> I successfully ran a job with 'flink run -c', but this is for the local
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> setup.
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> How should i proceed with a cluster? Will flink automagically instantiate
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> the job on all servers - i hope i don't have to start 'flink run -c' on all
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> machines.
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> New to flink and bigdata, so sorry for the probably silly question
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> Thanks!
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >> Rob
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>
> >
> >>