You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Giacomo Licari <gi...@gmail.com> on 2015/04/15 11:22:10 UTC

Google Compute Engine Cluster

Hi guys,
I'm trying to setup a simple Flink cluster on Google Compute Engine.

I'm running 3 nodes (1 master, 2 workers).

On master node I set up ssh key and moved it into authorized_keys.

When I try to copy my key to each worker node I got Permission denied
(publickey).

Someone had the same problem?
Thanks for your help.

Re: Google Compute Engine Cluster

Posted by Giacomo Licari <gi...@gmail.com>.

Thank you Max,
everything works well.

On Wed, Apr 15, 2015 at 1:04 PM, Maximilian Michels <mx...@apache.org> wrote:

> Sure, just ran an included example.
>
> ./bin/flink run examples/*WordCount.jar
>
> By default, it will use a really small example DataSet but it should be
> enough to verify the cluster setup.
>
> You're welcome.
>
> Best,
> Max
>
> On Wed, Apr 15, 2015 at 12:24 PM, Giacomo Licari <giacomo.licari@gmail.com
> > wrote:
>
>> Thanks a lot Max,
>> now I can set up ssh keys.
>>
>> I started my cluster correctly with ./start-cluster.sh, do you use an
>> example program to test if everything works fine?
>>
>> Thanks again,
>> Giacomo
>>
>> On Wed, Apr 15, 2015 at 11:48 AM, Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Hi Giacomo,
>>>
>>> Have you tried setting up Flink on GCE using bdutil? It is very easy:
>>> http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html
>>>
>>> If you don't want to use bdutil:
>>> I'm assuming you want to start Flink using the "start_cluster.sh" script
>>> which uses ssh to start the task managers on the workers. By default,
>>> instances on GCE are not authenticated via ssh with each other. Using the
>>> gcloud tool, you can configure an ssh key on all hosts (gcloud compute
>>> config-ssh) which lets you ssh into all instances. To enable public key ssh
>>> authentication between your GCE nodes, I'd advise to create a new key on
>>> your machine for just this purpose and then distribute the key (public and
>>> private) to the master. Then add the public key to all worker nodes's
>>> "authorized_keys" file from your machine.
>>>
>>>
>>> Best,
>>> Max
>>>
>>> On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <
>>> giacomo.licari@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>> I'm trying to setup a simple Flink cluster on Google Compute Engine.
>>>>
>>>> I'm running 3 nodes (1 master, 2 workers).
>>>>
>>>> On master node I set up ssh key and moved it into authorized_keys.
>>>>
>>>> When I try to copy my key to each worker node I got Permission denied
>>>> (publickey).
>>>>
>>>> Someone had the same problem?
>>>> Thanks for your help.
>>>>
>>>
>>>
>>
>

Re: Google Compute Engine Cluster

Posted by Maximilian Michels <mx...@apache.org>.

Sure, just ran an included example.

./bin/flink run examples/*WordCount.jar

By default, it will use a really small example DataSet but it should be
enough to verify the cluster setup.

You're welcome.

Best,
Max

On Wed, Apr 15, 2015 at 12:24 PM, Giacomo Licari <gi...@gmail.com>
wrote:

> Thanks a lot Max,
> now I can set up ssh keys.
>
> I started my cluster correctly with ./start-cluster.sh, do you use an
> example program to test if everything works fine?
>
> Thanks again,
> Giacomo
>
> On Wed, Apr 15, 2015 at 11:48 AM, Maximilian Michels <mx...@apache.org>
> wrote:
>
>> Hi Giacomo,
>>
>> Have you tried setting up Flink on GCE using bdutil? It is very easy:
>> http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html
>>
>> If you don't want to use bdutil:
>> I'm assuming you want to start Flink using the "start_cluster.sh" script
>> which uses ssh to start the task managers on the workers. By default,
>> instances on GCE are not authenticated via ssh with each other. Using the
>> gcloud tool, you can configure an ssh key on all hosts (gcloud compute
>> config-ssh) which lets you ssh into all instances. To enable public key ssh
>> authentication between your GCE nodes, I'd advise to create a new key on
>> your machine for just this purpose and then distribute the key (public and
>> private) to the master. Then add the public key to all worker nodes's
>> "authorized_keys" file from your machine.
>>
>>
>> Best,
>> Max
>>
>> On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <
>> giacomo.licari@gmail.com> wrote:
>>
>>> Hi guys,
>>> I'm trying to setup a simple Flink cluster on Google Compute Engine.
>>>
>>> I'm running 3 nodes (1 master, 2 workers).
>>>
>>> On master node I set up ssh key and moved it into authorized_keys.
>>>
>>> When I try to copy my key to each worker node I got Permission denied
>>> (publickey).
>>>
>>> Someone had the same problem?
>>> Thanks for your help.
>>>
>>
>>
>

Re: Google Compute Engine Cluster

Posted by Giacomo Licari <gi...@gmail.com>.

Thanks a lot Max,
now I can set up ssh keys.

I started my cluster correctly with ./start-cluster.sh, do you use an
example program to test if everything works fine?

Thanks again,
Giacomo

On Wed, Apr 15, 2015 at 11:48 AM, Maximilian Michels <mx...@apache.org> wrote:

> Hi Giacomo,
>
> Have you tried setting up Flink on GCE using bdutil? It is very easy:
> http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html
>
> If you don't want to use bdutil:
> I'm assuming you want to start Flink using the "start_cluster.sh" script
> which uses ssh to start the task managers on the workers. By default,
> instances on GCE are not authenticated via ssh with each other. Using the
> gcloud tool, you can configure an ssh key on all hosts (gcloud compute
> config-ssh) which lets you ssh into all instances. To enable public key ssh
> authentication between your GCE nodes, I'd advise to create a new key on
> your machine for just this purpose and then distribute the key (public and
> private) to the master. Then add the public key to all worker nodes's
> "authorized_keys" file from your machine.
>
>
> Best,
> Max
>
> On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <giacomo.licari@gmail.com
> > wrote:
>
>> Hi guys,
>> I'm trying to setup a simple Flink cluster on Google Compute Engine.
>>
>> I'm running 3 nodes (1 master, 2 workers).
>>
>> On master node I set up ssh key and moved it into authorized_keys.
>>
>> When I try to copy my key to each worker node I got Permission denied
>> (publickey).
>>
>> Someone had the same problem?
>> Thanks for your help.
>>
>
>

Re: Google Compute Engine Cluster

Posted by Maximilian Michels <mx...@apache.org>.

Hi Giacomo,

Have you tried setting up Flink on GCE using bdutil? It is very easy:
http://ci.apache.org/projects/flink/flink-docs-master/gce_setup.html

If you don't want to use bdutil:
I'm assuming you want to start Flink using the "start_cluster.sh" script
which uses ssh to start the task managers on the workers. By default,
instances on GCE are not authenticated via ssh with each other. Using the
gcloud tool, you can configure an ssh key on all hosts (gcloud compute
config-ssh) which lets you ssh into all instances. To enable public key ssh
authentication between your GCE nodes, I'd advise to create a new key on
your machine for just this purpose and then distribute the key (public and
private) to the master. Then add the public key to all worker nodes's
"authorized_keys" file from your machine.

Best,
Max

On Wed, Apr 15, 2015 at 11:22 AM, Giacomo Licari <gi...@gmail.com>
wrote:

> Hi guys,
> I'm trying to setup a simple Flink cluster on Google Compute Engine.
>
> I'm running 3 nodes (1 master, 2 workers).
>
> On master node I set up ssh key and moved it into authorized_keys.
>
> When I try to copy my key to each worker node I got Permission denied
> (publickey).
>
> Someone had the same problem?
> Thanks for your help.
>