You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Kartashov, Andy" <An...@mpac.ca> on 2012/11/14 15:34:54 UTC
shutting down instance
Yinghau,
Last week you mentioned that you are running your cluster on EC2 that you shut them down instance(s) over the weekend.
QTE
Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot.
UNQTE
Whenever I re-start an instance on Amazon cloud my both internal and external ip addresses for the instance change... ?
p.s. Has anyone successfully configured Zookeeper to curb this matter?
From: yinghua hu [mailto:yinghua.hu@gmail.com]
Sent: Sunday, November 11, 2012 2:06 PM
To: user@hadoop.apache.org
Subject: Re: error running pi program
Hi, Steve
These are very helpful suggestions! Thanks!
I will try using SSH proxy and see how it goes.
On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>> wrote:
On 10 November 2012 18:30, yinghua hu <yi...@gmail.com>> wrote:
Thanks Steve,
Your suggestion is very helpful. But I only have terminal access to the EC2 NN. To run internet browser there will be a lot of more work.
the SSH proxy sets up something that appears to be a SOCKs proxy -but which routes to the far end; you set your browser to go through this proxy (foxyproxy is a good firefox plugin here), and then your browser has access to all the machines
see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
Since my cluster will be shut down at night and weekend. I will prefer to have an IP that does not change. How do you think of using EC2 Elastic IP?
not tried that; I have used dyndns.org<http://dyndns.org> to self register an IP address that changes every time you bring up an instance, but whose browser bookmarks can remain fixed
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: shutting down instance
Posted by Samir Eljazovic <sa...@gmail.com>.
Hi,
You should consider using Amazon Virtual Private Cloud (VPC) service to
avoid re-assigning IP addresses every time you restart your instances.
Keep in mind limitation of 5 ElasticIPs per VPC.
Regards,
Samir
On 14 November 2012 15:34, Kartashov, Andy <An...@mpac.ca> wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Since I only have 4 nodes, I just copy and paste the external address from
amazon aws console web-interface. It is really much more work if you have
20 or more nodes in the cluster.
Other than that, I have a script to scp the xml files to all the hosts
listed in the slaves file. More nodes won't bring more work on transferring
since it is automatic.
Did you try to explore the option of VPC in AWS? I do not have experience
on that.
On Thu, Nov 15, 2012 at 6:53 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> I also use scp and remote ssh to push configuration files |
> start-stop-restart respective daemons in the cluster. This can simply
> resolution of every node in the cluster having an updated address of the
> new NN. But what about slave file on the NN?
>
>
>
> If you have 20 DN/TTs in a cluster, you will have to go inside each and
> every one to get internal ip-address or copy external ip-addresses from EC2
> web-inteface for each node and then replace the list of new nodes in your
> slave file. Is this how you do it?
>
>
>
> Rgds,
>
> AK-47
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Wednesday, November 14, 2012 12:03 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: shutting down instance
>
>
>
> Hi,Andy
>
>
>
> Yes. I shutdown EC2 instance every day to save some money for my company.
> I use shell scripts to reduce manual editing as much as possible. I just
> edit core-site.xml and yarn-site.xml in single node and then scp it to all
> the other nodes. This can be automated. You can add to /etc/ssh/ssh-config
> the two lines below
>
>
>
> StrictHostKeyChecking no
>
> UserKnownHostsFile /dev/null
>
>
>
> so that you will not need to type "yes" when connecting to an unknown
> host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and
> master on namenode everyday. It take only a few minutes.
>
>
>
> I did not use Zookeeper to deal with this. Please let me know if you have
> better approach.
>
>
>
> On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>
> wrote:
>
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Since I only have 4 nodes, I just copy and paste the external address from
amazon aws console web-interface. It is really much more work if you have
20 or more nodes in the cluster.
Other than that, I have a script to scp the xml files to all the hosts
listed in the slaves file. More nodes won't bring more work on transferring
since it is automatic.
Did you try to explore the option of VPC in AWS? I do not have experience
on that.
On Thu, Nov 15, 2012 at 6:53 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> I also use scp and remote ssh to push configuration files |
> start-stop-restart respective daemons in the cluster. This can simply
> resolution of every node in the cluster having an updated address of the
> new NN. But what about slave file on the NN?
>
>
>
> If you have 20 DN/TTs in a cluster, you will have to go inside each and
> every one to get internal ip-address or copy external ip-addresses from EC2
> web-inteface for each node and then replace the list of new nodes in your
> slave file. Is this how you do it?
>
>
>
> Rgds,
>
> AK-47
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Wednesday, November 14, 2012 12:03 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: shutting down instance
>
>
>
> Hi,Andy
>
>
>
> Yes. I shutdown EC2 instance every day to save some money for my company.
> I use shell scripts to reduce manual editing as much as possible. I just
> edit core-site.xml and yarn-site.xml in single node and then scp it to all
> the other nodes. This can be automated. You can add to /etc/ssh/ssh-config
> the two lines below
>
>
>
> StrictHostKeyChecking no
>
> UserKnownHostsFile /dev/null
>
>
>
> so that you will not need to type "yes" when connecting to an unknown
> host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and
> master on namenode everyday. It take only a few minutes.
>
>
>
> I did not use Zookeeper to deal with this. Please let me know if you have
> better approach.
>
>
>
> On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>
> wrote:
>
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Since I only have 4 nodes, I just copy and paste the external address from
amazon aws console web-interface. It is really much more work if you have
20 or more nodes in the cluster.
Other than that, I have a script to scp the xml files to all the hosts
listed in the slaves file. More nodes won't bring more work on transferring
since it is automatic.
Did you try to explore the option of VPC in AWS? I do not have experience
on that.
On Thu, Nov 15, 2012 at 6:53 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> I also use scp and remote ssh to push configuration files |
> start-stop-restart respective daemons in the cluster. This can simply
> resolution of every node in the cluster having an updated address of the
> new NN. But what about slave file on the NN?
>
>
>
> If you have 20 DN/TTs in a cluster, you will have to go inside each and
> every one to get internal ip-address or copy external ip-addresses from EC2
> web-inteface for each node and then replace the list of new nodes in your
> slave file. Is this how you do it?
>
>
>
> Rgds,
>
> AK-47
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Wednesday, November 14, 2012 12:03 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: shutting down instance
>
>
>
> Hi,Andy
>
>
>
> Yes. I shutdown EC2 instance every day to save some money for my company.
> I use shell scripts to reduce manual editing as much as possible. I just
> edit core-site.xml and yarn-site.xml in single node and then scp it to all
> the other nodes. This can be automated. You can add to /etc/ssh/ssh-config
> the two lines below
>
>
>
> StrictHostKeyChecking no
>
> UserKnownHostsFile /dev/null
>
>
>
> so that you will not need to type "yes" when connecting to an unknown
> host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and
> master on namenode everyday. It take only a few minutes.
>
>
>
> I did not use Zookeeper to deal with this. Please let me know if you have
> better approach.
>
>
>
> On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>
> wrote:
>
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Since I only have 4 nodes, I just copy and paste the external address from
amazon aws console web-interface. It is really much more work if you have
20 or more nodes in the cluster.
Other than that, I have a script to scp the xml files to all the hosts
listed in the slaves file. More nodes won't bring more work on transferring
since it is automatic.
Did you try to explore the option of VPC in AWS? I do not have experience
on that.
On Thu, Nov 15, 2012 at 6:53 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> I also use scp and remote ssh to push configuration files |
> start-stop-restart respective daemons in the cluster. This can simply
> resolution of every node in the cluster having an updated address of the
> new NN. But what about slave file on the NN?
>
>
>
> If you have 20 DN/TTs in a cluster, you will have to go inside each and
> every one to get internal ip-address or copy external ip-addresses from EC2
> web-inteface for each node and then replace the list of new nodes in your
> slave file. Is this how you do it?
>
>
>
> Rgds,
>
> AK-47
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Wednesday, November 14, 2012 12:03 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: shutting down instance
>
>
>
> Hi,Andy
>
>
>
> Yes. I shutdown EC2 instance every day to save some money for my company.
> I use shell scripts to reduce manual editing as much as possible. I just
> edit core-site.xml and yarn-site.xml in single node and then scp it to all
> the other nodes. This can be automated. You can add to /etc/ssh/ssh-config
> the two lines below
>
>
>
> StrictHostKeyChecking no
>
> UserKnownHostsFile /dev/null
>
>
>
> so that you will not need to type "yes" when connecting to an unknown
> host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and
> master on namenode everyday. It take only a few minutes.
>
>
>
> I did not use Zookeeper to deal with this. Please let me know if you have
> better approach.
>
>
>
> On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>
> wrote:
>
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
RE: shutting down instance
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I also use scp and remote ssh to push configuration files | start-stop-restart respective daemons in the cluster. This can simply resolution of every node in the cluster having an updated address of the new NN. But what about slave file on the NN?
If you have 20 DN/TTs in a cluster, you will have to go inside each and every one to get internal ip-address or copy external ip-addresses from EC2 web-inteface for each node and then replace the list of new nodes in your slave file. Is this how you do it?
Rgds,
AK-47
From: yinghua hu [mailto:yinghua.hu@gmail.com]
Sent: Wednesday, November 14, 2012 12:03 PM
To: user@hadoop.apache.org
Subject: Re: shutting down instance
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I use shell scripts to reduce manual editing as much as possible. I just edit core-site.xml and yarn-site.xml in single node and then scp it to all the other nodes. This can be automated. You can add to /etc/ssh/ssh-config the two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>> wrote:
Yinghau,
Last week you mentioned that you are running your cluster on EC2 that you shut them down instance(s) over the weekend.
QTE
Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot.
UNQTE
Whenever I re-start an instance on Amazon cloud my both internal and external ip addresses for the instance change... ?
p.s. Has anyone successfully configured Zookeeper to curb this matter?
From: yinghua hu [mailto:yinghua.hu@gmail.com<ma...@gmail.com>]
Sent: Sunday, November 11, 2012 2:06 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: error running pi program
Hi, Steve
These are very helpful suggestions! Thanks!
I will try using SSH proxy and see how it goes.
On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>> wrote:
On 10 November 2012 18:30, yinghua hu <yi...@gmail.com>> wrote:
Thanks Steve,
Your suggestion is very helpful. But I only have terminal access to the EC2 NN. To run internet browser there will be a lot of more work.
the SSH proxy sets up something that appears to be a SOCKs proxy -but which routes to the far end; you set your browser to go through this proxy (foxyproxy is a good firefox plugin here), and then your browser has access to all the machines
see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
Since my cluster will be shut down at night and weekend. I will prefer to have an IP that does not change. How do you think of using EC2 Elastic IP?
not tried that; I have used dyndns.org<http://dyndns.org> to self register an IP address that changes every time you bring up an instance, but whose browser bookmarks can remain fixed
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: shutting down instance
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I also use scp and remote ssh to push configuration files | start-stop-restart respective daemons in the cluster. This can simply resolution of every node in the cluster having an updated address of the new NN. But what about slave file on the NN?
If you have 20 DN/TTs in a cluster, you will have to go inside each and every one to get internal ip-address or copy external ip-addresses from EC2 web-inteface for each node and then replace the list of new nodes in your slave file. Is this how you do it?
Rgds,
AK-47
From: yinghua hu [mailto:yinghua.hu@gmail.com]
Sent: Wednesday, November 14, 2012 12:03 PM
To: user@hadoop.apache.org
Subject: Re: shutting down instance
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I use shell scripts to reduce manual editing as much as possible. I just edit core-site.xml and yarn-site.xml in single node and then scp it to all the other nodes. This can be automated. You can add to /etc/ssh/ssh-config the two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>> wrote:
Yinghau,
Last week you mentioned that you are running your cluster on EC2 that you shut them down instance(s) over the weekend.
QTE
Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot.
UNQTE
Whenever I re-start an instance on Amazon cloud my both internal and external ip addresses for the instance change... ?
p.s. Has anyone successfully configured Zookeeper to curb this matter?
From: yinghua hu [mailto:yinghua.hu@gmail.com<ma...@gmail.com>]
Sent: Sunday, November 11, 2012 2:06 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: error running pi program
Hi, Steve
These are very helpful suggestions! Thanks!
I will try using SSH proxy and see how it goes.
On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>> wrote:
On 10 November 2012 18:30, yinghua hu <yi...@gmail.com>> wrote:
Thanks Steve,
Your suggestion is very helpful. But I only have terminal access to the EC2 NN. To run internet browser there will be a lot of more work.
the SSH proxy sets up something that appears to be a SOCKs proxy -but which routes to the far end; you set your browser to go through this proxy (foxyproxy is a good firefox plugin here), and then your browser has access to all the machines
see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
Since my cluster will be shut down at night and weekend. I will prefer to have an IP that does not change. How do you think of using EC2 Elastic IP?
not tried that; I have used dyndns.org<http://dyndns.org> to self register an IP address that changes every time you bring up an instance, but whose browser bookmarks can remain fixed
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: shutting down instance
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I also use scp and remote ssh to push configuration files | start-stop-restart respective daemons in the cluster. This can simply resolution of every node in the cluster having an updated address of the new NN. But what about slave file on the NN?
If you have 20 DN/TTs in a cluster, you will have to go inside each and every one to get internal ip-address or copy external ip-addresses from EC2 web-inteface for each node and then replace the list of new nodes in your slave file. Is this how you do it?
Rgds,
AK-47
From: yinghua hu [mailto:yinghua.hu@gmail.com]
Sent: Wednesday, November 14, 2012 12:03 PM
To: user@hadoop.apache.org
Subject: Re: shutting down instance
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I use shell scripts to reduce manual editing as much as possible. I just edit core-site.xml and yarn-site.xml in single node and then scp it to all the other nodes. This can be automated. You can add to /etc/ssh/ssh-config the two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>> wrote:
Yinghau,
Last week you mentioned that you are running your cluster on EC2 that you shut them down instance(s) over the weekend.
QTE
Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot.
UNQTE
Whenever I re-start an instance on Amazon cloud my both internal and external ip addresses for the instance change... ?
p.s. Has anyone successfully configured Zookeeper to curb this matter?
From: yinghua hu [mailto:yinghua.hu@gmail.com<ma...@gmail.com>]
Sent: Sunday, November 11, 2012 2:06 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: error running pi program
Hi, Steve
These are very helpful suggestions! Thanks!
I will try using SSH proxy and see how it goes.
On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>> wrote:
On 10 November 2012 18:30, yinghua hu <yi...@gmail.com>> wrote:
Thanks Steve,
Your suggestion is very helpful. But I only have terminal access to the EC2 NN. To run internet browser there will be a lot of more work.
the SSH proxy sets up something that appears to be a SOCKs proxy -but which routes to the far end; you set your browser to go through this proxy (foxyproxy is a good firefox plugin here), and then your browser has access to all the machines
see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
Since my cluster will be shut down at night and weekend. I will prefer to have an IP that does not change. How do you think of using EC2 Elastic IP?
not tried that; I have used dyndns.org<http://dyndns.org> to self register an IP address that changes every time you bring up an instance, but whose browser bookmarks can remain fixed
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: shutting down instance
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I also use scp and remote ssh to push configuration files | start-stop-restart respective daemons in the cluster. This can simply resolution of every node in the cluster having an updated address of the new NN. But what about slave file on the NN?
If you have 20 DN/TTs in a cluster, you will have to go inside each and every one to get internal ip-address or copy external ip-addresses from EC2 web-inteface for each node and then replace the list of new nodes in your slave file. Is this how you do it?
Rgds,
AK-47
From: yinghua hu [mailto:yinghua.hu@gmail.com]
Sent: Wednesday, November 14, 2012 12:03 PM
To: user@hadoop.apache.org
Subject: Re: shutting down instance
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I use shell scripts to reduce manual editing as much as possible. I just edit core-site.xml and yarn-site.xml in single node and then scp it to all the other nodes. This can be automated. You can add to /etc/ssh/ssh-config the two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>> wrote:
Yinghau,
Last week you mentioned that you are running your cluster on EC2 that you shut them down instance(s) over the weekend.
QTE
Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot.
UNQTE
Whenever I re-start an instance on Amazon cloud my both internal and external ip addresses for the instance change... ?
p.s. Has anyone successfully configured Zookeeper to curb this matter?
From: yinghua hu [mailto:yinghua.hu@gmail.com<ma...@gmail.com>]
Sent: Sunday, November 11, 2012 2:06 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: error running pi program
Hi, Steve
These are very helpful suggestions! Thanks!
I will try using SSH proxy and see how it goes.
On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>> wrote:
On 10 November 2012 18:30, yinghua hu <yi...@gmail.com>> wrote:
Thanks Steve,
Your suggestion is very helpful. But I only have terminal access to the EC2 NN. To run internet browser there will be a lot of more work.
the SSH proxy sets up something that appears to be a SOCKs proxy -but which routes to the far end; you set your browser to go through this proxy (foxyproxy is a good firefox plugin here), and then your browser has access to all the machines
see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
Since my cluster will be shut down at night and weekend. I will prefer to have an IP that does not change. How do you think of using EC2 Elastic IP?
not tried that; I have used dyndns.org<http://dyndns.org> to self register an IP address that changes every time you bring up an instance, but whose browser bookmarks can remain fixed
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
--
Regards,
Yinghua
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I
use shell scripts to reduce manual editing as much as possible. I just edit
core-site.xml and yarn-site.xml in single node and then scp it to all the
other nodes. This can be automated. You can add to /etc/ssh/ssh-config the
two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host.
So I re-edited four files core-site.xml and yarn-site.xml, slaves and
master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have
better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
Re: shutting down instance
Posted by Samir Eljazovic <sa...@gmail.com>.
Hi,
You should consider using Amazon Virtual Private Cloud (VPC) service to
avoid re-assigning IP addresses every time you restart your instances.
Keep in mind limitation of 5 ElasticIPs per VPC.
Regards,
Samir
On 14 November 2012 15:34, Kartashov, Andy <An...@mpac.ca> wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: shutting down instance
Posted by Samir Eljazovic <sa...@gmail.com>.
Hi,
You should consider using Amazon Virtual Private Cloud (VPC) service to
avoid re-assigning IP addresses every time you restart your instances.
Keep in mind limitation of 5 ElasticIPs per VPC.
Regards,
Samir
On 14 November 2012 15:34, Kartashov, Andy <An...@mpac.ca> wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I
use shell scripts to reduce manual editing as much as possible. I just edit
core-site.xml and yarn-site.xml in single node and then scp it to all the
other nodes. This can be automated. You can add to /etc/ssh/ssh-config the
two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host.
So I re-edited four files core-site.xml and yarn-site.xml, slaves and
master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have
better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
Re: shutting down instance
Posted by Samir Eljazovic <sa...@gmail.com>.
Hi,
You should consider using Amazon Virtual Private Cloud (VPC) service to
avoid re-assigning IP addresses every time you restart your instances.
Keep in mind limitation of 5 ElasticIPs per VPC.
Regards,
Samir
On 14 November 2012 15:34, Kartashov, Andy <An...@mpac.ca> wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I
use shell scripts to reduce manual editing as much as possible. I just edit
core-site.xml and yarn-site.xml in single node and then scp it to all the
other nodes. This can be automated. You can add to /etc/ssh/ssh-config the
two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host.
So I re-edited four files core-site.xml and yarn-site.xml, slaves and
master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have
better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua
Re: shutting down instance
Posted by yinghua hu <yi...@gmail.com>.
Hi,Andy
Yes. I shutdown EC2 instance every day to save some money for my company. I
use shell scripts to reduce manual editing as much as possible. I just edit
core-site.xml and yarn-site.xml in single node and then scp it to all the
other nodes. This can be automated. You can add to /etc/ssh/ssh-config the
two lines below
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
so that you will not need to type "yes" when connecting to an unknown host.
So I re-edited four files core-site.xml and yarn-site.xml, slaves and
master on namenode everyday. It take only a few minutes.
I did not use Zookeeper to deal with this. Please let me know if you have
better approach.
On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <An...@mpac.ca>wrote:
> Yinghau,
>
>
>
> Last week you mentioned that you are running your cluster on EC2 that you
> shut them down instance(s) over the weekend.
>
>
>
> QTE
>
> Since I shutdown the EC2 instance every night, I thought that using
> 'master','slave1','slave2' will save typing after the full host name change
> with reboot.
>
> UNQTE
>
>
>
> Whenever I re-start an instance on Amazon cloud my both internal and
> external ip addresses for the instance change… ?
>
>
>
> p.s. Has anyone successfully configured Zookeeper to curb this matter?
>
>
>
> *From:* yinghua hu [mailto:yinghua.hu@gmail.com]
> *Sent:* Sunday, November 11, 2012 2:06 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: error running pi program
>
>
>
> Hi, Steve
>
>
>
> These are very helpful suggestions! Thanks!
>
>
>
> I will try using SSH proxy and see how it goes.
>
> On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>
>
> On 10 November 2012 18:30, yinghua hu <yi...@gmail.com> wrote:
>
> Thanks Steve,
>
>
>
> Your suggestion is very helpful. But I only have terminal access to the
> EC2 NN. To run internet browser there will be a lot of more work.
>
>
>
> the SSH proxy sets up something that appears to be a SOCKs proxy -but
> which routes to the far end; you set your browser to go through this proxy
> (foxyproxy is a good firefox plugin here), and then your browser has access
> to all the machines
>
>
>
> see http://www.linuxjournal.com/content/use-ssh-create-http-proxy
>
>
>
>
>
>
>
> Since my cluster will be shut down at night and weekend. I will prefer to
> have an IP that does not change. How do you think of using EC2 Elastic IP?
>
>
>
>
>
> not tried that; I have used dyndns.org to self register an IP address
> that changes every time you bring up an instance, but whose browser
> bookmarks can remain fixed
>
>
>
>
>
> --
> Regards,
>
> Yinghua
> NOTICE: This e-mail message and any attachments are confidential,
> subject to copyright and may be privileged. Any unauthorized use, copying
> or disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>
--
Regards,
Yinghua