You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Tony Burton <TB...@SportingIndex.com> on 2013/01/18 16:54:22 UTC
HWI use on AWS/EMR
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
Re: HWI use on AWS/EMR
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
Yes, this is straightforward to set up in Putty. Ariel, thanks for
providing the details I was too lazy to mention ;)
One flaw with this approach is that only your machine will have access to
HWI. That is, the port is forwarded only to your machine. Making the ports
public on the master node means anyone on the interwebs can see it. You can
also set up security groups in AWS so that only specifically authorized
people can access the ports. That's a bit involved...
dean
On Fri, Jan 18, 2013 at 10:31 AM, Ariel Marcus <ar...@openbi.com>wrote:
> Hey Tony,
>
> Port forwarding is the same as what Dean referred to as ssh tunneling.
>
> Here is a website explaining how to set up port forwarding in putty (use
> port 9999):
> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
>
> You can edit either hive-default or hive-site but the general practice is
> to store changes from the default configuration in hive-site.
>
> Best,
> Ariel
>
>
> ---------------------------------
> Ariel Marcus, Consultant
> www.openbi.com | ariel.marcus@openbi.com
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356
>
>
> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>wrote:
>
>> Hi Ariel,****
>>
>> ** **
>>
>> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
>> rather Linux desktops, so can you help me out with understanding how to
>> carry out the port forwarding from a Windows environment? Currently I use
>> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
>> session.****
>>
>> ** **
>>
>> The changes to the hive config look identical to what I’ve changed in my
>> hive-default.xml. Does it make any difference which config file the
>> changesare applied to (hive-default.xml vs hive-site.xml)?****
>>
>> ** **
>>
>> Tony****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
>> *Sent:* 18 January 2013 16:16
>>
>> *To:* user@hive.apache.org
>> *Subject:* Re: HWI use on AWS/EMR****
>>
>> ** **
>>
>> Hey Tony,****
>>
>> ** **
>>
>> I would recommend using ssh port forwarding instead of making your hwi
>> publicly available on the internet. When you connect to the master node
>> using ssh you should use a command like the following:****
>>
>> ** **
>>
>> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
>> MASTER_HOSTNAME****
>>
>> ** **
>>
>> After you have connected, add the following to your
>> hive/conf/hive-site.xml file:****
>>
>> ** **
>>
>> <property>****
>>
>> <name>hive.hwi.war.file</name>****
>>
>> <value>lib/hive-hwi-0.8.1.war</value>****
>>
>> <description>This is the WAR file with the jsp content for Hive Web
>> Interface</description>****
>>
>> </property>****
>>
>> ** **
>>
>> Run this command to start up hwi:****
>>
>> ** **
>>
>> hive --service hwi****
>>
>> ** **
>>
>> And finally point your browser to:****
>>
>> ** **
>>
>> localhost:9999****
>>
>> ** **
>>
>> That worked for me.****
>>
>> ** **
>>
>> Best,****
>>
>> Ariel****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>>
>> ****
>>
>> ---------------------------------****
>>
>> Ariel Marcus, Consultant****
>>
>> www.openbi.com | ariel.marcus@openbi.com****
>>
>> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
>> Cell: 314-827-4356****
>>
>> ** **
>>
>> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
>> dean.wampler@thinkbiganalytics.com> wrote:****
>>
>> Oops, I overlooked that you have the public domain name in your message.
>> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
>> ****
>>
>> ** **
>>
>> If it's blocked you can use ssh to tunnel the port through.****
>>
>> ** **
>>
>> dean****
>>
>> ** **
>>
>> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
>> dean.wampler@thinkbiganalytics.com> wrote:****
>>
>> That's the internal hostname, not visible outside. Use the name like
>> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
>> and the elastic-mapreduce script you might have used to launch the cluster.
>> ****
>>
>> ** **
>>
>> If that doesn't work, verify that port 80 is not blocked by default.
>> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>>
>> ** **
>>
>> dean****
>>
>> ** **
>>
>> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
>> wrote:****
>>
>> Hi,
>>
>> I'm trying to get HWI running and accessible from an Amazon Web Services
>> EMR instance. I've hit a blocker early on though, and the documentation is
>> less than illuminating. Can you share any experiences you have had?
>> Specifically, here's what I'm curious about.
>>
>> - Running on AWS. I've created a Hive job flow on AWS, edited
>> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
>> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
>> - HWI starts up fine, but when I try to connect with
>> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
>> timeout message, in Firefox it's "The connection has timed out - The server
>> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
>> to respond".
>>
>> Has anyone successfully connected to HWI running on an AWS EMR instance?
>> From the same browser I can connect to HWI on a local Ubuntu box.
>>
>> Thanks!
>>
>> Tony
>>
>>
>> **********************************************************************
>> Please consider the environment before printing this email or attachments
>>
>> This email and any attachments are confidential, protected by copyright
>> and may be legally privileged. If you are not the intended recipient, then
>> the dissemination or copying of this email is prohibited. If you have
>> received this in error, please notify the sender by replying by email and
>> then delete the email completely from your system. Neither Sporting Index
>> nor the sender accepts responsibility for any virus, or any other defect
>> which might affect any computer or IT system into which the email is
>> received and/or opened. It is the responsibility of the recipient to scan
>> the email and no responsibility is accepted for any loss or damage arising
>> in any way from receipt or use of this email. Sporting Index Ltd is a
>> company registered in England and Wales with company number 2636842, whose
>> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
>> Sporting Index Ltd is authorised and regulated by the UK Financial
>> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
>> 000-027343-R-308898-001). Any financial promotion contained herein has
>> been issued
>> and approved by Sporting Index Ltd.
>>
>> Outbound email has been scanned for viruses and SPAM****
>>
>>
>>
>> ****
>>
>> ** **
>>
>> --
>> *Dean Wampler, Ph.D.*****
>>
>> thinkbiganalytics.com****
>>
>> +1-312-339-1330****
>>
>> ** **
>>
>>
>>
>> ****
>>
>> ** **
>>
>> --
>> *Dean Wampler, Ph.D.*****
>>
>> thinkbiganalytics.com****
>>
>> +1-312-339-1330****
>>
>> ** **
>>
>> ** **
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.****
>>
>> ** **
>>
>> ** **
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.****
>>
>> ** **
>>
>>
>>
>> *****************************************************************************
>> P *Please consider the environment before printing this email* ****
>>
>>
>> www.sportingindex.com
>>
>> Inbound email has been scanned for viruses & spam****
>>
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.
>>
>
>
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
Hi Nitin (and others who have contributed)
Thanks again for the suggestions - however my original objective was to investigate HWI as an alternative to hive in CLI mode for less techie analysts to get their hands on data in S3. At the moment it's easier for them to start an EMR instance running hive than it is to spawn instances from other instances with the appropriate security group; so I'm letting them carry on with the former for now.
Tony
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: 21 January 2013 13:17
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
if you are not inside VPC ec2 does not let you change security groups.
so best option would be create a new ami from ur instance and then launch a new instance with a security group you newly created.
or change the default security group to have the changes you want
On Mon, Jan 21, 2013 at 6:43 PM, Tony Burton <TB...@sportingindex.com>> wrote:
Thanks for the suggestion Nitin.
I can see the "Change Security Groups" option for my instance (when looking at the EC2 console), but it's greyed out. The only option I have on the EMR console for the instance is "Terminate Job".
Is your "Change Security Groups" option enabled? If so, any idea what steps you took in creating the instance to enable it?
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 21 January 2013 11:57
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
right click on the instance in your aws console and you should be able to change the security group for the node.
This feature was not there previously, came in few months back
On Mon, Jan 21, 2013 at 5:09 PM, Tony Burton <TB...@sportingindex.com>> wrote:
I've tracked down this document on AWS:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html
which shows how to create security groups to associate with your AWS instances. Simple enough to create a rule - but is there anyway to associate a rule with an instance either when the instance is running, or at creation time using the AWS Management Console? Or can it only be done using the commandline client or by an http request?
Thanks,
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 17:25
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Nitin's approach would work for you as well. Amazon has very good documentation of their services. A quick google search should get you the right info.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TB...@sportingindex.com>> wrote:
If you could provide the steps to do this I'd be grateful - thanks.
Thanks for all your input so far, time to head home for the weekend so I'll try it again with a fresh EMR instance next week.
Tony
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 18 January 2013 17:08
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
ok so much discussion around this but why dont you open access to port 9999 for your specific machine (home/office) and access it normally without setting up port forwarding.
AWS does give you ip level access control in the security groups
On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>> wrote:
This should be localhost instead of proxy:
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, localhost:9999 in the Destination box. Not sure if this is correct.
Best,
Aril
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>> wrote:
Ok, thanks Ariel (and Dean in the next email!)
I've set up PuTTY (v0.62) as follows:
- EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
- In Connection/SSH/Auth I've added my private key file, downloaded from AWS ages for this account.
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, proxy:9999 in the Destination box. Not sure if this is correct.
Hit the Open button, I can connect. (And I can see my changes in hive-default.xml, so I know I'm in the right place:)
Then pointing my browser at http://localhost:9999 I get the same timeout again. Have I set up everything right?
Also, following good advice I copied the config changes to hive-site.xml. How do I stop hive so I can restart with 'hive -service hwi' so that it picks up the new config?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:31
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Port forwarding is the same as what Dean referred to as ssh tunneling.
Here is a website explaining how to set up port forwarding in putty (use port 9999):
http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
You can edit either hive-default or hive-site but the general practice is to store changes from the default configuration in hive-site.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi Ariel,
Thanks for the speedy reply. We'll be accessing the HWI from Windows rather Linux desktops, so can you help me out with understanding how to carry out the port forwarding from a Windows environment? Currently I use WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY session.
The changes to the hive config look identical to what I've changed in my hive-default.xml. Does it make any difference which config file the changesare applied to (hive-default.xml vs hive-site.xml)?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi publicly available on the internet. When you connect to the master node using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
--
Nitin Pawar
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
--
Nitin Pawar
--
Nitin Pawar
Re: HWI use on AWS/EMR
Posted by Nitin Pawar <ni...@gmail.com>.
if you are not inside VPC ec2 does not let you change security groups.
so best option would be create a new ami from ur instance and then launch a
new instance with a security group you newly created.
or change the default security group to have the changes you want
On Mon, Jan 21, 2013 at 6:43 PM, Tony Burton <TB...@sportingindex.com>wrote:
> Thanks for the suggestion Nitin.****
>
> ** **
>
> I can see the “Change Security Groups” option for my instance (when
> looking at the EC2 console), but it’s greyed out. The only option I have on
> the EMR console for the instance is “Terminate Job”.****
>
> ** **
>
> Is your “Change Security Groups” option enabled? If so, any idea what
> steps you took in creating the instance to enable it?****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 21 January 2013 11:57
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ** **
>
> right click on the instance in your aws console and you should be able to
> change the security group for the node. ****
>
> ** **
>
> This feature was not there previously, came in few months back ****
>
> ** **
>
> On Mon, Jan 21, 2013 at 5:09 PM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> I’ve tracked down this document on AWS:****
>
> ****
>
>
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html
> ****
>
> ****
>
> which shows how to create security groups to associate with your AWS
> instances. Simple enough to create a rule – but is there anyway to
> associate a rule with an instance either when the instance is running, or
> at creation time using the AWS Management Console? Or can it only be done
> using the commandline client or by an http request? ****
>
> ****
>
> Thanks,****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 17:25****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> Nitin's approach would work for you as well. Amazon has very good
> documentation of their services. A quick google search should get you the
> right info.****
>
> ****
>
> Best,****
>
> Ariel****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> If you could provide the steps to do this I’d be grateful – thanks.****
>
> ****
>
> Thanks for all your input so far, time to head home for the weekend so
> I’ll try it again with a fresh EMR instance next week.****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 18 January 2013 17:08****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> ok so much discussion around this but why dont you open access to port
> 9999 for your specific machine (home/office) and access it normally without
> setting up port forwarding. ****
>
> ****
>
> AWS does give you ip level access control in the security groups ****
>
> ****
>
> On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>
> wrote:****
>
> This should be localhost instead of proxy:****
>
> ****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box, *
> localhost*:9999 in the Destination box. Not sure if this is correct.****
>
> ****
>
> Best,****
>
> Aril****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> ****
>
> Ok, thanks Ariel (and Dean in the next email!)****
>
> ****
>
> I’ve set up PuTTY (v0.62) as follows:****
>
> - EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
> ****
>
> - In Connection/SSH/Auth I’ve added my private key file, downloaded from
> AWS ages for this account.****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box,
> proxy:9999 in the Destination box. Not sure if this is correct.****
>
> ****
>
> Hit the Open button, I can connect. (And I can see my changes in
> hive-default.xml, so I know I’m in the right place:)****
>
> Then pointing my browser at http://localhost:9999 I get the same timeout
> again. Have I set up everything right?****
>
> ****
>
> Also, following good advice I copied the config changes to hive-site.xml.
> How do I stop hive so I can restart with ‘hive –service hwi’ so that it
> picks up the new config?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:31****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> Port forwarding is the same as what Dean referred to as ssh tunneling.****
>
> ****
>
> Here is a website explaining how to set up port forwarding in putty (use
> port 9999):****
>
> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html****
>
> ****
>
> You can edit either hive-default or hive-site but the general practice is
> to store changes from the default configuration in hive-site.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi Ariel,****
>
> ****
>
> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
> rather Linux desktops, so can you help me out with understanding how to
> carry out the port forwarding from a Windows environment? Currently I use
> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
> session.****
>
> ****
>
> The changes to the hive config look identical to what I’ve changed in my
> hive-default.xml. Does it make any difference which config file the
> changesare applied to (hive-default.xml vs hive-site.xml)?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:16****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> I would recommend using ssh port forwarding instead of making your hwi
> publicly available on the internet. When you connect to the master node
> using ssh you should use a command like the following:****
>
> ****
>
> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
> MASTER_HOSTNAME****
>
> ****
>
> After you have connected, add the following to your
> hive/conf/hive-site.xml file:****
>
> ****
>
> <property>****
>
> <name>hive.hwi.war.file</name>****
>
> <value>lib/hive-hwi-0.8.1.war</value>****
>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>****
>
> </property>****
>
> ****
>
> Run this command to start up hwi:****
>
> ****
>
> hive --service hwi****
>
> ****
>
> And finally point your browser to:****
>
> ****
>
> localhost:9999****
>
> ****
>
> That worked for me.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
> ****
>
> ****
>
> If it's blocked you can use ssh to tunnel the port through.****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
> ****
>
> ****
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged. If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system. Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened. It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email. Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
> Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001). Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
>
>
> *****************************************************************************
> P *Please consider the environment before printing this email* ****
>
>
> www.sportingindex.com
>
> Inbound email has been scanned for viruses & spam****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>
>
> ****
>
> ****
>
> --
> Nitin Pawar****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>
--
Nitin Pawar
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
Thanks for the suggestion Nitin.
I can see the "Change Security Groups" option for my instance (when looking at the EC2 console), but it's greyed out. The only option I have on the EMR console for the instance is "Terminate Job".
Is your "Change Security Groups" option enabled? If so, any idea what steps you took in creating the instance to enable it?
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: 21 January 2013 11:57
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
right click on the instance in your aws console and you should be able to change the security group for the node.
This feature was not there previously, came in few months back
On Mon, Jan 21, 2013 at 5:09 PM, Tony Burton <TB...@sportingindex.com>> wrote:
I've tracked down this document on AWS:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html
which shows how to create security groups to associate with your AWS instances. Simple enough to create a rule - but is there anyway to associate a rule with an instance either when the instance is running, or at creation time using the AWS Management Console? Or can it only be done using the commandline client or by an http request?
Thanks,
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 17:25
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Nitin's approach would work for you as well. Amazon has very good documentation of their services. A quick google search should get you the right info.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TB...@sportingindex.com>> wrote:
If you could provide the steps to do this I'd be grateful - thanks.
Thanks for all your input so far, time to head home for the weekend so I'll try it again with a fresh EMR instance next week.
Tony
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 18 January 2013 17:08
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
ok so much discussion around this but why dont you open access to port 9999 for your specific machine (home/office) and access it normally without setting up port forwarding.
AWS does give you ip level access control in the security groups
On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>> wrote:
This should be localhost instead of proxy:
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, localhost:9999 in the Destination box. Not sure if this is correct.
Best,
Aril
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>> wrote:
Ok, thanks Ariel (and Dean in the next email!)
I've set up PuTTY (v0.62) as follows:
- EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
- In Connection/SSH/Auth I've added my private key file, downloaded from AWS ages for this account.
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, proxy:9999 in the Destination box. Not sure if this is correct.
Hit the Open button, I can connect. (And I can see my changes in hive-default.xml, so I know I'm in the right place:)
Then pointing my browser at http://localhost:9999 I get the same timeout again. Have I set up everything right?
Also, following good advice I copied the config changes to hive-site.xml. How do I stop hive so I can restart with 'hive -service hwi' so that it picks up the new config?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:31
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Port forwarding is the same as what Dean referred to as ssh tunneling.
Here is a website explaining how to set up port forwarding in putty (use port 9999):
http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
You can edit either hive-default or hive-site but the general practice is to store changes from the default configuration in hive-site.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi Ariel,
Thanks for the speedy reply. We'll be accessing the HWI from Windows rather Linux desktops, so can you help me out with understanding how to carry out the port forwarding from a Windows environment? Currently I use WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY session.
The changes to the hive config look identical to what I've changed in my hive-default.xml. Does it make any difference which config file the changesare applied to (hive-default.xml vs hive-site.xml)?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi publicly available on the internet. When you connect to the master node using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
--
Nitin Pawar
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
--
Nitin Pawar
Re: HWI use on AWS/EMR
Posted by Nitin Pawar <ni...@gmail.com>.
right click on the instance in your aws console and you should be able to
change the security group for the node.
This feature was not there previously, came in few months back
On Mon, Jan 21, 2013 at 5:09 PM, Tony Burton <TB...@sportingindex.com>wrote:
> I’ve tracked down this document on AWS:****
>
> ** **
>
>
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html
> ****
>
> ** **
>
> which shows how to create security groups to associate with your AWS
> instances. Simple enough to create a rule – but is there anyway to
> associate a rule with an instance either when the instance is running, or
> at creation time using the AWS Management Console? Or can it only be done
> using the commandline client or by an http request? ****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Tony****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 17:25
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ** **
>
> Hey Tony,****
>
> ** **
>
> Nitin's approach would work for you as well. Amazon has very good
> documentation of their services. A quick google search should get you the
> right info.****
>
> ** **
>
> Best,****
>
> Ariel****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ** **
>
> On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> If you could provide the steps to do this I’d be grateful – thanks.****
>
> ****
>
> Thanks for all your input so far, time to head home for the weekend so
> I’ll try it again with a fresh EMR instance next week.****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 18 January 2013 17:08****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> ok so much discussion around this but why dont you open access to port
> 9999 for your specific machine (home/office) and access it normally without
> setting up port forwarding. ****
>
> ****
>
> AWS does give you ip level access control in the security groups ****
>
> ****
>
> On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>
> wrote:****
>
> This should be localhost instead of proxy:****
>
> ****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box, *
> localhost*:9999 in the Destination box. Not sure if this is correct.****
>
> ****
>
> Best,****
>
> Aril****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> ****
>
> Ok, thanks Ariel (and Dean in the next email!)****
>
> ****
>
> I’ve set up PuTTY (v0.62) as follows:****
>
> - EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
> ****
>
> - In Connection/SSH/Auth I’ve added my private key file, downloaded from
> AWS ages for this account.****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box,
> proxy:9999 in the Destination box. Not sure if this is correct.****
>
> ****
>
> Hit the Open button, I can connect. (And I can see my changes in
> hive-default.xml, so I know I’m in the right place:)****
>
> Then pointing my browser at http://localhost:9999 I get the same timeout
> again. Have I set up everything right?****
>
> ****
>
> Also, following good advice I copied the config changes to hive-site.xml.
> How do I stop hive so I can restart with ‘hive –service hwi’ so that it
> picks up the new config?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:31****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> Port forwarding is the same as what Dean referred to as ssh tunneling.****
>
> ****
>
> Here is a website explaining how to set up port forwarding in putty (use
> port 9999):****
>
> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html****
>
> ****
>
> You can edit either hive-default or hive-site but the general practice is
> to store changes from the default configuration in hive-site.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi Ariel,****
>
> ****
>
> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
> rather Linux desktops, so can you help me out with understanding how to
> carry out the port forwarding from a Windows environment? Currently I use
> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
> session.****
>
> ****
>
> The changes to the hive config look identical to what I’ve changed in my
> hive-default.xml. Does it make any difference which config file the
> changesare applied to (hive-default.xml vs hive-site.xml)?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:16****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> I would recommend using ssh port forwarding instead of making your hwi
> publicly available on the internet. When you connect to the master node
> using ssh you should use a command like the following:****
>
> ****
>
> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
> MASTER_HOSTNAME****
>
> ****
>
> After you have connected, add the following to your
> hive/conf/hive-site.xml file:****
>
> ****
>
> <property>****
>
> <name>hive.hwi.war.file</name>****
>
> <value>lib/hive-hwi-0.8.1.war</value>****
>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>****
>
> </property>****
>
> ****
>
> Run this command to start up hwi:****
>
> ****
>
> hive --service hwi****
>
> ****
>
> And finally point your browser to:****
>
> ****
>
> localhost:9999****
>
> ****
>
> That worked for me.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
> ****
>
> ****
>
> If it's blocked you can use ssh to tunnel the port through.****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
> ****
>
> ****
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged. If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system. Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened. It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email. Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
> Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001). Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
>
>
> *****************************************************************************
> P *Please consider the environment before printing this email* ****
>
>
> www.sportingindex.com
>
> Inbound email has been scanned for viruses & spam****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>
>
> ****
>
> ****
>
> --
> Nitin Pawar****
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ** **
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
--
Nitin Pawar
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
I've tracked down this document on AWS:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html
which shows how to create security groups to associate with your AWS instances. Simple enough to create a rule - but is there anyway to associate a rule with an instance either when the instance is running, or at creation time using the AWS Management Console? Or can it only be done using the commandline client or by an http request?
Thanks,
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com]
Sent: 18 January 2013 17:25
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Nitin's approach would work for you as well. Amazon has very good documentation of their services. A quick google search should get you the right info.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TB...@sportingindex.com>> wrote:
If you could provide the steps to do this I'd be grateful - thanks.
Thanks for all your input so far, time to head home for the weekend so I'll try it again with a fresh EMR instance next week.
Tony
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 18 January 2013 17:08
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
ok so much discussion around this but why dont you open access to port 9999 for your specific machine (home/office) and access it normally without setting up port forwarding.
AWS does give you ip level access control in the security groups
On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>> wrote:
This should be localhost instead of proxy:
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, localhost:9999 in the Destination box. Not sure if this is correct.
Best,
Aril
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>> wrote:
Ok, thanks Ariel (and Dean in the next email!)
I've set up PuTTY (v0.62) as follows:
- EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
- In Connection/SSH/Auth I've added my private key file, downloaded from AWS ages for this account.
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, proxy:9999 in the Destination box. Not sure if this is correct.
Hit the Open button, I can connect. (And I can see my changes in hive-default.xml, so I know I'm in the right place:)
Then pointing my browser at http://localhost:9999 I get the same timeout again. Have I set up everything right?
Also, following good advice I copied the config changes to hive-site.xml. How do I stop hive so I can restart with 'hive -service hwi' so that it picks up the new config?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:31
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Port forwarding is the same as what Dean referred to as ssh tunneling.
Here is a website explaining how to set up port forwarding in putty (use port 9999):
http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
You can edit either hive-default or hive-site but the general practice is to store changes from the default configuration in hive-site.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi Ariel,
Thanks for the speedy reply. We'll be accessing the HWI from Windows rather Linux desktops, so can you help me out with understanding how to carry out the port forwarding from a Windows environment? Currently I use WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY session.
The changes to the hive config look identical to what I've changed in my hive-default.xml. Does it make any difference which config file the changesare applied to (hive-default.xml vs hive-site.xml)?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi publicly available on the internet. When you connect to the master node using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
--
Nitin Pawar
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
Re: HWI use on AWS/EMR
Posted by Ariel Marcus <ar...@openbi.com>.
Hey Tony,
Nitin's approach would work for you as well. Amazon has very good
documentation of their services. A quick google search should get you the
right info.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com | ariel.marcus@openbi.com
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 12:17 PM, Tony Burton <TB...@sportingindex.com>wrote:
> If you could provide the steps to do this I’d be grateful – thanks.****
>
> ** **
>
> Thanks for all your input so far, time to head home for the weekend so
> I’ll try it again with a fresh EMR instance next week.****
>
> ** **
>
> Tony****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 18 January 2013 17:08
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ** **
>
> ok so much discussion around this but why dont you open access to port
> 9999 for your specific machine (home/office) and access it normally without
> setting up port forwarding. ****
>
> ** **
>
> AWS does give you ip level access control in the security groups ****
>
> ** **
>
> On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>
> wrote:****
>
> This should be localhost instead of proxy:****
>
> ** **
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box, *
> localhost*:9999 in the Destination box. Not sure if this is correct.****
>
> ** **
>
> Best,****
>
> Aril****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ** **
>
> On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> ****
>
> Ok, thanks Ariel (and Dean in the next email!)****
>
> ****
>
> I’ve set up PuTTY (v0.62) as follows:****
>
> - EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
> ****
>
> - In Connection/SSH/Auth I’ve added my private key file, downloaded from
> AWS ages for this account.****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box,
> proxy:9999 in the Destination box. Not sure if this is correct.****
>
> ****
>
> Hit the Open button, I can connect. (And I can see my changes in
> hive-default.xml, so I know I’m in the right place:)****
>
> Then pointing my browser at http://localhost:9999 I get the same timeout
> again. Have I set up everything right?****
>
> ****
>
> Also, following good advice I copied the config changes to hive-site.xml.
> How do I stop hive so I can restart with ‘hive –service hwi’ so that it
> picks up the new config?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:31****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> Port forwarding is the same as what Dean referred to as ssh tunneling.****
>
> ****
>
> Here is a website explaining how to set up port forwarding in putty (use
> port 9999):****
>
> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html****
>
> ****
>
> You can edit either hive-default or hive-site but the general practice is
> to store changes from the default configuration in hive-site.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi Ariel,****
>
> ****
>
> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
> rather Linux desktops, so can you help me out with understanding how to
> carry out the port forwarding from a Windows environment? Currently I use
> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
> session.****
>
> ****
>
> The changes to the hive config look identical to what I’ve changed in my
> hive-default.xml. Does it make any difference which config file the
> changesare applied to (hive-default.xml vs hive-site.xml)?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:16****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> I would recommend using ssh port forwarding instead of making your hwi
> publicly available on the internet. When you connect to the master node
> using ssh you should use a command like the following:****
>
> ****
>
> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
> MASTER_HOSTNAME****
>
> ****
>
> After you have connected, add the following to your
> hive/conf/hive-site.xml file:****
>
> ****
>
> <property>****
>
> <name>hive.hwi.war.file</name>****
>
> <value>lib/hive-hwi-0.8.1.war</value>****
>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>****
>
> </property>****
>
> ****
>
> Run this command to start up hwi:****
>
> ****
>
> hive --service hwi****
>
> ****
>
> And finally point your browser to:****
>
> ****
>
> localhost:9999****
>
> ****
>
> That worked for me.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
> ****
>
> ****
>
> If it's blocked you can use ssh to tunnel the port through.****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
> ****
>
> ****
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged. If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system. Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened. It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email. Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
> Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001). Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
>
>
> *****************************************************************************
> P *Please consider the environment before printing this email* ****
>
>
> www.sportingindex.com
>
> Inbound email has been scanned for viruses & spam****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ** **
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.
>
--
------------------------------
This transmission is confidential and intended solely for the use of the
recipient named above. It may contain confidential, proprietary, or legally
privileged information. If you are not the intended recipient, you are
hereby notified that any unauthorized review, use, disclosure or
distribution is strictly prohibited. If you have received this transmission
in error, please contact the sender by reply e-mail and delete the original
transmission and all copies from your system.
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
If you could provide the steps to do this I'd be grateful - thanks.
Thanks for all your input so far, time to head home for the weekend so I'll try it again with a fresh EMR instance next week.
Tony
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: 18 January 2013 17:08
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
ok so much discussion around this but why dont you open access to port 9999 for your specific machine (home/office) and access it normally without setting up port forwarding.
AWS does give you ip level access control in the security groups
On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>> wrote:
This should be localhost instead of proxy:
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, localhost:9999 in the Destination box. Not sure if this is correct.
Best,
Aril
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>> wrote:
Ok, thanks Ariel (and Dean in the next email!)
I've set up PuTTY (v0.62) as follows:
- EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
- In Connection/SSH/Auth I've added my private key file, downloaded from AWS ages for this account.
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, proxy:9999 in the Destination box. Not sure if this is correct.
Hit the Open button, I can connect. (And I can see my changes in hive-default.xml, so I know I'm in the right place:)
Then pointing my browser at http://localhost:9999 I get the same timeout again. Have I set up everything right?
Also, following good advice I copied the config changes to hive-site.xml. How do I stop hive so I can restart with 'hive -service hwi' so that it picks up the new config?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:31
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Port forwarding is the same as what Dean referred to as ssh tunneling.
Here is a website explaining how to set up port forwarding in putty (use port 9999):
http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
You can edit either hive-default or hive-site but the general practice is to store changes from the default configuration in hive-site.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi Ariel,
Thanks for the speedy reply. We'll be accessing the HWI from Windows rather Linux desktops, so can you help me out with understanding how to carry out the port forwarding from a Windows environment? Currently I use WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY session.
The changes to the hive config look identical to what I've changed in my hive-default.xml. Does it make any difference which config file the changesare applied to (hive-default.xml vs hive-site.xml)?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi publicly available on the internet. When you connect to the master node using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
--
Nitin Pawar
Re: HWI use on AWS/EMR
Posted by Nitin Pawar <ni...@gmail.com>.
ok so much discussion around this but why dont you open access to port 9999
for your specific machine (home/office) and access it normally without
setting up port forwarding.
AWS does give you ip level access control in the security groups
On Fri, Jan 18, 2013 at 10:35 PM, Ariel Marcus <ar...@openbi.com>wrote:
> This should be localhost instead of proxy:
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box, *
> localhost*:9999 in the Destination box. Not sure if this is correct.
>
> Best,
> Aril
>
> ---------------------------------
> Ariel Marcus, Consultant
> www.openbi.com | ariel.marcus@openbi.com
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356
>
>
> On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>wrote:
>
>> ** **
>>
>> Ok, thanks Ariel (and Dean in the next email!)****
>>
>> ** **
>>
>> I’ve set up PuTTY (v0.62) as follows:****
>>
>> - EMR instance hostname in the Session/Host Name box (port left at 22,
>> ok?)****
>>
>> - In Connection/SSH/Auth I’ve added my private key file, downloaded from
>> AWS ages for this account.****
>>
>> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box,
>> proxy:9999 in the Destination box. Not sure if this is correct.****
>>
>> ** **
>>
>> Hit the Open button, I can connect. (And I can see my changes in
>> hive-default.xml, so I know I’m in the right place:)****
>>
>> Then pointing my browser at http://localhost:9999 I get the same timeout
>> again. Have I set up everything right?****
>>
>> ** **
>>
>> Also, following good advice I copied the config changes to hive-site.xml.
>> How do I stop hive so I can restart with ‘hive –service hwi’ so that it
>> picks up the new config?****
>>
>> ** **
>>
>> Tony****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
>> *Sent:* 18 January 2013 16:31
>>
>> *To:* user@hive.apache.org
>> *Subject:* Re: HWI use on AWS/EMR****
>>
>> ** **
>>
>> Hey Tony,****
>>
>> ** **
>>
>> Port forwarding is the same as what Dean referred to as ssh tunneling.***
>> *
>>
>> ** **
>>
>> Here is a website explaining how to set up port forwarding in putty (use
>> port 9999):****
>>
>> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html****
>>
>> ** **
>>
>> You can edit either hive-default or hive-site but the general practice is
>> to store changes from the default configuration in hive-site.****
>>
>> ** **
>>
>> Best,****
>>
>> Ariel****
>>
>> ** **
>>
>>
>> ****
>>
>> ---------------------------------****
>>
>> Ariel Marcus, Consultant****
>>
>> www.openbi.com | ariel.marcus@openbi.com****
>>
>> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
>> Cell: 314-827-4356****
>>
>> ** **
>>
>> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>
>> wrote:****
>>
>> Hi Ariel,****
>>
>> ****
>>
>> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
>> rather Linux desktops, so can you help me out with understanding how to
>> carry out the port forwarding from a Windows environment? Currently I use
>> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
>> session.****
>>
>> ****
>>
>> The changes to the hive config look identical to what I’ve changed in my
>> hive-default.xml. Does it make any difference which config file the
>> changesare applied to (hive-default.xml vs hive-site.xml)?****
>>
>> ****
>>
>> Tony****
>>
>> ****
>>
>> ****
>>
>> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
>> *Sent:* 18 January 2013 16:16****
>>
>>
>> *To:* user@hive.apache.org
>> *Subject:* Re: HWI use on AWS/EMR****
>>
>> ****
>>
>> Hey Tony,****
>>
>> ****
>>
>> I would recommend using ssh port forwarding instead of making your hwi
>> publicly available on the internet. When you connect to the master node
>> using ssh you should use a command like the following:****
>>
>> ****
>>
>> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
>> MASTER_HOSTNAME****
>>
>> ****
>>
>> After you have connected, add the following to your
>> hive/conf/hive-site.xml file:****
>>
>> ****
>>
>> <property>****
>>
>> <name>hive.hwi.war.file</name>****
>>
>> <value>lib/hive-hwi-0.8.1.war</value>****
>>
>> <description>This is the WAR file with the jsp content for Hive Web
>> Interface</description>****
>>
>> </property>****
>>
>> ****
>>
>> Run this command to start up hwi:****
>>
>> ****
>>
>> hive --service hwi****
>>
>> ****
>>
>> And finally point your browser to:****
>>
>> ****
>>
>> localhost:9999****
>>
>> ****
>>
>> That worked for me.****
>>
>> ****
>>
>> Best,****
>>
>> Ariel****
>>
>> ****
>>
>> ****
>>
>> ****
>>
>> ****
>>
>> ****
>>
>>
>> ****
>>
>> ---------------------------------****
>>
>> Ariel Marcus, Consultant****
>>
>> www.openbi.com | ariel.marcus@openbi.com****
>>
>> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
>> Cell: 314-827-4356****
>>
>> ****
>>
>> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
>> dean.wampler@thinkbiganalytics.com> wrote:****
>>
>> Oops, I overlooked that you have the public domain name in your message.
>> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
>> ****
>>
>> ****
>>
>> If it's blocked you can use ssh to tunnel the port through.****
>>
>> ****
>>
>> dean****
>>
>> ****
>>
>> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
>> dean.wampler@thinkbiganalytics.com> wrote:****
>>
>> That's the internal hostname, not visible outside. Use the name like
>> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
>> and the elastic-mapreduce script you might have used to launch the cluster.
>> ****
>>
>> ****
>>
>> If that doesn't work, verify that port 80 is not blocked by default.
>> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>>
>> ****
>>
>> dean****
>>
>> ****
>>
>> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
>> wrote:****
>>
>> Hi,
>>
>> I'm trying to get HWI running and accessible from an Amazon Web Services
>> EMR instance. I've hit a blocker early on though, and the documentation is
>> less than illuminating. Can you share any experiences you have had?
>> Specifically, here's what I'm curious about.
>>
>> - Running on AWS. I've created a Hive job flow on AWS, edited
>> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
>> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
>> - HWI starts up fine, but when I try to connect with
>> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
>> timeout message, in Firefox it's "The connection has timed out - The server
>> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
>> to respond".
>>
>> Has anyone successfully connected to HWI running on an AWS EMR instance?
>> From the same browser I can connect to HWI on a local Ubuntu box.
>>
>> Thanks!
>>
>> Tony
>>
>>
>> **********************************************************************
>> Please consider the environment before printing this email or attachments
>>
>> This email and any attachments are confidential, protected by copyright
>> and may be legally privileged. If you are not the intended recipient, then
>> the dissemination or copying of this email is prohibited. If you have
>> received this in error, please notify the sender by replying by email and
>> then delete the email completely from your system. Neither Sporting Index
>> nor the sender accepts responsibility for any virus, or any other defect
>> which might affect any computer or IT system into which the email is
>> received and/or opened. It is the responsibility of the recipient to scan
>> the email and no responsibility is accepted for any loss or damage arising
>> in any way from receipt or use of this email. Sporting Index Ltd is a
>> company registered in England and Wales with company number 2636842, whose
>> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
>> Sporting Index Ltd is authorised and regulated by the UK Financial
>> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
>> 000-027343-R-308898-001). Any financial promotion contained herein has
>> been issued
>> and approved by Sporting Index Ltd.
>>
>> Outbound email has been scanned for viruses and SPAM****
>>
>>
>>
>> ****
>>
>> ****
>>
>> --
>> *Dean Wampler, Ph.D.*****
>>
>> thinkbiganalytics.com****
>>
>> +1-312-339-1330****
>>
>> ****
>>
>>
>>
>> ****
>>
>> ****
>>
>> --
>> *Dean Wampler, Ph.D.*****
>>
>> thinkbiganalytics.com****
>>
>> +1-312-339-1330****
>>
>> ****
>>
>> ****
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.****
>>
>> ****
>>
>> ****
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.****
>>
>> ****
>>
>>
>>
>> *****************************************************************************
>> P *Please consider the environment before printing this email* ****
>>
>>
>> www.sportingindex.com
>>
>> Inbound email has been scanned for viruses & spam****
>>
>> ** **
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.****
>>
>> ** **
>>
>> ** **
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.****
>>
>> ------------------------------
>>
>> This transmission is confidential and intended solely for the use of the
>> recipient named above. It may contain confidential, proprietary, or legally
>> privileged information. If you are not the intended recipient, you are
>> hereby notified that any unauthorized review, use, disclosure or
>> distribution is strictly prohibited. If you have received this transmission
>> in error, please contact the sender by reply e-mail and delete the original
>> transmission and all copies from your system.
>>
>
>
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.
>
--
Nitin Pawar
Re: HWI use on AWS/EMR
Posted by Ariel Marcus <ar...@openbi.com>.
This should be localhost instead of proxy:
- In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box, *
localhost*:9999 in the Destination box. Not sure if this is correct.
Best,
Aril
---------------------------------
Ariel Marcus, Consultant
www.openbi.com | ariel.marcus@openbi.com
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 12:00 PM, Tony Burton <TB...@sportingindex.com>wrote:
> ** **
>
> Ok, thanks Ariel (and Dean in the next email!)****
>
> ** **
>
> I’ve set up PuTTY (v0.62) as follows:****
>
> - EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
> ****
>
> - In Connection/SSH/Auth I’ve added my private key file, downloaded from
> AWS ages for this account.****
>
> - In Connection/SSH/Tunnels I’ve added 9999 in the Source Port box,
> proxy:9999 in the Destination box. Not sure if this is correct.****
>
> ** **
>
> Hit the Open button, I can connect. (And I can see my changes in
> hive-default.xml, so I know I’m in the right place:)****
>
> Then pointing my browser at http://localhost:9999 I get the same timeout
> again. Have I set up everything right?****
>
> ** **
>
> Also, following good advice I copied the config changes to hive-site.xml.
> How do I stop hive so I can restart with ‘hive –service hwi’ so that it
> picks up the new config?****
>
> ** **
>
> Tony****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:31
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ** **
>
> Hey Tony,****
>
> ** **
>
> Port forwarding is the same as what Dean referred to as ssh tunneling.****
>
> ** **
>
> Here is a website explaining how to set up port forwarding in putty (use
> port 9999):****
>
> http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html****
>
> ** **
>
> You can edit either hive-default or hive-site but the general practice is
> to store changes from the default configuration in hive-site.****
>
> ** **
>
> Best,****
>
> Ariel****
>
> ** **
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ** **
>
> On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi Ariel,****
>
> ****
>
> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
> rather Linux desktops, so can you help me out with understanding how to
> carry out the port forwarding from a Windows environment? Currently I use
> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
> session.****
>
> ****
>
> The changes to the hive config look identical to what I’ve changed in my
> hive-default.xml. Does it make any difference which config file the
> changesare applied to (hive-default.xml vs hive-site.xml)?****
>
> ****
>
> Tony****
>
> ****
>
> ****
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:16****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ****
>
> Hey Tony,****
>
> ****
>
> I would recommend using ssh port forwarding instead of making your hwi
> publicly available on the internet. When you connect to the master node
> using ssh you should use a command like the following:****
>
> ****
>
> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
> MASTER_HOSTNAME****
>
> ****
>
> After you have connected, add the following to your
> hive/conf/hive-site.xml file:****
>
> ****
>
> <property>****
>
> <name>hive.hwi.war.file</name>****
>
> <value>lib/hive-hwi-0.8.1.war</value>****
>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>****
>
> </property>****
>
> ****
>
> Run this command to start up hwi:****
>
> ****
>
> hive --service hwi****
>
> ****
>
> And finally point your browser to:****
>
> ****
>
> localhost:9999****
>
> ****
>
> That worked for me.****
>
> ****
>
> Best,****
>
> Ariel****
>
> ****
>
> ****
>
> ****
>
> ****
>
> ****
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ****
>
> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
> ****
>
> ****
>
> If it's blocked you can use ssh to tunnel the port through.****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
> ****
>
> ****
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>
> ****
>
> dean****
>
> ****
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged. If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system. Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened. It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email. Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
> Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001). Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
>
>
> ****
>
> ****
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
> ****
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ****
>
>
>
> *****************************************************************************
> P *Please consider the environment before printing this email* ****
>
>
> www.sportingindex.com
>
> Inbound email has been scanned for viruses & spam****
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ** **
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.
>
--
------------------------------
This transmission is confidential and intended solely for the use of the
recipient named above. It may contain confidential, proprietary, or legally
privileged information. If you are not the intended recipient, you are
hereby notified that any unauthorized review, use, disclosure or
distribution is strictly prohibited. If you have received this transmission
in error, please contact the sender by reply e-mail and delete the original
transmission and all copies from your system.
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
Ok, thanks Ariel (and Dean in the next email!)
I've set up PuTTY (v0.62) as follows:
- EMR instance hostname in the Session/Host Name box (port left at 22, ok?)
- In Connection/SSH/Auth I've added my private key file, downloaded from AWS ages for this account.
- In Connection/SSH/Tunnels I've added 9999 in the Source Port box, proxy:9999 in the Destination box. Not sure if this is correct.
Hit the Open button, I can connect. (And I can see my changes in hive-default.xml, so I know I'm in the right place:)
Then pointing my browser at http://localhost:9999 I get the same timeout again. Have I set up everything right?
Also, following good advice I copied the config changes to hive-site.xml. How do I stop hive so I can restart with 'hive -service hwi' so that it picks up the new config?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com]
Sent: 18 January 2013 16:31
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
Hey Tony,
Port forwarding is the same as what Dean referred to as ssh tunneling.
Here is a website explaining how to set up port forwarding in putty (use port 9999):
http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
You can edit either hive-default or hive-site but the general practice is to store changes from the default configuration in hive-site.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi Ariel,
Thanks for the speedy reply. We'll be accessing the HWI from Windows rather Linux desktops, so can you help me out with understanding how to carry out the port forwarding from a Windows environment? Currently I use WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY session.
The changes to the hive config look identical to what I've changed in my hive-default.xml. Does it make any difference which config file the changesare applied to (hive-default.xml vs hive-site.xml)?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com<ma...@openbi.com>]
Sent: 18 January 2013 16:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: HWI use on AWS/EMR
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi publicly available on the internet. When you connect to the master node using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356<tel:314-827-4356>
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
Re: HWI use on AWS/EMR
Posted by Ariel Marcus <ar...@openbi.com>.
Hey Tony,
Port forwarding is the same as what Dean referred to as ssh tunneling.
Here is a website explaining how to set up port forwarding in putty (use
port 9999):
http://www.cs.uu.nl/technical/services/ssh/putty/puttyfw.html
You can edit either hive-default or hive-site but the general practice is
to store changes from the default configuration in hive-site.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com | ariel.marcus@openbi.com
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 11:26 AM, Tony Burton <TB...@sportingindex.com>wrote:
> Hi Ariel,****
>
> ** **
>
> Thanks for the speedy reply. We’ll be accessing the HWI from Windows
> rather Linux desktops, so can you help me out with understanding how to
> carry out the port forwarding from a Windows environment? Currently I use
> WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY
> session.****
>
> ** **
>
> The changes to the hive config look identical to what I’ve changed in my
> hive-default.xml. Does it make any difference which config file the
> changesare applied to (hive-default.xml vs hive-site.xml)?****
>
> ** **
>
> Tony****
>
> ** **
>
> ** **
>
> *From:* Ariel Marcus [mailto:ariel.marcus@openbi.com]
> *Sent:* 18 January 2013 16:16
>
> *To:* user@hive.apache.org
> *Subject:* Re: HWI use on AWS/EMR****
>
> ** **
>
> Hey Tony,****
>
> ** **
>
> I would recommend using ssh port forwarding instead of making your hwi
> publicly available on the internet. When you connect to the master node
> using ssh you should use a command like the following:****
>
> ** **
>
> ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop
> MASTER_HOSTNAME****
>
> ** **
>
> After you have connected, add the following to your
> hive/conf/hive-site.xml file:****
>
> ** **
>
> <property>****
>
> <name>hive.hwi.war.file</name>****
>
> <value>lib/hive-hwi-0.8.1.war</value>****
>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>****
>
> </property>****
>
> ** **
>
> Run this command to start up hwi:****
>
> ** **
>
> hive --service hwi****
>
> ** **
>
> And finally point your browser to:****
>
> ** **
>
> localhost:9999****
>
> ** **
>
> That worked for me.****
>
> ** **
>
> Best,****
>
> Ariel****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
>
> ****
>
> ---------------------------------****
>
> Ariel Marcus, Consultant****
>
> www.openbi.com | ariel.marcus@openbi.com****
>
> 150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
> Cell: 314-827-4356****
>
> ** **
>
> On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
> ****
>
> ** **
>
> If it's blocked you can use ssh to tunnel the port through.****
>
> ** **
>
> dean****
>
> ** **
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:****
>
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
> ****
>
> ** **
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc. ****
>
> ** **
>
> dean****
>
> ** **
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>
> wrote:****
>
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged. If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system. Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened. It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email. Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
> Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001). Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM****
>
>
>
> ****
>
> ** **
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ** **
>
>
>
> ****
>
> ** **
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ** **
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ** **
>
> ** **
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.****
>
> ** **
>
>
>
> *****************************************************************************
> P *Please consider the environment before printing this email* ****
>
>
> www.sportingindex.com
>
> Inbound email has been scanned for viruses & spam****
>
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.
>
--
------------------------------
This transmission is confidential and intended solely for the use of the
recipient named above. It may contain confidential, proprietary, or legally
privileged information. If you are not the intended recipient, you are
hereby notified that any unauthorized review, use, disclosure or
distribution is strictly prohibited. If you have received this transmission
in error, please contact the sender by reply e-mail and delete the original
transmission and all copies from your system.
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
Hi Ariel,
Thanks for the speedy reply. We'll be accessing the HWI from Windows rather Linux desktops, so can you help me out with understanding how to carry out the port forwarding from a Windows environment? Currently I use WinSCP to initiate a connection to the EMR instance, then spawn a PuTTY session.
The changes to the hive config look identical to what I've changed in my hive-default.xml. Does it make any difference which config file the changesare applied to (hive-default.xml vs hive-site.xml)?
Tony
From: Ariel Marcus [mailto:ariel.marcus@openbi.com]
Sent: 18 January 2013 16:16
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi publicly available on the internet. When you connect to the master node using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com<http://www.openbi.com/> | ariel.marcus@openbi.com<ma...@openbi.com>
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
________________________________
This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
Re: HWI use on AWS/EMR
Posted by Ariel Marcus <ar...@openbi.com>.
Hey Tony,
I would recommend using ssh port forwarding instead of making your hwi
publicly available on the internet. When you connect to the master node
using ssh you should use a command like the following:
ssh -i /path/to/keyfile.pem -L 9999:localhost:9999 -l hadoop MASTER_HOSTNAME
After you have connected, add the following to your hive/conf/hive-site.xml
file:
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.8.1.war</value>
<description>This is the WAR file with the jsp content for Hive Web
Interface</description>
</property>
Run this command to start up hwi:
hive --service hwi
And finally point your browser to:
localhost:9999
That worked for me.
Best,
Ariel
---------------------------------
Ariel Marcus, Consultant
www.openbi.com | ariel.marcus@openbi.com
150 N Michigan Avenue, Suite 2800, Chicago, IL 60601
Cell: 314-827-4356
On Fri, Jan 18, 2013 at 11:09 AM, Dean Wampler <
dean.wampler@thinkbiganalytics.com> wrote:
> Oops, I overlooked that you have the public domain name in your message.
> Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
>
> If it's blocked you can use ssh to tunnel the port through.
>
> dean
>
>
> On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
> dean.wampler@thinkbiganalytics.com> wrote:
>
>> That's the internal hostname, not visible outside. Use the name like
>> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
>> and the elastic-mapreduce script you might have used to launch the cluster.
>>
>> If that doesn't work, verify that port 80 is not blocked by default.
>> That's certainly true for ports 9XXX used by the JobTracker, etc.
>>
>> dean
>>
>>
>> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>wrote:
>>
>>> Hi,
>>>
>>> I'm trying to get HWI running and accessible from an Amazon Web Services
>>> EMR instance. I've hit a blocker early on though, and the documentation is
>>> less than illuminating. Can you share any experiences you have had?
>>> Specifically, here's what I'm curious about.
>>>
>>> - Running on AWS. I've created a Hive job flow on AWS, edited
>>> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
>>> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
>>> - HWI starts up fine, but when I try to connect with
>>> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
>>> timeout message, in Firefox it's "The connection has timed out - The server
>>> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
>>> to respond".
>>>
>>> Has anyone successfully connected to HWI running on an AWS EMR instance?
>>> From the same browser I can connect to HWI on a local Ubuntu box.
>>>
>>> Thanks!
>>>
>>> Tony
>>>
>>>
>>> **********************************************************************
>>> Please consider the environment before printing this email or attachments
>>>
>>> This email and any attachments are confidential, protected by copyright
>>> and may be legally privileged. If you are not the intended recipient, then
>>> the dissemination or copying of this email is prohibited. If you have
>>> received this in error, please notify the sender by replying by email and
>>> then delete the email completely from your system. Neither Sporting Index
>>> nor the sender accepts responsibility for any virus, or any other defect
>>> which might affect any computer or IT system into which the email is
>>> received and/or opened. It is the responsibility of the recipient to scan
>>> the email and no responsibility is accepted for any loss or damage arising
>>> in any way from receipt or use of this email. Sporting Index Ltd is a
>>> company registered in England and Wales with company number 2636842, whose
>>> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
>>> Sporting Index Ltd is authorised and regulated by the UK Financial
>>> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
>>> 000-027343-R-308898-001). Any financial promotion contained herein has
>>> been issued
>>> and approved by Sporting Index Ltd.
>>>
>>> Outbound email has been scanned for viruses and SPAM
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
> ------------------------------
>
> This transmission is confidential and intended solely for the use of the
> recipient named above. It may contain confidential, proprietary, or legally
> privileged information. If you are not the intended recipient, you are
> hereby notified that any unauthorized review, use, disclosure or
> distribution is strictly prohibited. If you have received this transmission
> in error, please contact the sender by reply e-mail and delete the original
> transmission and all copies from your system.
>
--
------------------------------
This transmission is confidential and intended solely for the use of the
recipient named above. It may contain confidential, proprietary, or legally
privileged information. If you are not the intended recipient, you are
hereby notified that any unauthorized review, use, disclosure or
distribution is strictly prohibited. If you have received this transmission
in error, please contact the sender by reply e-mail and delete the original
transmission and all copies from your system.
RE: HWI use on AWS/EMR
Posted by Tony Burton <TB...@SportingIndex.com>.
Thanks for the quick reponse Dean.
Confirmed that http://ec2-XX-AAA-BB-CCC.eu-west-1.compute.amazonaws.com gives the same timeout.
Doesn't specifying http://<hostname>:9999<http://%3chostname%3e:9999> override the requirement for port 80 being open?
Is ssh tunnelling the same as the process described by Ariel in the next reply?
From: Dean Wampler [mailto:dean.wampler@thinkbiganalytics.com]
Sent: 18 January 2013 16:09
To: user@hive.apache.org
Subject: Re: HWI use on AWS/EMR
Oops, I overlooked that you have the public domain name in your message. Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ? If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <de...@thinkbiganalytics.com>> wrote:
That's the internal hostname, not visible outside. Use the name like ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com<http://ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com>. It's shown in the EMR console and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>> wrote:
Hi,
I'm trying to get HWI running and accessible from an Amazon Web Services EMR instance. I've hit a blocker early on though, and the documentation is less than illuminating. Can you share any experiences you have had? Specifically, here's what I'm curious about.
- Running on AWS. I've created a Hive job flow on AWS, edited hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
- HWI starts up fine, but when I try to connect with http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a timeout message, in Firefox it's "The connection has timed out - The server at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com<http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com> is taking too long to respond".
Has anyone successfully connected to HWI running on an AWS EMR instance? From the same browser I can connect to HWI on a local Ubuntu box.
Thanks!
Tony
**********************************************************************
Please consider the environment before printing this email or attachments
This email and any attachments are confidential, protected by copyright and may be legally privileged. If you are not the intended recipient, then the dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Neither Sporting Index nor the sender accepts responsibility for any virus, or any other defect which might affect any computer or IT system into which the email is received and/or opened. It is the responsibility of the recipient to scan the email and no responsibility is accepted for any loss or damage arising in any way from receipt or use of this email. Sporting Index Ltd is a company registered in England and Wales with company number 2636842, whose registered office is at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is authorised and regulated by the UK Financial Services Authority (reg. no. 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any financial promotion contained herein has been issued
and approved by Sporting Index Ltd.
Outbound email has been scanned for viruses and SPAM
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330<tel:%2B1-312-339-1330>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com<http://thinkbiganalytics.com>
+1-312-339-1330
*****************************************************************************
P Please consider the environment before printing this email
www.sportingindex.com<http://www.sportingindex.com>
Inbound email has been scanned for viruses & spam
Re: HWI use on AWS/EMR
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
Oops, I overlooked that you have the public domain name in your message.
Can you surf to http://ec2-54-247-61-206.eu-west-1.compute.amazonaws.com ?
If not, does HWI use port 80? Is whatever port it uses blocked by EC2?
If it's blocked you can use ssh to tunnel the port through.
dean
On Fri, Jan 18, 2013 at 10:06 AM, Dean Wampler <
dean.wampler@thinkbiganalytics.com> wrote:
> That's the internal hostname, not visible outside. Use the name like
> ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
> and the elastic-mapreduce script you might have used to launch the cluster.
>
> If that doesn't work, verify that port 80 is not blocked by default.
> That's certainly true for ports 9XXX used by the JobTracker, etc.
>
> dean
>
>
> On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>wrote:
>
>> Hi,
>>
>> I'm trying to get HWI running and accessible from an Amazon Web Services
>> EMR instance. I've hit a blocker early on though, and the documentation is
>> less than illuminating. Can you share any experiences you have had?
>> Specifically, here's what I'm curious about.
>>
>> - Running on AWS. I've created a Hive job flow on AWS, edited
>> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
>> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
>> - HWI starts up fine, but when I try to connect with
>> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
>> timeout message, in Firefox it's "The connection has timed out - The server
>> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
>> to respond".
>>
>> Has anyone successfully connected to HWI running on an AWS EMR instance?
>> From the same browser I can connect to HWI on a local Ubuntu box.
>>
>> Thanks!
>>
>> Tony
>>
>>
>> **********************************************************************
>> Please consider the environment before printing this email or attachments
>>
>> This email and any attachments are confidential, protected by copyright
>> and may be legally privileged. If you are not the intended recipient, then
>> the dissemination or copying of this email is prohibited. If you have
>> received this in error, please notify the sender by replying by email and
>> then delete the email completely from your system. Neither Sporting Index
>> nor the sender accepts responsibility for any virus, or any other defect
>> which might affect any computer or IT system into which the email is
>> received and/or opened. It is the responsibility of the recipient to scan
>> the email and no responsibility is accepted for any loss or damage arising
>> in any way from receipt or use of this email. Sporting Index Ltd is a
>> company registered in England and Wales with company number 2636842, whose
>> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
>> Sporting Index Ltd is authorised and regulated by the UK Financial
>> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
>> 000-027343-R-308898-001). Any financial promotion contained herein has
>> been issued
>> and approved by Sporting Index Ltd.
>>
>> Outbound email has been scanned for viruses and SPAM
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: HWI use on AWS/EMR
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
That's the internal hostname, not visible outside. Use the name like
ec2-NNN-NN-NN-NNN.compute-1.amazonaws.com. It's shown in the EMR console
and the elastic-mapreduce script you might have used to launch the cluster.
If that doesn't work, verify that port 80 is not blocked by default. That's
certainly true for ports 9XXX used by the JobTracker, etc.
dean
On Fri, Jan 18, 2013 at 9:54 AM, Tony Burton <TB...@sportingindex.com>wrote:
> Hi,
>
> I'm trying to get HWI running and accessible from an Amazon Web Services
> EMR instance. I've hit a blocker early on though, and the documentation is
> less than illuminating. Can you share any experiences you have had?
> Specifically, here's what I'm curious about.
>
> - Running on AWS. I've created a Hive job flow on AWS, edited
> hive-default.xml (in /home/hadoop/.versions/hive-0.8.1/conf) and changed
> the hive.hwi.war.file to lib/hive-hwi-0.8.1.war.
> - HWI starts up fine, but when I try to connect with
> http://ip-XX-AAA-BBB-CCC.eu-west-1.compute.internal:9999/hwi, I get a
> timeout message, in Firefox it's "The connection has timed out - The server
> at ec2-54-247-61-206.eu-west-1.compute.amazonaws.com is taking too long
> to respond".
>
> Has anyone successfully connected to HWI running on an AWS EMR instance?
> From the same browser I can connect to HWI on a local Ubuntu box.
>
> Thanks!
>
> Tony
>
>
> **********************************************************************
> Please consider the environment before printing this email or attachments
>
> This email and any attachments are confidential, protected by copyright
> and may be legally privileged. If you are not the intended recipient, then
> the dissemination or copying of this email is prohibited. If you have
> received this in error, please notify the sender by replying by email and
> then delete the email completely from your system. Neither Sporting Index
> nor the sender accepts responsibility for any virus, or any other defect
> which might affect any computer or IT system into which the email is
> received and/or opened. It is the responsibility of the recipient to scan
> the email and no responsibility is accepted for any loss or damage arising
> in any way from receipt or use of this email. Sporting Index Ltd is a
> company registered in England and Wales with company number 2636842, whose
> registered office is at Gateway House, Milverton Street, London, SE11 4AP.
> Sporting Index Ltd is authorised and regulated by the UK Financial
> Services Authority (reg. no. 150404) and Gambling Commission (reg. no.
> 000-027343-R-308898-001). Any financial promotion contained herein has
> been issued
> and approved by Sporting Index Ltd.
>
> Outbound email has been scanned for viruses and SPAM
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330