You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Something Something <ma...@gmail.com> on 2012/01/15 09:16:35 UTC
Starting Map Reduce Job on EC2
Hello,
Our Hadoop cluster is setup on EC2, but our client machine which will
trigger the M/R job is in our data center. I am trying to start a M/R job
from our client machine, but getting this:
00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server:
ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried
0 time(s).
00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server:
ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried
1 time(s).
00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server:
ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried
2 time(s).
& so on...
I tried putting our 'pem' file in the classpath, but that didn't help. I
am assuming it should be possible to trigger a job on EC2 cluster from a
machine that's not in EC2, correct?
Please note that I can trigger a M/R job from the client machine to a
Hadoop running on the same machine - localhost:9000 (hdfs) as well as
localhost:9001 (jobtracker).
Please help. Thanks.
Re: Starting Map Reduce Job on EC2
Posted by Harsh J <ha...@cloudera.com>.
Best to use a secured Hadoop cluster [0], and/or setup appropriate
firewall rules that block traffic from other than your trusted IPs.
[0] - https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide
On Mon, Jan 16, 2012 at 4:33 AM, Something Something
<ma...@gmail.com> wrote:
> Good point. Those ports may not be open. So next question - is it safe to
> open these ports? How do we securely open these ports to avoid malicious
> attacks under EC2?
>
> (Sorry, I know some of these questions are dumb - but we are a startup and
> don't have a big sysadmin group - I guess that's why we are using EC2 -:)
>
>
> On Sun, Jan 15, 2012 at 12:03 PM, Ronald Petty <ro...@gmail.com>
> wrote:
>>
>> Something Something,
>>
>> Have you confirmed you can connect to the port from your remote machine?
>>
>> telnet ec2-xxxxxx.... 9000
>>
>> Kindest regards.
>>
>> Ron
>>
>> On Sun, Jan 15, 2012 at 12:16 AM, Something Something
>> <ma...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Our Hadoop cluster is setup on EC2, but our client machine which will
>>> trigger the M/R job is in our data center. I am trying to start a M/R job
>>> from our client machine, but getting this:
>>>
>>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>>> Retrying connect to server:
>>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried 0
>>> time(s).
>>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>>> Retrying connect to server:
>>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried 1
>>> time(s).
>>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>>> Retrying connect to server:
>>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried 2
>>> time(s).
>>> & so on...
>>>
>>>
>>> I tried putting our 'pem' file in the classpath, but that didn't help. I
>>> am assuming it should be possible to trigger a job on EC2 cluster from a
>>> machine that's not in EC2, correct?
>>>
>>> Please note that I can trigger a M/R job from the client machine to a
>>> Hadoop running on the same machine - localhost:9000 (hdfs) as well as
>>> localhost:9001 (jobtracker).
>>>
>>> Please help. Thanks.
>>>
>>>
>>>
>>>
>>
>
--
Harsh J
Customer Ops. Engineer, Cloudera
Re: Starting Map Reduce Job on EC2
Posted by Something Something <ma...@gmail.com>.
All monitoring browser ports.. such as????
On Sun, Jan 15, 2012 at 5:00 PM, Lance Norskog <go...@gmail.com> wrote:
> Can you open all of the monitoring browser ports?
>
> On Sun, Jan 15, 2012 at 3:03 PM, Something Something
> <ma...@gmail.com> wrote:
> > Good point. Those ports may not be open. So next question - is it safe
> to
> > open these ports? How do we securely open these ports to avoid malicious
> > attacks under EC2?
> >
> > (Sorry, I know some of these questions are dumb - but we are a startup
> and
> > don't have a big sysadmin group - I guess that's why we are using EC2 -:)
> >
> >
> > On Sun, Jan 15, 2012 at 12:03 PM, Ronald Petty <ro...@gmail.com>
> > wrote:
> >>
> >> Something Something,
> >>
> >> Have you confirmed you can connect to the port from your remote machine?
> >>
> >> telnet ec2-xxxxxx.... 9000
> >>
> >> Kindest regards.
> >>
> >> Ron
> >>
> >> On Sun, Jan 15, 2012 at 12:16 AM, Something Something
> >> <ma...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Our Hadoop cluster is setup on EC2, but our client machine which will
> >>> trigger the M/R job is in our data center. I am trying to start a M/R
> job
> >>> from our client machine, but getting this:
> >>>
> >>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> >>> Retrying connect to server:
> >>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 0
> >>> time(s).
> >>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> >>> Retrying connect to server:
> >>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 1
> >>> time(s).
> >>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> >>> Retrying connect to server:
> >>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 2
> >>> time(s).
> >>> & so on...
> >>>
> >>>
> >>> I tried putting our 'pem' file in the classpath, but that didn't
> help. I
> >>> am assuming it should be possible to trigger a job on EC2 cluster from
> a
> >>> machine that's not in EC2, correct?
> >>>
> >>> Please note that I can trigger a M/R job from the client machine to a
> >>> Hadoop running on the same machine - localhost:9000 (hdfs) as well as
> >>> localhost:9001 (jobtracker).
> >>>
> >>> Please help. Thanks.
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Starting Map Reduce Job on EC2
Posted by Lance Norskog <go...@gmail.com>.
Can you open all of the monitoring browser ports?
On Sun, Jan 15, 2012 at 3:03 PM, Something Something
<ma...@gmail.com> wrote:
> Good point. Those ports may not be open. So next question - is it safe to
> open these ports? How do we securely open these ports to avoid malicious
> attacks under EC2?
>
> (Sorry, I know some of these questions are dumb - but we are a startup and
> don't have a big sysadmin group - I guess that's why we are using EC2 -:)
>
>
> On Sun, Jan 15, 2012 at 12:03 PM, Ronald Petty <ro...@gmail.com>
> wrote:
>>
>> Something Something,
>>
>> Have you confirmed you can connect to the port from your remote machine?
>>
>> telnet ec2-xxxxxx.... 9000
>>
>> Kindest regards.
>>
>> Ron
>>
>> On Sun, Jan 15, 2012 at 12:16 AM, Something Something
>> <ma...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Our Hadoop cluster is setup on EC2, but our client machine which will
>>> trigger the M/R job is in our data center. I am trying to start a M/R job
>>> from our client machine, but getting this:
>>>
>>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>>> Retrying connect to server:
>>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried 0
>>> time(s).
>>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>>> Retrying connect to server:
>>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried 1
>>> time(s).
>>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>>> Retrying connect to server:
>>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already tried 2
>>> time(s).
>>> & so on...
>>>
>>>
>>> I tried putting our 'pem' file in the classpath, but that didn't help. I
>>> am assuming it should be possible to trigger a job on EC2 cluster from a
>>> machine that's not in EC2, correct?
>>>
>>> Please note that I can trigger a M/R job from the client machine to a
>>> Hadoop running on the same machine - localhost:9000 (hdfs) as well as
>>> localhost:9001 (jobtracker).
>>>
>>> Please help. Thanks.
>>>
>>>
>>>
>>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Starting Map Reduce Job on EC2
Posted by Something Something <ma...@gmail.com>.
Good point. Those ports may not be open. So next question - is it safe to
open these ports? How do we securely open these ports to avoid malicious
attacks under EC2?
(Sorry, I know some of these questions are dumb - but we are a startup and
don't have a big sysadmin group - I guess that's why we are using EC2 -:)
On Sun, Jan 15, 2012 at 12:03 PM, Ronald Petty <ro...@gmail.com>wrote:
> Something Something,
>
> Have you confirmed you can connect to the port from your remote machine?
>
> telnet ec2-xxxxxx.... 9000
>
> Kindest regards.
>
> Ron
>
> On Sun, Jan 15, 2012 at 12:16 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
>> Hello,
>>
>> Our Hadoop cluster is setup on EC2, but our client machine which will
>> trigger the M/R job is in our data center. I am trying to start a M/R job
>> from our client machine, but getting this:
>>
>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server:
>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
>> tried 0 time(s).
>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server:
>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
>> tried 1 time(s).
>> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server:
>> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
>> tried 2 time(s).
>> & so on...
>>
>>
>> I tried putting our 'pem' file in the classpath, but that didn't help. I
>> am assuming it should be possible to trigger a job on EC2 cluster from a
>> machine that's not in EC2, correct?
>>
>> Please note that I can trigger a M/R job from the client machine to a
>> Hadoop running on the same machine - localhost:9000 (hdfs) as well as
>> localhost:9001 (jobtracker).
>>
>> Please help. Thanks.
>>
>>
>>
>>
>>
>
Re: Starting Map Reduce Job on EC2
Posted by Ronald Petty <ro...@gmail.com>.
Something Something,
Have you confirmed you can connect to the port from your remote machine?
telnet ec2-xxxxxx.... 9000
Kindest regards.
Ron
On Sun, Jan 15, 2012 at 12:16 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Hello,
>
> Our Hadoop cluster is setup on EC2, but our client machine which will
> trigger the M/R job is in our data center. I am trying to start a M/R job
> from our client machine, but getting this:
>
> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server:
> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 0 time(s).
> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server:
> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 1 time(s).
> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server:
> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 2 time(s).
> & so on...
>
>
> I tried putting our 'pem' file in the classpath, but that didn't help. I
> am assuming it should be possible to trigger a job on EC2 cluster from a
> machine that's not in EC2, correct?
>
> Please note that I can trigger a M/R job from the client machine to a
> Hadoop running on the same machine - localhost:9000 (hdfs) as well as
> localhost:9001 (jobtracker).
>
> Please help. Thanks.
>
>
>
>
>
Re: Starting Map Reduce Job on EC2
Posted by Ronald Petty <ro...@gmail.com>.
Something Something,
Have you confirmed you can connect to the port from your remote machine?
telnet ec2-xxxxxx.... 9000
Kindest regards.
Ron
On Sun, Jan 15, 2012 at 12:16 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Hello,
>
> Our Hadoop cluster is setup on EC2, but our client machine which will
> trigger the M/R job is in our data center. I am trying to start a M/R job
> from our client machine, but getting this:
>
> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server:
> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 0 time(s).
> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server:
> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 1 time(s).
> 00:01:16.885 [pool-6-thread-1] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server:
> ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com/xx.xx.xxx.xx:9000. Already
> tried 2 time(s).
> & so on...
>
>
> I tried putting our 'pem' file in the classpath, but that didn't help. I
> am assuming it should be possible to trigger a job on EC2 cluster from a
> machine that's not in EC2, correct?
>
> Please note that I can trigger a M/R job from the client machine to a
> Hadoop running on the same machine - localhost:9000 (hdfs) as well as
> localhost:9001 (jobtracker).
>
> Please help. Thanks.
>
>
>
>
>