You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by hua beatls <be...@gmail.com> on 2013/02/26 19:02:39 UTC
datanodes with differenet cpus&mem
HI,
i have some existing PC servers, which would be used to form a hadoop
cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and another 4
have 8cpu(6ocres),256G memory. Big gap.
can i user these 8 servers to make a hadoop cluster? any suggestion?
Thanks!
beatls
Re: datanodes with differenet cpus&mem
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Hua,
Hadoop will spread the data evenly between all the nodes. And HBase
balancer too.
However, you can still assign more map/reduce jobs to the bigger
servers so they will try to handle more load.
Else, you can manually (or with a customed balancer) move more regions
to the bigger servers.
And even without all those modifications, you can still use all the 8
servers together.
JM
2013/2/26 hua beatls <be...@gmail.com>:
> Hi,
> Yes, i already have these servers, and hope can find solution to use
> them efficiently.
> does hadoop/hbase system has the 'intelligence' to arrange more
> work(load, lookup,etc.) to the 8cpu servers compared with 2cpu servers?
>
>
> thanks!
>
> beatls
>
>
>
>
> On Wed, Feb 27, 2013 at 2:30 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Hua,
>>
>> It all depend what you want to do with your servers. They are already
>> pretty big. They will be able to handle big load.
>>
>> What's about the hard drives?
>>
>> What can of load are you going to apply? CPU intensive (Like big
>> regular expression on big text content to extract keywors from rows)?
>> Disk intensive (Like fast MRs job storring many data in an output
>> table)? Network intensive (Big transfers, lookups, etc.)?
>>
>> Also, is it for a lab? Or a production?
>>
>> If you already have those servers, you can simply install them with
>> HBase/Hadoop and run some load tests base on your usecase?
>>
>> JM
>>
>> 2013/2/26 hua beatls <be...@gmail.com>:
>> > Hi JM,
>> > should I add more storage to the bigger ones ?
>> >
>> > many thanks!
>> >
>> >
>> > beatls
>> >
>> > On Wed, Feb 27, 2013 at 2:08 AM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> Hi Hua,
>> >>
>> >> You can use servers with different configuration, but the entire
>> >> cluster will tend to run as slow as your slowest server. However, that
>> >> will still help to to spread the load.
>> >>
>> >> JM
>> >>
>> >> 2013/2/26 hua beatls <be...@gmail.com>:
>> >> > shoudl we use PC servers with same configuration?
>> >> >
>> >> > beatlsd
>> >> >
>> >> > On Wed, Feb 27, 2013 at 2:02 AM, hua beatls <be...@gmail.com> wrote:
>> >> >
>> >> >> HI,
>> >> >> i have some existing PC servers, which would be used to form a
>> hadoop
>> >> >> cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and
>> >> another 4
>> >> >> have 8cpu(6ocres),256G memory. Big gap.
>> >> >> can i user these 8 servers to make a hadoop cluster? any
>> suggestion?
>> >> >>
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >> beatls
>> >> >>
>> >>
>>
Re: datanodes with differenet cpus&mem
Posted by hua beatls <be...@gmail.com>.
Hi,
Yes, i already have these servers, and hope can find solution to use
them efficiently.
does hadoop/hbase system has the 'intelligence' to arrange more
work(load, lookup,etc.) to the 8cpu servers compared with 2cpu servers?
thanks!
beatls
On Wed, Feb 27, 2013 at 2:30 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Hua,
>
> It all depend what you want to do with your servers. They are already
> pretty big. They will be able to handle big load.
>
> What's about the hard drives?
>
> What can of load are you going to apply? CPU intensive (Like big
> regular expression on big text content to extract keywors from rows)?
> Disk intensive (Like fast MRs job storring many data in an output
> table)? Network intensive (Big transfers, lookups, etc.)?
>
> Also, is it for a lab? Or a production?
>
> If you already have those servers, you can simply install them with
> HBase/Hadoop and run some load tests base on your usecase?
>
> JM
>
> 2013/2/26 hua beatls <be...@gmail.com>:
> > Hi JM,
> > should I add more storage to the bigger ones ?
> >
> > many thanks!
> >
> >
> > beatls
> >
> > On Wed, Feb 27, 2013 at 2:08 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi Hua,
> >>
> >> You can use servers with different configuration, but the entire
> >> cluster will tend to run as slow as your slowest server. However, that
> >> will still help to to spread the load.
> >>
> >> JM
> >>
> >> 2013/2/26 hua beatls <be...@gmail.com>:
> >> > shoudl we use PC servers with same configuration?
> >> >
> >> > beatlsd
> >> >
> >> > On Wed, Feb 27, 2013 at 2:02 AM, hua beatls <be...@gmail.com> wrote:
> >> >
> >> >> HI,
> >> >> i have some existing PC servers, which would be used to form a
> hadoop
> >> >> cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and
> >> another 4
> >> >> have 8cpu(6ocres),256G memory. Big gap.
> >> >> can i user these 8 servers to make a hadoop cluster? any
> suggestion?
> >> >>
> >> >>
> >> >> Thanks!
> >> >>
> >> >> beatls
> >> >>
> >>
>
Re: datanodes with differenet cpus&mem
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Hua,
It all depend what you want to do with your servers. They are already
pretty big. They will be able to handle big load.
What's about the hard drives?
What can of load are you going to apply? CPU intensive (Like big
regular expression on big text content to extract keywors from rows)?
Disk intensive (Like fast MRs job storring many data in an output
table)? Network intensive (Big transfers, lookups, etc.)?
Also, is it for a lab? Or a production?
If you already have those servers, you can simply install them with
HBase/Hadoop and run some load tests base on your usecase?
JM
2013/2/26 hua beatls <be...@gmail.com>:
> Hi JM,
> should I add more storage to the bigger ones ?
>
> many thanks!
>
>
> beatls
>
> On Wed, Feb 27, 2013 at 2:08 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Hua,
>>
>> You can use servers with different configuration, but the entire
>> cluster will tend to run as slow as your slowest server. However, that
>> will still help to to spread the load.
>>
>> JM
>>
>> 2013/2/26 hua beatls <be...@gmail.com>:
>> > shoudl we use PC servers with same configuration?
>> >
>> > beatlsd
>> >
>> > On Wed, Feb 27, 2013 at 2:02 AM, hua beatls <be...@gmail.com> wrote:
>> >
>> >> HI,
>> >> i have some existing PC servers, which would be used to form a hadoop
>> >> cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and
>> another 4
>> >> have 8cpu(6ocres),256G memory. Big gap.
>> >> can i user these 8 servers to make a hadoop cluster? any suggestion?
>> >>
>> >>
>> >> Thanks!
>> >>
>> >> beatls
>> >>
>>
Re: datanodes with differenet cpus&mem
Posted by hua beatls <be...@gmail.com>.
Hi JM,
should I add more storage to the bigger ones ?
many thanks!
beatls
On Wed, Feb 27, 2013 at 2:08 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Hua,
>
> You can use servers with different configuration, but the entire
> cluster will tend to run as slow as your slowest server. However, that
> will still help to to spread the load.
>
> JM
>
> 2013/2/26 hua beatls <be...@gmail.com>:
> > shoudl we use PC servers with same configuration?
> >
> > beatlsd
> >
> > On Wed, Feb 27, 2013 at 2:02 AM, hua beatls <be...@gmail.com> wrote:
> >
> >> HI,
> >> i have some existing PC servers, which would be used to form a hadoop
> >> cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and
> another 4
> >> have 8cpu(6ocres),256G memory. Big gap.
> >> can i user these 8 servers to make a hadoop cluster? any suggestion?
> >>
> >>
> >> Thanks!
> >>
> >> beatls
> >>
>
Re: datanodes with differenet cpus&mem
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Hua,
You can use servers with different configuration, but the entire
cluster will tend to run as slow as your slowest server. However, that
will still help to to spread the load.
JM
2013/2/26 hua beatls <be...@gmail.com>:
> shoudl we use PC servers with same configuration?
>
> beatlsd
>
> On Wed, Feb 27, 2013 at 2:02 AM, hua beatls <be...@gmail.com> wrote:
>
>> HI,
>> i have some existing PC servers, which would be used to form a hadoop
>> cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and another 4
>> have 8cpu(6ocres),256G memory. Big gap.
>> can i user these 8 servers to make a hadoop cluster? any suggestion?
>>
>>
>> Thanks!
>>
>> beatls
>>
Re: datanodes with differenet cpus&mem
Posted by hua beatls <be...@gmail.com>.
shoudl we use PC servers with same configuration?
beatlsd
On Wed, Feb 27, 2013 at 2:02 AM, hua beatls <be...@gmail.com> wrote:
> HI,
> i have some existing PC servers, which would be used to form a hadoop
> cluster, among which 4 servers have 2cpu(6 cores), 96Gmemory, and another 4
> have 8cpu(6ocres),256G memory. Big gap.
> can i user these 8 servers to make a hadoop cluster? any suggestion?
>
>
> Thanks!
>
> beatls
>