You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Imran M Yousuf <im...@gmail.com> on 2010/04/05 18:52:00 UTC
About test/production server configuration
Hi,
We are a startup who have decided to use HBase purely because we want
to take advantage of HDFS based reliability, redundancy, MapReduce and
BigTable. For that we are thinking to go for a test environment with 5
servers and production environment with 10 servers in both case the
Hadoop cluster will be used for HBase + MapReduce + Solr Index.
Firstly, I would like some opinion on whether 10 servers is a good
number for all 3 purposes or not. Secondly what kind of test
environment is currently in use in different organizations. Thirdly, I
would like to learn some server configuration and purchase price (with
purchase location if possible).
Waiting eagerly for some feedback.
Thank you,
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: About test/production server configuration
Posted by Imran M Yousuf <im...@gmail.com>.
Hi,
I would really appreciate some guideline on it for 2 purpose: 1) To
asses whether we have enough budget to start up and 2) to know what
kind of setup we might need.
Eagerly looking forward for some help/guideline.
Best regards,
Imran
On Wed, Apr 7, 2010 at 7:24 AM, Imran M Yousuf <im...@gmail.com> wrote:
> Hi Jonathan,
>
> Thanks for your reply. Please find my replies inline.
>
> On Wed, Apr 7, 2010 at 4:04 AM, Jonathan Gray <jg...@facebook.com> wrote:
>> Or if you have a budget in mind, we can help you determine what would be the best way to allocate those dollars.
>>
>
> That would be just great. Budget provisioned for the whole system is
> approximately 27,000 USD. Among that we have budgeted for the
> Hadoop+HBase cluster to 13,500 USD (for 10 servers).
>
>>> <snip />
>>> Have you run Solr atop HDFS? I doubt this will be performant.
>>>
>
> We haven't tested it yet, but it is in our scope; after testing it we
> will decide on which path to take.
>
>>> Also, to properly scope your cluster, you need to come up with actual
>>> number targets if you want to be able to accurately provision hardware.
>>> "not much" data now, but "lots" of data later could mean anything.
>>> Decide what you want to provision for and then you can accurately do
>>> so.
>>>
>
> Hmm, I am not sure I understand correctly about provisioning but I am
> giving it a try.
> Our system composes of web applications for a CMS,
> Accounting+Inventory System (SaaS), another web application
> integrating CMS and Accounts, and Solr as a search engine. So in times
> of data for the setup I would like to support 6TB of data and 4
> Billion rows. Some details are as follows.
>
> 2000 organizations using the SaaS. Each with 500 inventory items. Each
> inventory item with MM would be at least 300k at an average. We want
> to support a million transactions per organizations (average) with
> each being 2k at an average. So total for the accounting system is
> 4,300 GB ~ 5 TB (approx.) of data and 2,001,002,000 rows minimum.
>
> Each inventory item will be a content in the CMS; in addition users
> can organize their contents; plus the MM will also be a content
> (basically a copy of the record mentioned above). 750 GB and 2M rows
> (minimum).
>
> I hope this helps. Eagerly waiting for some direction :).
>
> Thank you,
>
> Imran
>
>> <snip />
>
>
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: About test/production server configuration
Posted by Imran M Yousuf <im...@gmail.com>.
Hi Jonathan,
Thanks for your reply. Please find my replies inline.
On Wed, Apr 7, 2010 at 4:04 AM, Jonathan Gray <jg...@facebook.com> wrote:
> Or if you have a budget in mind, we can help you determine what would be the best way to allocate those dollars.
>
That would be just great. Budget provisioned for the whole system is
approximately 27,000 USD. Among that we have budgeted for the
Hadoop+HBase cluster to 13,500 USD (for 10 servers).
>> <snip />
>> Have you run Solr atop HDFS? I doubt this will be performant.
>>
We haven't tested it yet, but it is in our scope; after testing it we
will decide on which path to take.
>> Also, to properly scope your cluster, you need to come up with actual
>> number targets if you want to be able to accurately provision hardware.
>> "not much" data now, but "lots" of data later could mean anything.
>> Decide what you want to provision for and then you can accurately do
>> so.
>>
Hmm, I am not sure I understand correctly about provisioning but I am
giving it a try.
Our system composes of web applications for a CMS,
Accounting+Inventory System (SaaS), another web application
integrating CMS and Accounts, and Solr as a search engine. So in times
of data for the setup I would like to support 6TB of data and 4
Billion rows. Some details are as follows.
2000 organizations using the SaaS. Each with 500 inventory items. Each
inventory item with MM would be at least 300k at an average. We want
to support a million transactions per organizations (average) with
each being 2k at an average. So total for the accounting system is
4,300 GB ~ 5 TB (approx.) of data and 2,001,002,000 rows minimum.
Each inventory item will be a content in the CMS; in addition users
can organize their contents; plus the MM will also be a content
(basically a copy of the record mentioned above). 750 GB and 2M rows
(minimum).
I hope this helps. Eagerly waiting for some direction :).
Thank you,
Imran
> <snip />
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
RE: About test/production server configuration
Posted by Jonathan Gray <jg...@facebook.com>.
Or if you have a budget in mind, we can help you determine what would be the best way to allocate those dollars.
> -----Original Message-----
> From: Jonathan Gray [mailto:jgray@facebook.com]
> Sent: Tuesday, April 06, 2010 3:11 PM
> To: hbase-user@hadoop.apache.org
> Subject: RE: About test/production server configuration
>
> Imran,
>
> Have you run Solr atop HDFS? I doubt this will be performant.
>
> Also, to properly scope your cluster, you need to come up with actual
> number targets if you want to be able to accurately provision hardware.
> "not much" data now, but "lots" of data later could mean anything.
> Decide what you want to provision for and then you can accurately do
> so.
>
> > -----Original Message-----
> > From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> > Sent: Monday, April 05, 2010 6:11 PM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: About test/production server configuration
> >
> > On Mon, Apr 5, 2010 at 11:56 PM, Jonathan Gray <jg...@facebook.com>
> > wrote:
> > > Imran,
> > >
> > > It's impossible to give good advice on cluster size and hardware
> > configuration without some idea of the requirements.
> > >
> >
> > Sorry my mistake, I should have elaborated a little bit more. Please
> > find some requirements below inline.
> >
> > > How much data?
> >
> > To startup, initially we will not have much, but down the road it
> will
> > be a lot of data. Plus a lot of user created content. Initially there
> > will be a lot of log-like entries, plus transactions...
> >
> > > How will the data be queried?
> >
> > We are focusing on system design to lookup using key only. For
> > searching it will be Solr only. So the idea is Solr will be used for
> > all searching and then the data lookup will be performed in HBase. In
> > addition, we will have both Application layer caching in Ehcache and
> > Web Accelerator (Varnish).
> >
> > > What kind of load do you expect?
> >
> > Hard to estimate but we are planning for moderate installation, so
> > that if we have good response from the market we can expand, thats
> one
> > of the 2 primary reason to choose HBase we will be able to scale it
> on
> > the fly.
> >
> > > You are going to be doing offline batch/MapReduce, online random
> > access, as well as search all from the same nodes? This can be
> > dangerous.
> >
> > Yes the offline batch, HBase lookup will be on the same machine. But
> > not search as a whole...Solr will use HDFS only to store the index
> and
> > read it from, but no processing related to search will be done there.
> > It will be on a separate box all together. But your following
> > statement is tempting me to use RAID+DRDB for Solr based searching.
> >
> > Another thing to note is, the offline batch work would be summarize
> > tables. One example from our system would be to generate daily
> balance
> > sheet of ledgers, profit loss statement etc. for 100+ Journals in a
> > PoS SaaS.
> >
> > > I would strongly recommend against putting Hadoop+HBase on the same
> > nodes as something like Solr, unless you have dedicated disks for
> each.
> > Also, don't forget about ZooKeeper which you definitely will need
> > separate nodes/disks for if you will be co-locating so many other
> > things.
> >
> > Hmm.. What I understand from this discussion and Patrick's point on
> > ZK. I would go for:
> >
> > - 4 separate DN (each DN with its dedicated disk but may be not
> > physical server) for Solr only, as a side note, initially we will
> have
> > 2 Solr search boxes.
> > - ZK needs separate disk for performance, so would have dedicated
> > disks for it too.
> >
> > But what I am confused about is how spread out ZK, Multi-Master, RS,
> > DN, TT for HBase. Insight, comments, suggestions on it would be most
> > welcome.
> >
> > Another note on our perspective is that we want to scale horizontally
> > by adding more machines and not vertically (if we wanted it or could
> > afford it, we would have probably chosen a RDBMS). Being able to
> scale
> > horizontally as our user-base, load and revenue increases was/is
> > essential to us.
> >
> > Waiting eagerly for some insight, comments and/or suggestions.
> >
> > Thank you.
> >
> > Imran
> >
> > >
> > > JG
> > >
> > >> -----Original Message-----
> > >> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> > >> Sent: Monday, April 05, 2010 9:52 AM
> > >> To: hbase-user@hadoop.apache.org
> > >> Subject: About test/production server configuration
> > >>
> > >> Hi,
> > >>
> > >> We are a startup who have decided to use HBase purely because we
> > want
> > >> to take advantage of HDFS based reliability, redundancy, MapReduce
> > and
> > >> BigTable. For that we are thinking to go for a test environment
> with
> > 5
> > >> servers and production environment with 10 servers in both case
> the
> > >> Hadoop cluster will be used for HBase + MapReduce + Solr Index.
> > >>
> > >> Firstly, I would like some opinion on whether 10 servers is a good
> > >> number for all 3 purposes or not. Secondly what kind of test
> > >> environment is currently in use in different organizations.
> Thirdly,
> > I
> > >> would like to learn some server configuration and purchase price
> > (with
> > >> purchase location if possible).
> > >>
> > >> Waiting eagerly for some feedback.
> > >>
> > >> Thank you,
> > >>
> > >> --
> > >> Imran M Yousuf
> > >> Entrepreneur & Software Engineer
> > >> Smart IT Engineering
> > >> Dhaka, Bangladesh
> > >> Email: imran@smartitengineering.com
> > >> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> > >> Mobile: +880-1711402557
> > >
> >
> >
> >
> > --
> > Imran M Yousuf
> > Entrepreneur & Software Engineer
> > Smart IT Engineering
> > Dhaka, Bangladesh
> > Email: imran@smartitengineering.com
> > Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> > Mobile: +880-1711402557
RE: About test/production server configuration
Posted by Jonathan Gray <jg...@facebook.com>.
Imran,
Have you run Solr atop HDFS? I doubt this will be performant.
Also, to properly scope your cluster, you need to come up with actual number targets if you want to be able to accurately provision hardware. "not much" data now, but "lots" of data later could mean anything. Decide what you want to provision for and then you can accurately do so.
> -----Original Message-----
> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> Sent: Monday, April 05, 2010 6:11 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: About test/production server configuration
>
> On Mon, Apr 5, 2010 at 11:56 PM, Jonathan Gray <jg...@facebook.com>
> wrote:
> > Imran,
> >
> > It's impossible to give good advice on cluster size and hardware
> configuration without some idea of the requirements.
> >
>
> Sorry my mistake, I should have elaborated a little bit more. Please
> find some requirements below inline.
>
> > How much data?
>
> To startup, initially we will not have much, but down the road it will
> be a lot of data. Plus a lot of user created content. Initially there
> will be a lot of log-like entries, plus transactions...
>
> > How will the data be queried?
>
> We are focusing on system design to lookup using key only. For
> searching it will be Solr only. So the idea is Solr will be used for
> all searching and then the data lookup will be performed in HBase. In
> addition, we will have both Application layer caching in Ehcache and
> Web Accelerator (Varnish).
>
> > What kind of load do you expect?
>
> Hard to estimate but we are planning for moderate installation, so
> that if we have good response from the market we can expand, thats one
> of the 2 primary reason to choose HBase we will be able to scale it on
> the fly.
>
> > You are going to be doing offline batch/MapReduce, online random
> access, as well as search all from the same nodes? This can be
> dangerous.
>
> Yes the offline batch, HBase lookup will be on the same machine. But
> not search as a whole...Solr will use HDFS only to store the index and
> read it from, but no processing related to search will be done there.
> It will be on a separate box all together. But your following
> statement is tempting me to use RAID+DRDB for Solr based searching.
>
> Another thing to note is, the offline batch work would be summarize
> tables. One example from our system would be to generate daily balance
> sheet of ledgers, profit loss statement etc. for 100+ Journals in a
> PoS SaaS.
>
> > I would strongly recommend against putting Hadoop+HBase on the same
> nodes as something like Solr, unless you have dedicated disks for each.
> Also, don't forget about ZooKeeper which you definitely will need
> separate nodes/disks for if you will be co-locating so many other
> things.
>
> Hmm.. What I understand from this discussion and Patrick's point on
> ZK. I would go for:
>
> - 4 separate DN (each DN with its dedicated disk but may be not
> physical server) for Solr only, as a side note, initially we will have
> 2 Solr search boxes.
> - ZK needs separate disk for performance, so would have dedicated
> disks for it too.
>
> But what I am confused about is how spread out ZK, Multi-Master, RS,
> DN, TT for HBase. Insight, comments, suggestions on it would be most
> welcome.
>
> Another note on our perspective is that we want to scale horizontally
> by adding more machines and not vertically (if we wanted it or could
> afford it, we would have probably chosen a RDBMS). Being able to scale
> horizontally as our user-base, load and revenue increases was/is
> essential to us.
>
> Waiting eagerly for some insight, comments and/or suggestions.
>
> Thank you.
>
> Imran
>
> >
> > JG
> >
> >> -----Original Message-----
> >> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> >> Sent: Monday, April 05, 2010 9:52 AM
> >> To: hbase-user@hadoop.apache.org
> >> Subject: About test/production server configuration
> >>
> >> Hi,
> >>
> >> We are a startup who have decided to use HBase purely because we
> want
> >> to take advantage of HDFS based reliability, redundancy, MapReduce
> and
> >> BigTable. For that we are thinking to go for a test environment with
> 5
> >> servers and production environment with 10 servers in both case the
> >> Hadoop cluster will be used for HBase + MapReduce + Solr Index.
> >>
> >> Firstly, I would like some opinion on whether 10 servers is a good
> >> number for all 3 purposes or not. Secondly what kind of test
> >> environment is currently in use in different organizations. Thirdly,
> I
> >> would like to learn some server configuration and purchase price
> (with
> >> purchase location if possible).
> >>
> >> Waiting eagerly for some feedback.
> >>
> >> Thank you,
> >>
> >> --
> >> Imran M Yousuf
> >> Entrepreneur & Software Engineer
> >> Smart IT Engineering
> >> Dhaka, Bangladesh
> >> Email: imran@smartitengineering.com
> >> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> >> Mobile: +880-1711402557
> >
>
>
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557
Re: About test/production server configuration
Posted by Imran M Yousuf <im...@gmail.com>.
On Mon, Apr 5, 2010 at 11:56 PM, Jonathan Gray <jg...@facebook.com> wrote:
> Imran,
>
> It's impossible to give good advice on cluster size and hardware configuration without some idea of the requirements.
>
Sorry my mistake, I should have elaborated a little bit more. Please
find some requirements below inline.
> How much data?
To startup, initially we will not have much, but down the road it will
be a lot of data. Plus a lot of user created content. Initially there
will be a lot of log-like entries, plus transactions...
> How will the data be queried?
We are focusing on system design to lookup using key only. For
searching it will be Solr only. So the idea is Solr will be used for
all searching and then the data lookup will be performed in HBase. In
addition, we will have both Application layer caching in Ehcache and
Web Accelerator (Varnish).
> What kind of load do you expect?
Hard to estimate but we are planning for moderate installation, so
that if we have good response from the market we can expand, thats one
of the 2 primary reason to choose HBase we will be able to scale it on
the fly.
> You are going to be doing offline batch/MapReduce, online random access, as well as search all from the same nodes? This can be dangerous.
Yes the offline batch, HBase lookup will be on the same machine. But
not search as a whole...Solr will use HDFS only to store the index and
read it from, but no processing related to search will be done there.
It will be on a separate box all together. But your following
statement is tempting me to use RAID+DRDB for Solr based searching.
Another thing to note is, the offline batch work would be summarize
tables. One example from our system would be to generate daily balance
sheet of ledgers, profit loss statement etc. for 100+ Journals in a
PoS SaaS.
> I would strongly recommend against putting Hadoop+HBase on the same nodes as something like Solr, unless you have dedicated disks for each. Also, don't forget about ZooKeeper which you definitely will need separate nodes/disks for if you will be co-locating so many other things.
Hmm.. What I understand from this discussion and Patrick's point on
ZK. I would go for:
- 4 separate DN (each DN with its dedicated disk but may be not
physical server) for Solr only, as a side note, initially we will have
2 Solr search boxes.
- ZK needs separate disk for performance, so would have dedicated
disks for it too.
But what I am confused about is how spread out ZK, Multi-Master, RS,
DN, TT for HBase. Insight, comments, suggestions on it would be most
welcome.
Another note on our perspective is that we want to scale horizontally
by adding more machines and not vertically (if we wanted it or could
afford it, we would have probably chosen a RDBMS). Being able to scale
horizontally as our user-base, load and revenue increases was/is
essential to us.
Waiting eagerly for some insight, comments and/or suggestions.
Thank you.
Imran
>
> JG
>
>> -----Original Message-----
>> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
>> Sent: Monday, April 05, 2010 9:52 AM
>> To: hbase-user@hadoop.apache.org
>> Subject: About test/production server configuration
>>
>> Hi,
>>
>> We are a startup who have decided to use HBase purely because we want
>> to take advantage of HDFS based reliability, redundancy, MapReduce and
>> BigTable. For that we are thinking to go for a test environment with 5
>> servers and production environment with 10 servers in both case the
>> Hadoop cluster will be used for HBase + MapReduce + Solr Index.
>>
>> Firstly, I would like some opinion on whether 10 servers is a good
>> number for all 3 purposes or not. Secondly what kind of test
>> environment is currently in use in different organizations. Thirdly, I
>> would like to learn some server configuration and purchase price (with
>> purchase location if possible).
>>
>> Waiting eagerly for some feedback.
>>
>> Thank you,
>>
>> --
>> Imran M Yousuf
>> Entrepreneur & Software Engineer
>> Smart IT Engineering
>> Dhaka, Bangladesh
>> Email: imran@smartitengineering.com
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: About test/production server configuration
Posted by Imran M Yousuf <im...@gmail.com>.
On Tue, Apr 6, 2010 at 12:02 AM, Patrick Hunt <ph...@apache.org> wrote:
> The ZK servers are sensitive to disk (io) latency. I just troubleshot an
> issue last week where a user was seeing 80second (second!) latencies.
> Turns out they were running zk server, namenode, tasktracker, and hbase
> region server all on the same box, that box had a single spindle for all
> io activity and was at 100% utilization for long periods of time. If
> you want decent ZK API latencies (<100ms) you really need to ensure that
> there's at least a separate spindle available for the ZK transaction logs.
>
Great insight and info! So that means if ZK is using a separate
spindle the rest can reside in the same spindle, at least in your
case, nice!
Thanks for the info, noting it.
Imran
> Patrick
>
> On 04/05/2010 11:11 AM, Jonathan Gray wrote:
>>
>> Imran,
>>
>> It's impossible to give good advice on cluster size and hardware
>> configuration without some idea of the requirements.
>>
>> How much data? How will the data be queried? What kind of load do
>> you expect? You are going to be doing offline batch/MapReduce,
>> online random access, as well as search all from the same nodes?
>> This can be dangerous.
>>
>> I would strongly recommend against putting Hadoop+HBase on the same
>> nodes as something like Solr, unless you have dedicated disks for
>> each. Also, don't forget about ZooKeeper which you definitely will
>> need separate nodes/disks for if you will be co-locating so many
>> other things.
>>
>> JG
>>
>>> -----Original Message----- From: Imran M Yousuf
>>> [mailto:imyousuf@gmail.com] Sent: Monday, April 05, 2010 9:52 AM
>>> To: hbase-user@hadoop.apache.org Subject: About test/production
>>> server configuration
>>>
>>> Hi,
>>>
>>> We are a startup who have decided to use HBase purely because we
>>> want to take advantage of HDFS based reliability, redundancy,
>>> MapReduce and BigTable. For that we are thinking to go for a test
>>> environment with 5 servers and production environment with 10
>>> servers in both case the Hadoop cluster will be used for HBase +
>>> MapReduce + Solr Index.
>>>
>>> Firstly, I would like some opinion on whether 10 servers is a good
>>> number for all 3 purposes or not. Secondly what kind of test
>>> environment is currently in use in different organizations.
>>> Thirdly, I would like to learn some server configuration and
>>> purchase price (with purchase location if possible).
>>>
>>> Waiting eagerly for some feedback.
>>>
>>> Thank you,
>>>
>>> -- Imran M Yousuf Entrepreneur& Software Engineer Smart IT
>>> Engineering Dhaka, Bangladesh Email: imran@smartitengineering.com
>>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile:
>>> +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: About test/production server configuration
Posted by Patrick Hunt <ph...@apache.org>.
The ZK servers are sensitive to disk (io) latency. I just troubleshot an
issue last week where a user was seeing 80second (second!) latencies.
Turns out they were running zk server, namenode, tasktracker, and hbase
region server all on the same box, that box had a single spindle for all
io activity and was at 100% utilization for long periods of time. If
you want decent ZK API latencies (<100ms) you really need to ensure that
there's at least a separate spindle available for the ZK transaction logs.
Patrick
On 04/05/2010 11:11 AM, Jonathan Gray wrote:
> Imran,
>
> It's impossible to give good advice on cluster size and hardware
> configuration without some idea of the requirements.
>
> How much data? How will the data be queried? What kind of load do
> you expect? You are going to be doing offline batch/MapReduce,
> online random access, as well as search all from the same nodes?
> This can be dangerous.
>
> I would strongly recommend against putting Hadoop+HBase on the same
> nodes as something like Solr, unless you have dedicated disks for
> each. Also, don't forget about ZooKeeper which you definitely will
> need separate nodes/disks for if you will be co-locating so many
> other things.
>
> JG
>
>> -----Original Message----- From: Imran M Yousuf
>> [mailto:imyousuf@gmail.com] Sent: Monday, April 05, 2010 9:52 AM
>> To: hbase-user@hadoop.apache.org Subject: About test/production
>> server configuration
>>
>> Hi,
>>
>> We are a startup who have decided to use HBase purely because we
>> want to take advantage of HDFS based reliability, redundancy,
>> MapReduce and BigTable. For that we are thinking to go for a test
>> environment with 5 servers and production environment with 10
>> servers in both case the Hadoop cluster will be used for HBase +
>> MapReduce + Solr Index.
>>
>> Firstly, I would like some opinion on whether 10 servers is a good
>> number for all 3 purposes or not. Secondly what kind of test
>> environment is currently in use in different organizations.
>> Thirdly, I would like to learn some server configuration and
>> purchase price (with purchase location if possible).
>>
>> Waiting eagerly for some feedback.
>>
>> Thank you,
>>
>> -- Imran M Yousuf Entrepreneur& Software Engineer Smart IT
>> Engineering Dhaka, Bangladesh Email: imran@smartitengineering.com
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile:
>> +880-1711402557
RE: About test/production server configuration
Posted by Jonathan Gray <jg...@facebook.com>.
Imran,
It's impossible to give good advice on cluster size and hardware configuration without some idea of the requirements.
How much data? How will the data be queried? What kind of load do you expect? You are going to be doing offline batch/MapReduce, online random access, as well as search all from the same nodes? This can be dangerous.
I would strongly recommend against putting Hadoop+HBase on the same nodes as something like Solr, unless you have dedicated disks for each. Also, don't forget about ZooKeeper which you definitely will need separate nodes/disks for if you will be co-locating so many other things.
JG
> -----Original Message-----
> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> Sent: Monday, April 05, 2010 9:52 AM
> To: hbase-user@hadoop.apache.org
> Subject: About test/production server configuration
>
> Hi,
>
> We are a startup who have decided to use HBase purely because we want
> to take advantage of HDFS based reliability, redundancy, MapReduce and
> BigTable. For that we are thinking to go for a test environment with 5
> servers and production environment with 10 servers in both case the
> Hadoop cluster will be used for HBase + MapReduce + Solr Index.
>
> Firstly, I would like some opinion on whether 10 servers is a good
> number for all 3 purposes or not. Secondly what kind of test
> environment is currently in use in different organizations. Thirdly, I
> would like to learn some server configuration and purchase price (with
> purchase location if possible).
>
> Waiting eagerly for some feedback.
>
> Thank you,
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557