You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jimtronic <ji...@gmail.com> on 2013/02/21 06:02:06 UTC

SolrCloud as my primary data store

Now that I've been running Solr Cloud for a couple months and gotten
comfortable with it, I think it's time to revisit this subject. 

When I search for the topic of using Solr as a primary db online, I get lots
of discussions from 2-3 years ago and usually they point out a lot of
hurdles that have now largely been eliminated with the release of Solr
Cloud.

I've stopped using the standard method of writing to my db and pushing out
periodically to solr. Instead, I'm writing simultaneously to solr and the db
with less frequent syncs from the database just to be safe. I find this to
be much faster and easier than doing delta imports via the DIH handler. In
fact, it's gone so smoothly, I'm really wondering why I need to keep writing
it to the db at all. 

I've always got several nodes running and launching new ones takes only
minutes to be fully operational. I'm taking frequent snapshots and my test
restores have been painless and quick.

So, if I'm looking at other NoSQL solutions like MongoDB or Cassandra, why
wouldn't I just use Solr? It's distributed, fast, and stable. It has a great
http api and it's nearly schema-less using dynamic fields. And, most
importantly, it offers the most powerful query language available. 

I'd really like to hear from someone who has made the leap.

Cheers, Jim



--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-as-my-primary-data-store-tp4041774.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud as my primary data store

Posted by Gora Mohanty <go...@mimirtech.com>.
On 22 February 2013 21:32, jimtronic <ji...@gmail.com> wrote:
> Yes, these are good points. I'm using solr to leverage user preference data
> and I need that data available real time. SQL just can't do the kind of
> things I'm able to do in solr, so I have to wait until the write (a user
> action, a user preference, etc) gets to solr from the db anyway.

?
Does that not depend on how your data pipeline works?

> I'm kind of curious about how many single documents i can send through via
> the json update in a day. Millions would be nice, but I wonder what the
> upper limit would be.

The devil is in the details, of course, i.e., what is the average size of the
documents, where are they pulled from,what transformations are applied,
what hardware does one have to throw at the problem, etc., but in most
cases millions a day is quite easy to achieve. Other people have posted
benchmarks in this regard, but even with Solr 1.4 we were doing a complete
indexing of ~8 million heavy records (total index size of ~80GB) in 6-8 hours,
and the bottle-neck for us was the database.

Regards,
Gora

Re: SolrCloud as my primary data store

Posted by jimtronic <ji...@gmail.com>.
Ok, so this is a VERY simple test, but I ran a test that has 10 concurrent
users insert small documents (just a few fields each) as fast as they can
and I was able to insert just over 29,000 per second.

That's 2.5 billion a day. That's more than enough for me at the moment. I
was using commitWithin=500.


On Fri, Feb 22, 2013 at 11:59 AM, Shawn Heisey-4 [via Lucene] <
ml-node+s472066n4042281h27@n3.nabble.com> wrote:

> On 2/22/2013 9:02 AM, jimtronic wrote:
> > Yes, these are good points. I'm using solr to leverage user preference
> data
> > and I need that data available real time. SQL just can't do the kind of
> > things I'm able to do in solr, so I have to wait until the write (a user
> > action, a user preference, etc) gets to solr from the db anyway.
> >
> > I'm kind of curious about how many single documents i can send through
> via
> > the json update in a day. Millions would be nice, but I wonder what the
> > upper limit would be.
>
> I have a distributed index with about 76 million documents in it,
> original source is MySQL.  It's comprised of seven shards - six of them
> are large, with over 12 millon docs each.  One shard is small, usually
> containing only a few hundred thousand docs.  The full-import updates
> all seven shards in parallel, but other than that, it is not a
> multi-threaded operation.
>
> On my dev environment, I'm absolutely positive that my bottleneck is I/O
> on the Solr server.  That server has 7200RPM SAS drives in basic RAID1
> and takes about 8 hours for a full-import.  It contains the entire index.
>
> In production, I am not sure where the bottleneck is - my guess is that
> it's I/O, but it might be in the database.  These servers have RAID10
> with six 7200RPM SATA drives, a caching RAID controller, plenty of RAM,
> and each one contains only half the index.  On version 3.5, it takes
> about 3.5 hours for a full-import.  On 4.2-SNAPSHOT, it takes about 4
> hours.  The new version has the updateLog enabled.
>
> Thanks,
> Shawn
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/SolrCloud-as-my-primary-data-store-tp4041774p4042281.html
>  To unsubscribe from SolrCloud as my primary data store, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4041774&code=amltdHJvbmljQGdtYWlsLmNvbXw0MDQxNzc0fDEzMjQ4NDk0MTQ=>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-as-my-primary-data-store-tp4041774p4042287.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud as my primary data store

Posted by Shawn Heisey <so...@elyograg.org>.
On 2/22/2013 9:02 AM, jimtronic wrote:
> Yes, these are good points. I'm using solr to leverage user preference data
> and I need that data available real time. SQL just can't do the kind of
> things I'm able to do in solr, so I have to wait until the write (a user
> action, a user preference, etc) gets to solr from the db anyway.
>
> I'm kind of curious about how many single documents i can send through via
> the json update in a day. Millions would be nice, but I wonder what the
> upper limit would be.

I have a distributed index with about 76 million documents in it, 
original source is MySQL.  It's comprised of seven shards - six of them 
are large, with over 12 millon docs each.  One shard is small, usually 
containing only a few hundred thousand docs.  The full-import updates 
all seven shards in parallel, but other than that, it is not a 
multi-threaded operation.

On my dev environment, I'm absolutely positive that my bottleneck is I/O 
on the Solr server.  That server has 7200RPM SAS drives in basic RAID1 
and takes about 8 hours for a full-import.  It contains the entire index.

In production, I am not sure where the bottleneck is - my guess is that 
it's I/O, but it might be in the database.  These servers have RAID10 
with six 7200RPM SATA drives, a caching RAID controller, plenty of RAM, 
and each one contains only half the index.  On version 3.5, it takes 
about 3.5 hours for a full-import.  On 4.2-SNAPSHOT, it takes about 4 
hours.  The new version has the updateLog enabled.

Thanks,
Shawn


Re: SolrCloud as my primary data store

Posted by jimtronic <ji...@gmail.com>.
Yes, these are good points. I'm using solr to leverage user preference data
and I need that data available real time. SQL just can't do the kind of
things I'm able to do in solr, so I have to wait until the write (a user
action, a user preference, etc) gets to solr from the db anyway. 

I'm kind of curious about how many single documents i can send through via
the json update in a day. Millions would be nice, but I wonder what the
upper limit would be.





--
View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-as-my-primary-data-store-tp4041774p4042251.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud as my primary data store

Posted by Timothy Potter <th...@gmail.com>.
With Solr's atomic updates, optimistic locking, update log,
openSearcher=false on commits, etc. you can definitely do this.

Biggest question in my mind is whether you're willing to accept Solr's
emphasis on consistency vs. write-availability? With a db like
Cassandra, you can achieve better write-availability by giving up a
little on the consistency side. With Solr, you don't have that choice
- writes must succeed on the shard leader and replicas. With the tlog,
Solr still does pretty good here. The other concern is how frequently
(and how many) are you updating data in existing docs? Solr has to
delete and re-index the entire doc after updating a single field. We
abuse Solr with millions of atomic updates daily but it's not anywhere
near as fast as you get with database updates.

Lastly, have you seen Yonik's slides from Apache Eurocon - great read
if not: http://vimeopro.com/user11514798/apache-lucene-eurocon-2012/video/55387447

Cheers,
Tim

On Wed, Feb 20, 2013 at 10:02 PM, jimtronic <ji...@gmail.com> wrote:
> Now that I've been running Solr Cloud for a couple months and gotten
> comfortable with it, I think it's time to revisit this subject.
>
> When I search for the topic of using Solr as a primary db online, I get lots
> of discussions from 2-3 years ago and usually they point out a lot of
> hurdles that have now largely been eliminated with the release of Solr
> Cloud.
>
> I've stopped using the standard method of writing to my db and pushing out
> periodically to solr. Instead, I'm writing simultaneously to solr and the db
> with less frequent syncs from the database just to be safe. I find this to
> be much faster and easier than doing delta imports via the DIH handler. In
> fact, it's gone so smoothly, I'm really wondering why I need to keep writing
> it to the db at all.
>
> I've always got several nodes running and launching new ones takes only
> minutes to be fully operational. I'm taking frequent snapshots and my test
> restores have been painless and quick.
>
> So, if I'm looking at other NoSQL solutions like MongoDB or Cassandra, why
> wouldn't I just use Solr? It's distributed, fast, and stable. It has a great
> http api and it's nearly schema-less using dynamic fields. And, most
> importantly, it offers the most powerful query language available.
>
> I'd really like to hear from someone who has made the leap.
>
> Cheers, Jim
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-as-my-primary-data-store-tp4041774.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrCloud as my primary data store

Posted by Michael Sokolov <ms...@safaribooksonline.com>.
On 02/21/2013 12:02 AM, jimtronic wrote:
> Now that I've been running Solr Cloud for a couple months and gotten
> comfortable with it, I think it's time to revisit this subject.
>
>
>    
...
> I'd really like to hear from someone who has made the leap.
>
> Cheers, Jim
>    
We use Solr as our primary storage for XML documents for many customer 
installs, and we're quite happy with performance, reliability, update 
speed, and so on.  We don't handle inetractive updates from end users: 
only batch updates from an upstream content authoring system of some 
kind.  In this situation, we're quite content.  I looked at other NoSQL 
db's, but couldn't find a compelling reason to switch, and some 
disadvantages in some cases.

-Mike