Weird issues when upgrading from 1.4 to 3.4

[Willem Basson <wi...@gmail.com>]

Hi there

We are currently upgrading from solr 1.4 to 3.4 and have seen some issues
with our specific use-case.
As background we drop the whole index and then add all our documents in one
big build before we commit and then optimise.
This way we can revert the build if there are any issues and this won't be
replicated out to our slave instances.

We use the following flags: -Xmx2g -Xms2g
With 1.4 it works pretty well, and the process doesn't use much more than
2GB of memory as the index is being built. Garbage collector kicks in quite
a bit but performance are pretty decent throughout the build.
For 3.4 though the process builds up to use a lot more memory, about 20 to
30 GB and it starts swapping and grinding to a bit of a halt. This means it
takes about 10 times longer to complete the build, but it does complete.
We have about a 100k documents and they aren't massive, but they aren't
small either. They do have a lot of fields though, some have 6000+ fields.

Monitoring the heap size I can see that it stays under 2GB and garbage
collector seems to kick in just like with 1.4

While we could increase the memory and do a few other things to make this
less of an issue I would really like to know if anyone has any idea what the
problem could be and what we could do to try and change the behaviour in
config. Our solrconfig.xml file is the same for 1.4 and 3.4, we haven't made
any changes.

Thanks

Willem Basson

RE: Weird issues when upgrading from 1.4 to 3.4

["Jaeger, Jay - DOT" <Ja...@dot.wi.gov>]

I have no idea what might be causing your memory to increase like that (we haven't run 3.4, and our index so far has been at most 28 million rows with maybe 40 fields), but just as an aside, depending upon what you meant by "we drop the whole index", I'd think it might work better to do an <optimize> right after deleting everything, and then a second one right after all of the updates.  And if you did it that way, I would not expect you to need an <optimize> at the end, either.

If you are trying to keep the old one available for query while the new ones are being added, might it be better to do that with two cores, and then swap them when complete?

JRJ


-----Original Message-----
From: Willem Basson [mailto:willem.basson@gmail.com] 
Sent: Friday, September 30, 2011 7:44 AM
To: solr-user@lucene.apache.org
Subject: Re: Weird issues when upgrading from 1.4 to 3.4

Just to clarify, I'm not worried about the virtual memory getting bigger,
the issue is that after doing a lot of adds without a commit the performance
dramatically decreases until we do the commit.
This didn't use to be a problem with 1.4

Willem

On Fri, Sep 30, 2011 at 10:20 AM, Willem Basson <wi...@gmail.com>wrote

> Hi there
>
> We are currently upgrading from solr 1.4 to 3.4 and have seen some issues
> with our specific use-case.
> As background we drop the whole index and then add all our documents in one
> big build before we commit and then optimise.
> This way we can revert the build if there are any issues and this won't be
> replicated out to our slave instances.
>
> We use the following flags: -Xmx2g -Xms2g
> With 1.4 it works pretty well, and the process doesn't use much more than
> 2GB of memory as the index is being built. Garbage collector kicks in quite
> a bit but performance are pretty decent throughout the build.
> For 3.4 though the process builds up to use a lot more memory, about 20 to
> 30 GB and it starts swapping and grinding to a bit of a halt. This means it
> takes about 10 times longer to complete the build, but it does complete.
> We have about a 100k documents and they aren't massive, but they aren't
> small either. They do have a lot of fields though, some have 6000+ fields.
>
> Monitoring the heap size I can see that it stays under 2GB and garbage
> collector seems to kick in just like with 1.4
>
> While we could increase the memory and do a few other things to make this
> less of an issue I would really like to know if anyone has any idea what the
> problem could be and what we could do to try and change the behaviour in
> config. Our solrconfig.xml file is the same for 1.4 and 3.4, we haven't made
> any changes.
>
> Thanks
>
> Willem Basson
>

Re: Weird issues when upgrading from 1.4 to 3.4

[Willem Basson <wi...@gmail.com>]

Just to clarify, I'm not worried about the virtual memory getting bigger,
the issue is that after doing a lot of adds without a commit the performance
dramatically decreases until we do the commit.
This didn't use to be a problem with 1.4

Willem

On Fri, Sep 30, 2011 at 10:20 AM, Willem Basson <wi...@gmail.com>wrote

> Hi there
>
> We are currently upgrading from solr 1.4 to 3.4 and have seen some issues
> with our specific use-case.
> As background we drop the whole index and then add all our documents in one
> big build before we commit and then optimise.
> This way we can revert the build if there are any issues and this won't be
> replicated out to our slave instances.
>
> We use the following flags: -Xmx2g -Xms2g
> With 1.4 it works pretty well, and the process doesn't use much more than
> 2GB of memory as the index is being built. Garbage collector kicks in quite
> a bit but performance are pretty decent throughout the build.
> For 3.4 though the process builds up to use a lot more memory, about 20 to
> 30 GB and it starts swapping and grinding to a bit of a halt. This means it
> takes about 10 times longer to complete the build, but it does complete.
> We have about a 100k documents and they aren't massive, but they aren't
> small either. They do have a lot of fields though, some have 6000+ fields.
>
> Monitoring the heap size I can see that it stays under 2GB and garbage
> collector seems to kick in just like with 1.4
>
> While we could increase the memory and do a few other things to make this
> less of an issue I would really like to know if anyone has any idea what the
> problem could be and what we could do to try and change the behaviour in
> config. Our solrconfig.xml file is the same for 1.4 and 3.4, we haven't made
> any changes.
>
> Thanks
>
> Willem Basson
>