You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vinay Pothnis <po...@gmail.com> on 2014/04/11 03:25:58 UTC
deleting large amount data from solr cloud
[solr version 4.3.1]
Hello,
I have a solr cloud (4 nodes - 2 shards) with a fairly large amount
documents (~360G of index per shard). Now, a major portion of the data is
not required and I need to delete those documents. I would need to delete
around 75% of the data.
One of the solutions could be to drop the index completely re-index. But
this is not an option at the moment.
When we tried to delete the data through a query - say 1 day/month's worth
of data. But after deleting just 1 month's worth of data, the master node
is going out of memory - heap space.
Wondering is there any way to incrementally delete the data without
affecting the cluster adversely.
Thank!
Vinay
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Tried to increase the memory to 24G but that wasn't enough as well.
Agree that the index has now grown too much and had to monitor this and
take action much earlier.
The search operations seem to run ok with 16G - mainly because the bulk of
the data that we are trying to delete is not getting searched. So, now -
basically in a salvage mode.
Does the number of documents deleted at a time have any impact? If I
'trickle delete' - say 50K documents at a time - would that make a
difference?
When i delete, does solr try to bring in all the index to memory? Trying to
understand what happens under the hood.
Thanks
Vinay
On 11 April 2014 13:53, Erick Erickson <er...@gmail.com> wrote:
> Using 16G for a 360G index is probably pushing things. A lot. I'm
> actually a bit surprised that the problem only occurs when you delete
> docs....
>
> The simplest thing would be to increase the JVM memory. You should be
> looking at your index to see how big it is, be sure to subtract out
> the *.fdt and *.fdx files, those are used for verbatim copies of the
> raw data and don't really count towards the memory requirements.
>
> I suspect you're just not giving enough memory to your JVM and this is
> just the first OOM you've hit. Look on the Solr admin page and see how
> much is being reported, if it's near the limit of your 16G that's the
> "smoking gun"...
>
> Best,
> Erick
>
> On Fri, Apr 11, 2014 at 7:45 AM, Vinay Pothnis <po...@gmail.com> wrote:
> > Sorry - yes, I meant to say leader.
> > Each JVM has 16G of memory.
> >
> >
> > On 10 April 2014 20:54, Erick Erickson <er...@gmail.com> wrote:
> >
> >> First, there is no "master" node, just leaders and replicas. But that's
> a
> >> nit.
> >>
> >> No real clue why you would be going out of memory. Deleting a
> >> document, even by query should just mark the docs as deleted, a pretty
> >> low-cost operation.
> >>
> >> how much memory are you giving the JVM?
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Apr 10, 2014 at 6:25 PM, Vinay Pothnis <po...@gmail.com>
> wrote:
> >> > [solr version 4.3.1]
> >> >
> >> > Hello,
> >> >
> >> > I have a solr cloud (4 nodes - 2 shards) with a fairly large amount
> >> > documents (~360G of index per shard). Now, a major portion of the
> data is
> >> > not required and I need to delete those documents. I would need to
> delete
> >> > around 75% of the data.
> >> >
> >> > One of the solutions could be to drop the index completely re-index.
> But
> >> > this is not an option at the moment.
> >> >
> >> > When we tried to delete the data through a query - say 1 day/month's
> >> worth
> >> > of data. But after deleting just 1 month's worth of data, the master
> node
> >> > is going out of memory - heap space.
> >> >
> >> > Wondering is there any way to incrementally delete the data without
> >> > affecting the cluster adversely.
> >> >
> >> > Thank!
> >> > Vinay
> >>
>
Re: deleting large amount data from solr cloud
Posted by Erick Erickson <er...@gmail.com>.
Using 16G for a 360G index is probably pushing things. A lot. I'm
actually a bit surprised that the problem only occurs when you delete
docs....
The simplest thing would be to increase the JVM memory. You should be
looking at your index to see how big it is, be sure to subtract out
the *.fdt and *.fdx files, those are used for verbatim copies of the
raw data and don't really count towards the memory requirements.
I suspect you're just not giving enough memory to your JVM and this is
just the first OOM you've hit. Look on the Solr admin page and see how
much is being reported, if it's near the limit of your 16G that's the
"smoking gun"...
Best,
Erick
On Fri, Apr 11, 2014 at 7:45 AM, Vinay Pothnis <po...@gmail.com> wrote:
> Sorry - yes, I meant to say leader.
> Each JVM has 16G of memory.
>
>
> On 10 April 2014 20:54, Erick Erickson <er...@gmail.com> wrote:
>
>> First, there is no "master" node, just leaders and replicas. But that's a
>> nit.
>>
>> No real clue why you would be going out of memory. Deleting a
>> document, even by query should just mark the docs as deleted, a pretty
>> low-cost operation.
>>
>> how much memory are you giving the JVM?
>>
>> Best,
>> Erick
>>
>> On Thu, Apr 10, 2014 at 6:25 PM, Vinay Pothnis <po...@gmail.com> wrote:
>> > [solr version 4.3.1]
>> >
>> > Hello,
>> >
>> > I have a solr cloud (4 nodes - 2 shards) with a fairly large amount
>> > documents (~360G of index per shard). Now, a major portion of the data is
>> > not required and I need to delete those documents. I would need to delete
>> > around 75% of the data.
>> >
>> > One of the solutions could be to drop the index completely re-index. But
>> > this is not an option at the moment.
>> >
>> > When we tried to delete the data through a query - say 1 day/month's
>> worth
>> > of data. But after deleting just 1 month's worth of data, the master node
>> > is going out of memory - heap space.
>> >
>> > Wondering is there any way to incrementally delete the data without
>> > affecting the cluster adversely.
>> >
>> > Thank!
>> > Vinay
>>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Sorry - yes, I meant to say leader.
Each JVM has 16G of memory.
On 10 April 2014 20:54, Erick Erickson <er...@gmail.com> wrote:
> First, there is no "master" node, just leaders and replicas. But that's a
> nit.
>
> No real clue why you would be going out of memory. Deleting a
> document, even by query should just mark the docs as deleted, a pretty
> low-cost operation.
>
> how much memory are you giving the JVM?
>
> Best,
> Erick
>
> On Thu, Apr 10, 2014 at 6:25 PM, Vinay Pothnis <po...@gmail.com> wrote:
> > [solr version 4.3.1]
> >
> > Hello,
> >
> > I have a solr cloud (4 nodes - 2 shards) with a fairly large amount
> > documents (~360G of index per shard). Now, a major portion of the data is
> > not required and I need to delete those documents. I would need to delete
> > around 75% of the data.
> >
> > One of the solutions could be to drop the index completely re-index. But
> > this is not an option at the moment.
> >
> > When we tried to delete the data through a query - say 1 day/month's
> worth
> > of data. But after deleting just 1 month's worth of data, the master node
> > is going out of memory - heap space.
> >
> > Wondering is there any way to incrementally delete the data without
> > affecting the cluster adversely.
> >
> > Thank!
> > Vinay
>
Re: deleting large amount data from solr cloud
Posted by Erick Erickson <er...@gmail.com>.
First, there is no "master" node, just leaders and replicas. But that's a nit.
No real clue why you would be going out of memory. Deleting a
document, even by query should just mark the docs as deleted, a pretty
low-cost operation.
how much memory are you giving the JVM?
Best,
Erick
On Thu, Apr 10, 2014 at 6:25 PM, Vinay Pothnis <po...@gmail.com> wrote:
> [solr version 4.3.1]
>
> Hello,
>
> I have a solr cloud (4 nodes - 2 shards) with a fairly large amount
> documents (~360G of index per shard). Now, a major portion of the data is
> not required and I need to delete those documents. I would need to delete
> around 75% of the data.
>
> One of the solutions could be to drop the index completely re-index. But
> this is not an option at the moment.
>
> When we tried to delete the data through a query - say 1 day/month's worth
> of data. But after deleting just 1 month's worth of data, the master node
> is going out of memory - heap space.
>
> Wondering is there any way to incrementally delete the data without
> affecting the cluster adversely.
>
> Thank!
> Vinay
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Thanks Erick!
On 17 April 2014 08:35, Erick Erickson <er...@gmail.com> wrote:
> bq: Will it get split at any point later?
>
> "Split" is a little ambiguous here. Will it be copied into two or more
> segments? No. Will it disappear? Possibly. Eventually this segment
> will be merged if you add enough documents to the system. Consider
> this scenario:
> you add 1M docs to your system and it results in 10 segments (numbers
> made up). Then you optimize, and you have 1M docs in 1 segment. Fine
> so far.
>
> Now you add 750K of those docs over again, which will delete them from
> the 1 big segment. Your merge policy will, at some point, select this
> segment to merge and it'll disappear...
>
> FWIW,
> Erick@Pedantic.com
>
> On Thu, Apr 17, 2014 at 7:24 AM, Vinay Pothnis <po...@gmail.com> wrote:
> > Thanks a lot Shalin!
> >
> >
> > On 16 April 2014 21:26, Shalin Shekhar Mangar <shalinmangar@gmail.com
> >wrote:
> >
> >> You can specify maxSegments parameter e.g. maxSegments=5 while
> optimizing.
> >>
> >>
> >> On Thu, Apr 17, 2014 at 6:46 AM, Vinay Pothnis <po...@gmail.com>
> wrote:
> >>
> >> > Hello,
> >> >
> >> > Couple of follow up questions:
> >> >
> >> > * When the optimize command is run, looks like it creates one big
> segment
> >> > (forceMerge = 1). Will it get split at any point later? Or will that
> big
> >> > segment remain?
> >> >
> >> > * Is there anyway to maintain the number of segments - but still
> merge to
> >> > reclaim the deleted documents space? In other words, can I issue
> >> > "forceMerge=20"? If so, how would the command look like? Any examples
> for
> >> > this?
> >> >
> >> > Thanks
> >> > Vinay
> >> >
> >> >
> >> >
> >> > On 16 April 2014 07:59, Vinay Pothnis <po...@gmail.com> wrote:
> >> >
> >> > > Thank you Erick!
> >> > > Yes - I am using the expunge deletes option.
> >> > >
> >> > > Thanks for the note on disk space for the optimize command. I should
> >> have
> >> > > enough space for that. What about the heap space requirement? I
> hope it
> >> > can
> >> > > do the optimize with the memory that is allocated to it.
> >> > >
> >> > > Thanks
> >> > > Vinay
> >> > >
> >> > >
> >> > > On 16 April 2014 04:52, Erick Erickson <er...@gmail.com>
> >> wrote:
> >> > >
> >> > >> The optimize should, indeed, reduce the index size. Be aware that
> it
> >> > >> may consume 2x the disk space. You may also try expungedeletes, see
> >> > >> here: https://wiki.apache.org/solr/UpdateXmlMessages
> >> > >>
> >> > >> Best,
> >> > >> Erick
> >> > >>
> >> > >> On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <pothnis@gmail.com
> >
> >> > >> wrote:
> >> > >> > Another update:
> >> > >> >
> >> > >> > I removed the replicas - to avoid the replication doing a full
> >> copy. I
> >> > >> am
> >> > >> > able delete sizeable chunks of data.
> >> > >> > But the overall index size remains the same even after the
> deletes.
> >> It
> >> > >> does
> >> > >> > not seem to go down.
> >> > >> >
> >> > >> > I understand that Solr would do this in background - but I don't
> >> seem
> >> > to
> >> > >> > see the decrease in overall index size even after 1-2 hours.
> >> > >> > I can see a bunch of ".del" files in the index directory, but
> the it
> >> > >> does
> >> > >> > not seem to get cleaned up. Is there anyway to monitor/follow the
> >> > >> progress
> >> > >> > of index compaction?
> >> > >> >
> >> > >> > Also, does triggering "optimize" from the admin UI help to
> compact
> >> the
> >> > >> > index size on disk?
> >> > >> >
> >> > >> > Thanks
> >> > >> > Vinay
> >> > >> >
> >> > >> >
> >> > >> > On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
> >> > >> >
> >> > >> >> Some update:
> >> > >> >>
> >> > >> >> I removed the auto warm configurations for the various caches
> and
> >> > >> reduced
> >> > >> >> the cache sizes. I then issued a call to delete a day's worth of
> >> data
> >> > >> (800K
> >> > >> >> documents).
> >> > >> >>
> >> > >> >> There was no out of memory this time - but some of the nodes
> went
> >> > into
> >> > >> >> recovery mode. Was able to catch some logs this time around and
> >> this
> >> > is
> >> > >> >> what i see:
> >> > >> >>
> >> > >> >> ****************
> >> > >> >> *WARN [2014-04-14 18:11:00.381]
> [org.apache.solr.update.PeerSync]
> >> > >> >> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
> >> > >> >> <http://host1:8983/solr> too many updates received since start
> -
> >> > >> >> startingUpdates no longer overlaps with our currentUpdates*
> >> > >> >> *INFO [2014-04-14 18:11:00.476]
> >> > >> [org.apache.solr.cloud.RecoveryStrategy]
> >> > >> >> PeerSync Recovery was not successful - trying replication.
> >> > >> >> core=core1_shard1_replica2*
> >> > >> >> *INFO [2014-04-14 18:11:00.476]
> >> > >> [org.apache.solr.cloud.RecoveryStrategy]
> >> > >> >> Starting Replication Recovery. core=core1_shard1_replica2*
> >> > >> >> *INFO [2014-04-14 18:11:00.535]
> >> > >> [org.apache.solr.cloud.RecoveryStrategy]
> >> > >> >> Begin buffering updates. core=core1_shard1_replica2*
> >> > >> >> *INFO [2014-04-14 18:11:00.536]
> >> > >> [org.apache.solr.cloud.RecoveryStrategy]
> >> > >> >> Attempting to replicate from
> >> > >> http://host2:8983/solr/core1_shard1_replica1/
> >> > >> >> <http://host2:8983/solr/core1_shard1_replica1/>.
> >> > >> core=core1_shard1_replica2*
> >> > >> >> *INFO [2014-04-14 18:11:00.536]
> >> > >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new
> >> http
> >> > >> >> client,
> >> > >> >>
> >> > >>
> >> >
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
> >> > >> >> *INFO [2014-04-14 18:11:01.964]
> >> > >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new
> >> http
> >> > >> >> client,
> >> > >> >>
> >> > >>
> >> >
> >>
> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
> >> > >> >> *INFO [2014-04-14 18:11:01.969]
> >> [org.apache.solr.handler.SnapPuller]
> >> > >> No
> >> > >> >> value set for 'pollInterval'. Timer Task not started.*
> >> > >> >> *INFO [2014-04-14 18:11:01.973]
> >> [org.apache.solr.handler.SnapPuller]
> >> > >> >> Master's generation: 1108645*
> >> > >> >> *INFO [2014-04-14 18:11:01.973]
> >> [org.apache.solr.handler.SnapPuller]
> >> > >> >> Slave's generation: 1108627*
> >> > >> >> *INFO [2014-04-14 18:11:01.973]
> >> [org.apache.solr.handler.SnapPuller]
> >> > >> >> Starting replication process*
> >> > >> >> *INFO [2014-04-14 18:11:02.007]
> >> [org.apache.solr.handler.SnapPuller]
> >> > >> >> Number of files in latest index in master: 814*
> >> > >> >> *INFO [2014-04-14 18:11:02.007]
> >> > >> >> [org.apache.solr.core.CachingDirectoryFactory] return new
> directory
> >> > for
> >> > >> >>
> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
> >> > >> >> *INFO [2014-04-14 18:11:02.008]
> >> [org.apache.solr.handler.SnapPuller]
> >> > >> >> Starting download to
> >> > >> >> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@
> >> > >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
> >> > >> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe
> ;
> >> > >> >> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
> >> > >> >>
> >> > >> >> ****************
> >> > >> >>
> >> > >> >>
> >> > >> >> So, it looks like the number of updates is too huge for the
> regular
> >> > >> >> replication and then it goes into full copy of index. And since
> our
> >> > >> index
> >> > >> >> size is very huge (350G), this is causing the cluster to go into
> >> > >> recovery
> >> > >> >> mode forever - trying to copy that huge index.
> >> > >> >>
> >> > >> >> I also read in some thread
> >> > >> >>
> >> > >>
> >> >
> >>
> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthatthereisalimit of 100 documents.
> >> > >> >>
> >> > >> >> I wonder if this has been updated to make that configurable
> since
> >> > that
> >> > >> >> thread. If not, the only option I see is to do a "trickle"
> delete
> >> of
> >> > >> 100
> >> > >> >> documents per second or something.
> >> > >> >>
> >> > >> >> Also - the other suggestion of using "distributed=false" might
> not
> >> > help
> >> > >> >> because the issue currently is that the replication is going to
> >> "full
> >> > >> copy".
> >> > >> >>
> >> > >> >> Any thoughts?
> >> > >> >>
> >> > >> >> Thanks
> >> > >> >> Vinay
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> >> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com>
> wrote:
> >> > >> >>
> >> > >> >>> Yes, that is our approach. We did try deleting a day's worth of
> >> data
> >> > >> at a
> >> > >> >>> time, and that resulted in OOM as well.
> >> > >> >>>
> >> > >> >>> Thanks
> >> > >> >>> Vinay
> >> > >> >>>
> >> > >> >>>
> >> > >> >>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com>
> >> > wrote:
> >> > >> >>>
> >> > >> >>>> Hi;
> >> > >> >>>>
> >> > >> >>>> I mean you can divide the range (i.e. one week at each delete
> >> > >> instead of
> >> > >> >>>> one month) and try to check whether you still get an OOM or
> not.
> >> > >> >>>>
> >> > >> >>>> Thanks;
> >> > >> >>>> Furkan KAMACI
> >> > >> >>>>
> >> > >> >>>>
> >> > >> >>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
> >> > >> >>>>
> >> > >> >>>> > Aman,
> >> > >> >>>> > Yes - Will do!
> >> > >> >>>> >
> >> > >> >>>> > Furkan,
> >> > >> >>>> > How do you mean by 'bulk delete'?
> >> > >> >>>> >
> >> > >> >>>> > -Thanks
> >> > >> >>>> > Vinay
> >> > >> >>>> >
> >> > >> >>>> >
> >> > >> >>>> > On 12 April 2014 14:49, Furkan KAMACI <
> furkankamaci@gmail.com>
> >> > >> wrote:
> >> > >> >>>> >
> >> > >> >>>> > > Hi;
> >> > >> >>>> > >
> >> > >> >>>> > > Do you get any problems when you index your data? On the
> >> other
> >> > >> hand
> >> > >> >>>> > > deleting as bulks and reducing the size of documents may
> help
> >> > you
> >> > >> >>>> not to
> >> > >> >>>> > > hit OOM.
> >> > >> >>>> > >
> >> > >> >>>> > > Thanks;
> >> > >> >>>> > > Furkan KAMACI
> >> > >> >>>> > >
> >> > >> >>>> > >
> >> > >> >>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <
> >> amantandon.10@gmail.com
> >> > >:
> >> > >> >>>> > >
> >> > >> >>>> > > > Vinay please share your experience after trying this
> >> > solution.
> >> > >> >>>> > > >
> >> > >> >>>> > > >
> >> > >> >>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <
> >> > >> pothnis@gmail.com>
> >> > >> >>>> > > wrote:
> >> > >> >>>> > > >
> >> > >> >>>> > > > > The query is something like this:
> >> > >> >>>> > > > >
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > *curl -H 'Content-Type: text/xml' --data
> >> > >> >>>> '<delete><query>param1:(val1
> >> > >> >>>> > > OR
> >> > >> >>>> > > > > val2) AND -param2:(val3 OR val4) AND
> >> > >> date_param:[1383955200000 TO
> >> > >> >>>> > > > > 1385164800000]</query></delete>'
> >> > >> >>>> > > > > 'http://host:port
> /solr/coll-name1/update?commit=true'*
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > Trying to restrict the number of documents deleted via
> >> the
> >> > >> date
> >> > >> >>>> > > > parameter.
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > Had not tried the "distrib=false" option. I could give
> >> > that a
> >> > >> >>>> try.
> >> > >> >>>> > > Thanks
> >> > >> >>>> > > > > for the link! I will check on the cache sizes and
> >> autowarm
> >> > >> >>>> values.
> >> > >> >>>> > Will
> >> > >> >>>> > > > try
> >> > >> >>>> > > > > and disable the caches when I am deleting and give
> that a
> >> > >> try.
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > Thanks Erick and Shawn for your inputs!
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > -Vinay
> >> > >> >>>> > > > >
> >> > >> >>>> > > > >
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > On 11 April 2014 15:28, Shawn Heisey <
> solr@elyograg.org>
> >> > >> wrote:
> >> > >> >>>> > > > >
> >> > >> >>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > >> When we tried to delete the data through a query -
> >> say 1
> >> > >> >>>> > day/month's
> >> > >> >>>> > > > > worth
> >> > >> >>>> > > > > >> of data. But after deleting just 1 month's worth of
> >> > data,
> >> > >> the
> >> > >> >>>> > master
> >> > >> >>>> > > > > node
> >> > >> >>>> > > > > >> is going out of memory - heap space.
> >> > >> >>>> > > > > >>
> >> > >> >>>> > > > > >> Wondering is there any way to incrementally delete
> the
> >> > >> data
> >> > >> >>>> > without
> >> > >> >>>> > > > > >> affecting the cluster adversely.
> >> > >> >>>> > > > > >>
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > > I'm curious about the actual query being used here.
> >> Can
> >> > >> you
> >> > >> >>>> share
> >> > >> >>>> > > it,
> >> > >> >>>> > > > or
> >> > >> >>>> > > > > > a redacted version of it? Perhaps there might be a
> >> clue
> >> > >> there?
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > > Is this a fully distributed delete request? One
> thing
> >> > you
> >> > >> >>>> might
> >> > >> >>>> > try,
> >> > >> >>>> > > > > > assuming Solr even supports it, is sending the same
> >> > delete
> >> > >> >>>> request
> >> > >> >>>> > > > > directly
> >> > >> >>>> > > > > > to each shard core with distrib=false.
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > > Here's a very incomplete list about how you can
> reduce
> >> > Solr
> >> > >> >>>> heap
> >> > >> >>>> > > > > > requirements:
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > >
> http://wiki.apache.org/solr/SolrPerformanceProblems#
> >> > >> >>>> > > > > > Reducing_heap_requirements
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > > Thanks,
> >> > >> >>>> > > > > > Shawn
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > > >
> >> > >> >>>> > > > >
> >> > >> >>>> > > >
> >> > >> >>>> > > >
> >> > >> >>>> > > >
> >> > >> >>>> > > > --
> >> > >> >>>> > > > With Regards
> >> > >> >>>> > > > Aman Tandon
> >> > >> >>>> > > >
> >> > >> >>>> > >
> >> > >> >>>> >
> >> > >> >>>>
> >> > >> >>>
> >> > >> >>>
> >> > >> >>
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
>
Re: deleting large amount data from solr cloud
Posted by Erick Erickson <er...@gmail.com>.
bq: Will it get split at any point later?
"Split" is a little ambiguous here. Will it be copied into two or more
segments? No. Will it disappear? Possibly. Eventually this segment
will be merged if you add enough documents to the system. Consider
this scenario:
you add 1M docs to your system and it results in 10 segments (numbers
made up). Then you optimize, and you have 1M docs in 1 segment. Fine
so far.
Now you add 750K of those docs over again, which will delete them from
the 1 big segment. Your merge policy will, at some point, select this
segment to merge and it'll disappear...
FWIW,
Erick@Pedantic.com
On Thu, Apr 17, 2014 at 7:24 AM, Vinay Pothnis <po...@gmail.com> wrote:
> Thanks a lot Shalin!
>
>
> On 16 April 2014 21:26, Shalin Shekhar Mangar <sh...@gmail.com>wrote:
>
>> You can specify maxSegments parameter e.g. maxSegments=5 while optimizing.
>>
>>
>> On Thu, Apr 17, 2014 at 6:46 AM, Vinay Pothnis <po...@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > Couple of follow up questions:
>> >
>> > * When the optimize command is run, looks like it creates one big segment
>> > (forceMerge = 1). Will it get split at any point later? Or will that big
>> > segment remain?
>> >
>> > * Is there anyway to maintain the number of segments - but still merge to
>> > reclaim the deleted documents space? In other words, can I issue
>> > "forceMerge=20"? If so, how would the command look like? Any examples for
>> > this?
>> >
>> > Thanks
>> > Vinay
>> >
>> >
>> >
>> > On 16 April 2014 07:59, Vinay Pothnis <po...@gmail.com> wrote:
>> >
>> > > Thank you Erick!
>> > > Yes - I am using the expunge deletes option.
>> > >
>> > > Thanks for the note on disk space for the optimize command. I should
>> have
>> > > enough space for that. What about the heap space requirement? I hope it
>> > can
>> > > do the optimize with the memory that is allocated to it.
>> > >
>> > > Thanks
>> > > Vinay
>> > >
>> > >
>> > > On 16 April 2014 04:52, Erick Erickson <er...@gmail.com>
>> wrote:
>> > >
>> > >> The optimize should, indeed, reduce the index size. Be aware that it
>> > >> may consume 2x the disk space. You may also try expungedeletes, see
>> > >> here: https://wiki.apache.org/solr/UpdateXmlMessages
>> > >>
>> > >> Best,
>> > >> Erick
>> > >>
>> > >> On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <po...@gmail.com>
>> > >> wrote:
>> > >> > Another update:
>> > >> >
>> > >> > I removed the replicas - to avoid the replication doing a full
>> copy. I
>> > >> am
>> > >> > able delete sizeable chunks of data.
>> > >> > But the overall index size remains the same even after the deletes.
>> It
>> > >> does
>> > >> > not seem to go down.
>> > >> >
>> > >> > I understand that Solr would do this in background - but I don't
>> seem
>> > to
>> > >> > see the decrease in overall index size even after 1-2 hours.
>> > >> > I can see a bunch of ".del" files in the index directory, but the it
>> > >> does
>> > >> > not seem to get cleaned up. Is there anyway to monitor/follow the
>> > >> progress
>> > >> > of index compaction?
>> > >> >
>> > >> > Also, does triggering "optimize" from the admin UI help to compact
>> the
>> > >> > index size on disk?
>> > >> >
>> > >> > Thanks
>> > >> > Vinay
>> > >> >
>> > >> >
>> > >> > On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
>> > >> >
>> > >> >> Some update:
>> > >> >>
>> > >> >> I removed the auto warm configurations for the various caches and
>> > >> reduced
>> > >> >> the cache sizes. I then issued a call to delete a day's worth of
>> data
>> > >> (800K
>> > >> >> documents).
>> > >> >>
>> > >> >> There was no out of memory this time - but some of the nodes went
>> > into
>> > >> >> recovery mode. Was able to catch some logs this time around and
>> this
>> > is
>> > >> >> what i see:
>> > >> >>
>> > >> >> ****************
>> > >> >> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
>> > >> >> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
>> > >> >> <http://host1:8983/solr> too many updates received since start -
>> > >> >> startingUpdates no longer overlaps with our currentUpdates*
>> > >> >> *INFO [2014-04-14 18:11:00.476]
>> > >> [org.apache.solr.cloud.RecoveryStrategy]
>> > >> >> PeerSync Recovery was not successful - trying replication.
>> > >> >> core=core1_shard1_replica2*
>> > >> >> *INFO [2014-04-14 18:11:00.476]
>> > >> [org.apache.solr.cloud.RecoveryStrategy]
>> > >> >> Starting Replication Recovery. core=core1_shard1_replica2*
>> > >> >> *INFO [2014-04-14 18:11:00.535]
>> > >> [org.apache.solr.cloud.RecoveryStrategy]
>> > >> >> Begin buffering updates. core=core1_shard1_replica2*
>> > >> >> *INFO [2014-04-14 18:11:00.536]
>> > >> [org.apache.solr.cloud.RecoveryStrategy]
>> > >> >> Attempting to replicate from
>> > >> http://host2:8983/solr/core1_shard1_replica1/
>> > >> >> <http://host2:8983/solr/core1_shard1_replica1/>.
>> > >> core=core1_shard1_replica2*
>> > >> >> *INFO [2014-04-14 18:11:00.536]
>> > >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new
>> http
>> > >> >> client,
>> > >> >>
>> > >>
>> > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
>> > >> >> *INFO [2014-04-14 18:11:01.964]
>> > >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new
>> http
>> > >> >> client,
>> > >> >>
>> > >>
>> >
>> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
>> > >> >> *INFO [2014-04-14 18:11:01.969]
>> [org.apache.solr.handler.SnapPuller]
>> > >> No
>> > >> >> value set for 'pollInterval'. Timer Task not started.*
>> > >> >> *INFO [2014-04-14 18:11:01.973]
>> [org.apache.solr.handler.SnapPuller]
>> > >> >> Master's generation: 1108645*
>> > >> >> *INFO [2014-04-14 18:11:01.973]
>> [org.apache.solr.handler.SnapPuller]
>> > >> >> Slave's generation: 1108627*
>> > >> >> *INFO [2014-04-14 18:11:01.973]
>> [org.apache.solr.handler.SnapPuller]
>> > >> >> Starting replication process*
>> > >> >> *INFO [2014-04-14 18:11:02.007]
>> [org.apache.solr.handler.SnapPuller]
>> > >> >> Number of files in latest index in master: 814*
>> > >> >> *INFO [2014-04-14 18:11:02.007]
>> > >> >> [org.apache.solr.core.CachingDirectoryFactory] return new directory
>> > for
>> > >> >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
>> > >> >> *INFO [2014-04-14 18:11:02.008]
>> [org.apache.solr.handler.SnapPuller]
>> > >> >> Starting download to
>> > >> >> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@
>> > >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
>> > >> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
>> > >> >> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
>> > >> >>
>> > >> >> ****************
>> > >> >>
>> > >> >>
>> > >> >> So, it looks like the number of updates is too huge for the regular
>> > >> >> replication and then it goes into full copy of index. And since our
>> > >> index
>> > >> >> size is very huge (350G), this is causing the cluster to go into
>> > >> recovery
>> > >> >> mode forever - trying to copy that huge index.
>> > >> >>
>> > >> >> I also read in some thread
>> > >> >>
>> > >>
>> >
>> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthatthereisa limit of 100 documents.
>> > >> >>
>> > >> >> I wonder if this has been updated to make that configurable since
>> > that
>> > >> >> thread. If not, the only option I see is to do a "trickle" delete
>> of
>> > >> 100
>> > >> >> documents per second or something.
>> > >> >>
>> > >> >> Also - the other suggestion of using "distributed=false" might not
>> > help
>> > >> >> because the issue currently is that the replication is going to
>> "full
>> > >> copy".
>> > >> >>
>> > >> >> Any thoughts?
>> > >> >>
>> > >> >> Thanks
>> > >> >> Vinay
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
>> > >> >>
>> > >> >>> Yes, that is our approach. We did try deleting a day's worth of
>> data
>> > >> at a
>> > >> >>> time, and that resulted in OOM as well.
>> > >> >>>
>> > >> >>> Thanks
>> > >> >>> Vinay
>> > >> >>>
>> > >> >>>
>> > >> >>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com>
>> > wrote:
>> > >> >>>
>> > >> >>>> Hi;
>> > >> >>>>
>> > >> >>>> I mean you can divide the range (i.e. one week at each delete
>> > >> instead of
>> > >> >>>> one month) and try to check whether you still get an OOM or not.
>> > >> >>>>
>> > >> >>>> Thanks;
>> > >> >>>> Furkan KAMACI
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
>> > >> >>>>
>> > >> >>>> > Aman,
>> > >> >>>> > Yes - Will do!
>> > >> >>>> >
>> > >> >>>> > Furkan,
>> > >> >>>> > How do you mean by 'bulk delete'?
>> > >> >>>> >
>> > >> >>>> > -Thanks
>> > >> >>>> > Vinay
>> > >> >>>> >
>> > >> >>>> >
>> > >> >>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com>
>> > >> wrote:
>> > >> >>>> >
>> > >> >>>> > > Hi;
>> > >> >>>> > >
>> > >> >>>> > > Do you get any problems when you index your data? On the
>> other
>> > >> hand
>> > >> >>>> > > deleting as bulks and reducing the size of documents may help
>> > you
>> > >> >>>> not to
>> > >> >>>> > > hit OOM.
>> > >> >>>> > >
>> > >> >>>> > > Thanks;
>> > >> >>>> > > Furkan KAMACI
>> > >> >>>> > >
>> > >> >>>> > >
>> > >> >>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <
>> amantandon.10@gmail.com
>> > >:
>> > >> >>>> > >
>> > >> >>>> > > > Vinay please share your experience after trying this
>> > solution.
>> > >> >>>> > > >
>> > >> >>>> > > >
>> > >> >>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <
>> > >> pothnis@gmail.com>
>> > >> >>>> > > wrote:
>> > >> >>>> > > >
>> > >> >>>> > > > > The query is something like this:
>> > >> >>>> > > > >
>> > >> >>>> > > > >
>> > >> >>>> > > > > *curl -H 'Content-Type: text/xml' --data
>> > >> >>>> '<delete><query>param1:(val1
>> > >> >>>> > > OR
>> > >> >>>> > > > > val2) AND -param2:(val3 OR val4) AND
>> > >> date_param:[1383955200000 TO
>> > >> >>>> > > > > 1385164800000]</query></delete>'
>> > >> >>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
>> > >> >>>> > > > >
>> > >> >>>> > > > > Trying to restrict the number of documents deleted via
>> the
>> > >> date
>> > >> >>>> > > > parameter.
>> > >> >>>> > > > >
>> > >> >>>> > > > > Had not tried the "distrib=false" option. I could give
>> > that a
>> > >> >>>> try.
>> > >> >>>> > > Thanks
>> > >> >>>> > > > > for the link! I will check on the cache sizes and
>> autowarm
>> > >> >>>> values.
>> > >> >>>> > Will
>> > >> >>>> > > > try
>> > >> >>>> > > > > and disable the caches when I am deleting and give that a
>> > >> try.
>> > >> >>>> > > > >
>> > >> >>>> > > > > Thanks Erick and Shawn for your inputs!
>> > >> >>>> > > > >
>> > >> >>>> > > > > -Vinay
>> > >> >>>> > > > >
>> > >> >>>> > > > >
>> > >> >>>> > > > >
>> > >> >>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org>
>> > >> wrote:
>> > >> >>>> > > > >
>> > >> >>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
>> > >> >>>> > > > > >
>> > >> >>>> > > > > >> When we tried to delete the data through a query -
>> say 1
>> > >> >>>> > day/month's
>> > >> >>>> > > > > worth
>> > >> >>>> > > > > >> of data. But after deleting just 1 month's worth of
>> > data,
>> > >> the
>> > >> >>>> > master
>> > >> >>>> > > > > node
>> > >> >>>> > > > > >> is going out of memory - heap space.
>> > >> >>>> > > > > >>
>> > >> >>>> > > > > >> Wondering is there any way to incrementally delete the
>> > >> data
>> > >> >>>> > without
>> > >> >>>> > > > > >> affecting the cluster adversely.
>> > >> >>>> > > > > >>
>> > >> >>>> > > > > >
>> > >> >>>> > > > > > I'm curious about the actual query being used here.
>> Can
>> > >> you
>> > >> >>>> share
>> > >> >>>> > > it,
>> > >> >>>> > > > or
>> > >> >>>> > > > > > a redacted version of it? Perhaps there might be a
>> clue
>> > >> there?
>> > >> >>>> > > > > >
>> > >> >>>> > > > > > Is this a fully distributed delete request? One thing
>> > you
>> > >> >>>> might
>> > >> >>>> > try,
>> > >> >>>> > > > > > assuming Solr even supports it, is sending the same
>> > delete
>> > >> >>>> request
>> > >> >>>> > > > > directly
>> > >> >>>> > > > > > to each shard core with distrib=false.
>> > >> >>>> > > > > >
>> > >> >>>> > > > > > Here's a very incomplete list about how you can reduce
>> > Solr
>> > >> >>>> heap
>> > >> >>>> > > > > > requirements:
>> > >> >>>> > > > > >
>> > >> >>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
>> > >> >>>> > > > > > Reducing_heap_requirements
>> > >> >>>> > > > > >
>> > >> >>>> > > > > > Thanks,
>> > >> >>>> > > > > > Shawn
>> > >> >>>> > > > > >
>> > >> >>>> > > > > >
>> > >> >>>> > > > >
>> > >> >>>> > > >
>> > >> >>>> > > >
>> > >> >>>> > > >
>> > >> >>>> > > > --
>> > >> >>>> > > > With Regards
>> > >> >>>> > > > Aman Tandon
>> > >> >>>> > > >
>> > >> >>>> > >
>> > >> >>>> >
>> > >> >>>>
>> > >> >>>
>> > >> >>>
>> > >> >>
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Thanks a lot Shalin!
On 16 April 2014 21:26, Shalin Shekhar Mangar <sh...@gmail.com>wrote:
> You can specify maxSegments parameter e.g. maxSegments=5 while optimizing.
>
>
> On Thu, Apr 17, 2014 at 6:46 AM, Vinay Pothnis <po...@gmail.com> wrote:
>
> > Hello,
> >
> > Couple of follow up questions:
> >
> > * When the optimize command is run, looks like it creates one big segment
> > (forceMerge = 1). Will it get split at any point later? Or will that big
> > segment remain?
> >
> > * Is there anyway to maintain the number of segments - but still merge to
> > reclaim the deleted documents space? In other words, can I issue
> > "forceMerge=20"? If so, how would the command look like? Any examples for
> > this?
> >
> > Thanks
> > Vinay
> >
> >
> >
> > On 16 April 2014 07:59, Vinay Pothnis <po...@gmail.com> wrote:
> >
> > > Thank you Erick!
> > > Yes - I am using the expunge deletes option.
> > >
> > > Thanks for the note on disk space for the optimize command. I should
> have
> > > enough space for that. What about the heap space requirement? I hope it
> > can
> > > do the optimize with the memory that is allocated to it.
> > >
> > > Thanks
> > > Vinay
> > >
> > >
> > > On 16 April 2014 04:52, Erick Erickson <er...@gmail.com>
> wrote:
> > >
> > >> The optimize should, indeed, reduce the index size. Be aware that it
> > >> may consume 2x the disk space. You may also try expungedeletes, see
> > >> here: https://wiki.apache.org/solr/UpdateXmlMessages
> > >>
> > >> Best,
> > >> Erick
> > >>
> > >> On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <po...@gmail.com>
> > >> wrote:
> > >> > Another update:
> > >> >
> > >> > I removed the replicas - to avoid the replication doing a full
> copy. I
> > >> am
> > >> > able delete sizeable chunks of data.
> > >> > But the overall index size remains the same even after the deletes.
> It
> > >> does
> > >> > not seem to go down.
> > >> >
> > >> > I understand that Solr would do this in background - but I don't
> seem
> > to
> > >> > see the decrease in overall index size even after 1-2 hours.
> > >> > I can see a bunch of ".del" files in the index directory, but the it
> > >> does
> > >> > not seem to get cleaned up. Is there anyway to monitor/follow the
> > >> progress
> > >> > of index compaction?
> > >> >
> > >> > Also, does triggering "optimize" from the admin UI help to compact
> the
> > >> > index size on disk?
> > >> >
> > >> > Thanks
> > >> > Vinay
> > >> >
> > >> >
> > >> > On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
> > >> >
> > >> >> Some update:
> > >> >>
> > >> >> I removed the auto warm configurations for the various caches and
> > >> reduced
> > >> >> the cache sizes. I then issued a call to delete a day's worth of
> data
> > >> (800K
> > >> >> documents).
> > >> >>
> > >> >> There was no out of memory this time - but some of the nodes went
> > into
> > >> >> recovery mode. Was able to catch some logs this time around and
> this
> > is
> > >> >> what i see:
> > >> >>
> > >> >> ****************
> > >> >> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
> > >> >> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
> > >> >> <http://host1:8983/solr> too many updates received since start -
> > >> >> startingUpdates no longer overlaps with our currentUpdates*
> > >> >> *INFO [2014-04-14 18:11:00.476]
> > >> [org.apache.solr.cloud.RecoveryStrategy]
> > >> >> PeerSync Recovery was not successful - trying replication.
> > >> >> core=core1_shard1_replica2*
> > >> >> *INFO [2014-04-14 18:11:00.476]
> > >> [org.apache.solr.cloud.RecoveryStrategy]
> > >> >> Starting Replication Recovery. core=core1_shard1_replica2*
> > >> >> *INFO [2014-04-14 18:11:00.535]
> > >> [org.apache.solr.cloud.RecoveryStrategy]
> > >> >> Begin buffering updates. core=core1_shard1_replica2*
> > >> >> *INFO [2014-04-14 18:11:00.536]
> > >> [org.apache.solr.cloud.RecoveryStrategy]
> > >> >> Attempting to replicate from
> > >> http://host2:8983/solr/core1_shard1_replica1/
> > >> >> <http://host2:8983/solr/core1_shard1_replica1/>.
> > >> core=core1_shard1_replica2*
> > >> >> *INFO [2014-04-14 18:11:00.536]
> > >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new
> http
> > >> >> client,
> > >> >>
> > >>
> > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
> > >> >> *INFO [2014-04-14 18:11:01.964]
> > >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new
> http
> > >> >> client,
> > >> >>
> > >>
> >
> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
> > >> >> *INFO [2014-04-14 18:11:01.969]
> [org.apache.solr.handler.SnapPuller]
> > >> No
> > >> >> value set for 'pollInterval'. Timer Task not started.*
> > >> >> *INFO [2014-04-14 18:11:01.973]
> [org.apache.solr.handler.SnapPuller]
> > >> >> Master's generation: 1108645*
> > >> >> *INFO [2014-04-14 18:11:01.973]
> [org.apache.solr.handler.SnapPuller]
> > >> >> Slave's generation: 1108627*
> > >> >> *INFO [2014-04-14 18:11:01.973]
> [org.apache.solr.handler.SnapPuller]
> > >> >> Starting replication process*
> > >> >> *INFO [2014-04-14 18:11:02.007]
> [org.apache.solr.handler.SnapPuller]
> > >> >> Number of files in latest index in master: 814*
> > >> >> *INFO [2014-04-14 18:11:02.007]
> > >> >> [org.apache.solr.core.CachingDirectoryFactory] return new directory
> > for
> > >> >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
> > >> >> *INFO [2014-04-14 18:11:02.008]
> [org.apache.solr.handler.SnapPuller]
> > >> >> Starting download to
> > >> >> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@
> > >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
> > >> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
> > >> >> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
> > >> >>
> > >> >> ****************
> > >> >>
> > >> >>
> > >> >> So, it looks like the number of updates is too huge for the regular
> > >> >> replication and then it goes into full copy of index. And since our
> > >> index
> > >> >> size is very huge (350G), this is causing the cluster to go into
> > >> recovery
> > >> >> mode forever - trying to copy that huge index.
> > >> >>
> > >> >> I also read in some thread
> > >> >>
> > >>
> >
> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthatthereisa limit of 100 documents.
> > >> >>
> > >> >> I wonder if this has been updated to make that configurable since
> > that
> > >> >> thread. If not, the only option I see is to do a "trickle" delete
> of
> > >> 100
> > >> >> documents per second or something.
> > >> >>
> > >> >> Also - the other suggestion of using "distributed=false" might not
> > help
> > >> >> because the issue currently is that the replication is going to
> "full
> > >> copy".
> > >> >>
> > >> >> Any thoughts?
> > >> >>
> > >> >> Thanks
> > >> >> Vinay
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
> > >> >>
> > >> >>> Yes, that is our approach. We did try deleting a day's worth of
> data
> > >> at a
> > >> >>> time, and that resulted in OOM as well.
> > >> >>>
> > >> >>> Thanks
> > >> >>> Vinay
> > >> >>>
> > >> >>>
> > >> >>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com>
> > wrote:
> > >> >>>
> > >> >>>> Hi;
> > >> >>>>
> > >> >>>> I mean you can divide the range (i.e. one week at each delete
> > >> instead of
> > >> >>>> one month) and try to check whether you still get an OOM or not.
> > >> >>>>
> > >> >>>> Thanks;
> > >> >>>> Furkan KAMACI
> > >> >>>>
> > >> >>>>
> > >> >>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
> > >> >>>>
> > >> >>>> > Aman,
> > >> >>>> > Yes - Will do!
> > >> >>>> >
> > >> >>>> > Furkan,
> > >> >>>> > How do you mean by 'bulk delete'?
> > >> >>>> >
> > >> >>>> > -Thanks
> > >> >>>> > Vinay
> > >> >>>> >
> > >> >>>> >
> > >> >>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com>
> > >> wrote:
> > >> >>>> >
> > >> >>>> > > Hi;
> > >> >>>> > >
> > >> >>>> > > Do you get any problems when you index your data? On the
> other
> > >> hand
> > >> >>>> > > deleting as bulks and reducing the size of documents may help
> > you
> > >> >>>> not to
> > >> >>>> > > hit OOM.
> > >> >>>> > >
> > >> >>>> > > Thanks;
> > >> >>>> > > Furkan KAMACI
> > >> >>>> > >
> > >> >>>> > >
> > >> >>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <
> amantandon.10@gmail.com
> > >:
> > >> >>>> > >
> > >> >>>> > > > Vinay please share your experience after trying this
> > solution.
> > >> >>>> > > >
> > >> >>>> > > >
> > >> >>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <
> > >> pothnis@gmail.com>
> > >> >>>> > > wrote:
> > >> >>>> > > >
> > >> >>>> > > > > The query is something like this:
> > >> >>>> > > > >
> > >> >>>> > > > >
> > >> >>>> > > > > *curl -H 'Content-Type: text/xml' --data
> > >> >>>> '<delete><query>param1:(val1
> > >> >>>> > > OR
> > >> >>>> > > > > val2) AND -param2:(val3 OR val4) AND
> > >> date_param:[1383955200000 TO
> > >> >>>> > > > > 1385164800000]</query></delete>'
> > >> >>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
> > >> >>>> > > > >
> > >> >>>> > > > > Trying to restrict the number of documents deleted via
> the
> > >> date
> > >> >>>> > > > parameter.
> > >> >>>> > > > >
> > >> >>>> > > > > Had not tried the "distrib=false" option. I could give
> > that a
> > >> >>>> try.
> > >> >>>> > > Thanks
> > >> >>>> > > > > for the link! I will check on the cache sizes and
> autowarm
> > >> >>>> values.
> > >> >>>> > Will
> > >> >>>> > > > try
> > >> >>>> > > > > and disable the caches when I am deleting and give that a
> > >> try.
> > >> >>>> > > > >
> > >> >>>> > > > > Thanks Erick and Shawn for your inputs!
> > >> >>>> > > > >
> > >> >>>> > > > > -Vinay
> > >> >>>> > > > >
> > >> >>>> > > > >
> > >> >>>> > > > >
> > >> >>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org>
> > >> wrote:
> > >> >>>> > > > >
> > >> >>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> > >> >>>> > > > > >
> > >> >>>> > > > > >> When we tried to delete the data through a query -
> say 1
> > >> >>>> > day/month's
> > >> >>>> > > > > worth
> > >> >>>> > > > > >> of data. But after deleting just 1 month's worth of
> > data,
> > >> the
> > >> >>>> > master
> > >> >>>> > > > > node
> > >> >>>> > > > > >> is going out of memory - heap space.
> > >> >>>> > > > > >>
> > >> >>>> > > > > >> Wondering is there any way to incrementally delete the
> > >> data
> > >> >>>> > without
> > >> >>>> > > > > >> affecting the cluster adversely.
> > >> >>>> > > > > >>
> > >> >>>> > > > > >
> > >> >>>> > > > > > I'm curious about the actual query being used here.
> Can
> > >> you
> > >> >>>> share
> > >> >>>> > > it,
> > >> >>>> > > > or
> > >> >>>> > > > > > a redacted version of it? Perhaps there might be a
> clue
> > >> there?
> > >> >>>> > > > > >
> > >> >>>> > > > > > Is this a fully distributed delete request? One thing
> > you
> > >> >>>> might
> > >> >>>> > try,
> > >> >>>> > > > > > assuming Solr even supports it, is sending the same
> > delete
> > >> >>>> request
> > >> >>>> > > > > directly
> > >> >>>> > > > > > to each shard core with distrib=false.
> > >> >>>> > > > > >
> > >> >>>> > > > > > Here's a very incomplete list about how you can reduce
> > Solr
> > >> >>>> heap
> > >> >>>> > > > > > requirements:
> > >> >>>> > > > > >
> > >> >>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > >> >>>> > > > > > Reducing_heap_requirements
> > >> >>>> > > > > >
> > >> >>>> > > > > > Thanks,
> > >> >>>> > > > > > Shawn
> > >> >>>> > > > > >
> > >> >>>> > > > > >
> > >> >>>> > > > >
> > >> >>>> > > >
> > >> >>>> > > >
> > >> >>>> > > >
> > >> >>>> > > > --
> > >> >>>> > > > With Regards
> > >> >>>> > > > Aman Tandon
> > >> >>>> > > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> >>
> > >>
> > >
> > >
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: deleting large amount data from solr cloud
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
You can specify maxSegments parameter e.g. maxSegments=5 while optimizing.
On Thu, Apr 17, 2014 at 6:46 AM, Vinay Pothnis <po...@gmail.com> wrote:
> Hello,
>
> Couple of follow up questions:
>
> * When the optimize command is run, looks like it creates one big segment
> (forceMerge = 1). Will it get split at any point later? Or will that big
> segment remain?
>
> * Is there anyway to maintain the number of segments - but still merge to
> reclaim the deleted documents space? In other words, can I issue
> "forceMerge=20"? If so, how would the command look like? Any examples for
> this?
>
> Thanks
> Vinay
>
>
>
> On 16 April 2014 07:59, Vinay Pothnis <po...@gmail.com> wrote:
>
> > Thank you Erick!
> > Yes - I am using the expunge deletes option.
> >
> > Thanks for the note on disk space for the optimize command. I should have
> > enough space for that. What about the heap space requirement? I hope it
> can
> > do the optimize with the memory that is allocated to it.
> >
> > Thanks
> > Vinay
> >
> >
> > On 16 April 2014 04:52, Erick Erickson <er...@gmail.com> wrote:
> >
> >> The optimize should, indeed, reduce the index size. Be aware that it
> >> may consume 2x the disk space. You may also try expungedeletes, see
> >> here: https://wiki.apache.org/solr/UpdateXmlMessages
> >>
> >> Best,
> >> Erick
> >>
> >> On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <po...@gmail.com>
> >> wrote:
> >> > Another update:
> >> >
> >> > I removed the replicas - to avoid the replication doing a full copy. I
> >> am
> >> > able delete sizeable chunks of data.
> >> > But the overall index size remains the same even after the deletes. It
> >> does
> >> > not seem to go down.
> >> >
> >> > I understand that Solr would do this in background - but I don't seem
> to
> >> > see the decrease in overall index size even after 1-2 hours.
> >> > I can see a bunch of ".del" files in the index directory, but the it
> >> does
> >> > not seem to get cleaned up. Is there anyway to monitor/follow the
> >> progress
> >> > of index compaction?
> >> >
> >> > Also, does triggering "optimize" from the admin UI help to compact the
> >> > index size on disk?
> >> >
> >> > Thanks
> >> > Vinay
> >> >
> >> >
> >> > On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
> >> >
> >> >> Some update:
> >> >>
> >> >> I removed the auto warm configurations for the various caches and
> >> reduced
> >> >> the cache sizes. I then issued a call to delete a day's worth of data
> >> (800K
> >> >> documents).
> >> >>
> >> >> There was no out of memory this time - but some of the nodes went
> into
> >> >> recovery mode. Was able to catch some logs this time around and this
> is
> >> >> what i see:
> >> >>
> >> >> ****************
> >> >> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
> >> >> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
> >> >> <http://host1:8983/solr> too many updates received since start -
> >> >> startingUpdates no longer overlaps with our currentUpdates*
> >> >> *INFO [2014-04-14 18:11:00.476]
> >> [org.apache.solr.cloud.RecoveryStrategy]
> >> >> PeerSync Recovery was not successful - trying replication.
> >> >> core=core1_shard1_replica2*
> >> >> *INFO [2014-04-14 18:11:00.476]
> >> [org.apache.solr.cloud.RecoveryStrategy]
> >> >> Starting Replication Recovery. core=core1_shard1_replica2*
> >> >> *INFO [2014-04-14 18:11:00.535]
> >> [org.apache.solr.cloud.RecoveryStrategy]
> >> >> Begin buffering updates. core=core1_shard1_replica2*
> >> >> *INFO [2014-04-14 18:11:00.536]
> >> [org.apache.solr.cloud.RecoveryStrategy]
> >> >> Attempting to replicate from
> >> http://host2:8983/solr/core1_shard1_replica1/
> >> >> <http://host2:8983/solr/core1_shard1_replica1/>.
> >> core=core1_shard1_replica2*
> >> >> *INFO [2014-04-14 18:11:00.536]
> >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
> >> >> client,
> >> >>
> >>
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
> >> >> *INFO [2014-04-14 18:11:01.964]
> >> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
> >> >> client,
> >> >>
> >>
> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
> >> >> *INFO [2014-04-14 18:11:01.969] [org.apache.solr.handler.SnapPuller]
> >> No
> >> >> value set for 'pollInterval'. Timer Task not started.*
> >> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> >> >> Master's generation: 1108645*
> >> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> >> >> Slave's generation: 1108627*
> >> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> >> >> Starting replication process*
> >> >> *INFO [2014-04-14 18:11:02.007] [org.apache.solr.handler.SnapPuller]
> >> >> Number of files in latest index in master: 814*
> >> >> *INFO [2014-04-14 18:11:02.007]
> >> >> [org.apache.solr.core.CachingDirectoryFactory] return new directory
> for
> >> >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
> >> >> *INFO [2014-04-14 18:11:02.008] [org.apache.solr.handler.SnapPuller]
> >> >> Starting download to
> >> >> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@
> >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
> >> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
> >> >> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
> >> >>
> >> >> ****************
> >> >>
> >> >>
> >> >> So, it looks like the number of updates is too huge for the regular
> >> >> replication and then it goes into full copy of index. And since our
> >> index
> >> >> size is very huge (350G), this is causing the cluster to go into
> >> recovery
> >> >> mode forever - trying to copy that huge index.
> >> >>
> >> >> I also read in some thread
> >> >>
> >>
> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthatthereis a limit of 100 documents.
> >> >>
> >> >> I wonder if this has been updated to make that configurable since
> that
> >> >> thread. If not, the only option I see is to do a "trickle" delete of
> >> 100
> >> >> documents per second or something.
> >> >>
> >> >> Also - the other suggestion of using "distributed=false" might not
> help
> >> >> because the issue currently is that the replication is going to "full
> >> copy".
> >> >>
> >> >> Any thoughts?
> >> >>
> >> >> Thanks
> >> >> Vinay
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
> >> >>
> >> >>> Yes, that is our approach. We did try deleting a day's worth of data
> >> at a
> >> >>> time, and that resulted in OOM as well.
> >> >>>
> >> >>> Thanks
> >> >>> Vinay
> >> >>>
> >> >>>
> >> >>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com>
> wrote:
> >> >>>
> >> >>>> Hi;
> >> >>>>
> >> >>>> I mean you can divide the range (i.e. one week at each delete
> >> instead of
> >> >>>> one month) and try to check whether you still get an OOM or not.
> >> >>>>
> >> >>>> Thanks;
> >> >>>> Furkan KAMACI
> >> >>>>
> >> >>>>
> >> >>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
> >> >>>>
> >> >>>> > Aman,
> >> >>>> > Yes - Will do!
> >> >>>> >
> >> >>>> > Furkan,
> >> >>>> > How do you mean by 'bulk delete'?
> >> >>>> >
> >> >>>> > -Thanks
> >> >>>> > Vinay
> >> >>>> >
> >> >>>> >
> >> >>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com>
> >> wrote:
> >> >>>> >
> >> >>>> > > Hi;
> >> >>>> > >
> >> >>>> > > Do you get any problems when you index your data? On the other
> >> hand
> >> >>>> > > deleting as bulks and reducing the size of documents may help
> you
> >> >>>> not to
> >> >>>> > > hit OOM.
> >> >>>> > >
> >> >>>> > > Thanks;
> >> >>>> > > Furkan KAMACI
> >> >>>> > >
> >> >>>> > >
> >> >>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <amantandon.10@gmail.com
> >:
> >> >>>> > >
> >> >>>> > > > Vinay please share your experience after trying this
> solution.
> >> >>>> > > >
> >> >>>> > > >
> >> >>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <
> >> pothnis@gmail.com>
> >> >>>> > > wrote:
> >> >>>> > > >
> >> >>>> > > > > The query is something like this:
> >> >>>> > > > >
> >> >>>> > > > >
> >> >>>> > > > > *curl -H 'Content-Type: text/xml' --data
> >> >>>> '<delete><query>param1:(val1
> >> >>>> > > OR
> >> >>>> > > > > val2) AND -param2:(val3 OR val4) AND
> >> date_param:[1383955200000 TO
> >> >>>> > > > > 1385164800000]</query></delete>'
> >> >>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
> >> >>>> > > > >
> >> >>>> > > > > Trying to restrict the number of documents deleted via the
> >> date
> >> >>>> > > > parameter.
> >> >>>> > > > >
> >> >>>> > > > > Had not tried the "distrib=false" option. I could give
> that a
> >> >>>> try.
> >> >>>> > > Thanks
> >> >>>> > > > > for the link! I will check on the cache sizes and autowarm
> >> >>>> values.
> >> >>>> > Will
> >> >>>> > > > try
> >> >>>> > > > > and disable the caches when I am deleting and give that a
> >> try.
> >> >>>> > > > >
> >> >>>> > > > > Thanks Erick and Shawn for your inputs!
> >> >>>> > > > >
> >> >>>> > > > > -Vinay
> >> >>>> > > > >
> >> >>>> > > > >
> >> >>>> > > > >
> >> >>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org>
> >> wrote:
> >> >>>> > > > >
> >> >>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> >> >>>> > > > > >
> >> >>>> > > > > >> When we tried to delete the data through a query - say 1
> >> >>>> > day/month's
> >> >>>> > > > > worth
> >> >>>> > > > > >> of data. But after deleting just 1 month's worth of
> data,
> >> the
> >> >>>> > master
> >> >>>> > > > > node
> >> >>>> > > > > >> is going out of memory - heap space.
> >> >>>> > > > > >>
> >> >>>> > > > > >> Wondering is there any way to incrementally delete the
> >> data
> >> >>>> > without
> >> >>>> > > > > >> affecting the cluster adversely.
> >> >>>> > > > > >>
> >> >>>> > > > > >
> >> >>>> > > > > > I'm curious about the actual query being used here. Can
> >> you
> >> >>>> share
> >> >>>> > > it,
> >> >>>> > > > or
> >> >>>> > > > > > a redacted version of it? Perhaps there might be a clue
> >> there?
> >> >>>> > > > > >
> >> >>>> > > > > > Is this a fully distributed delete request? One thing
> you
> >> >>>> might
> >> >>>> > try,
> >> >>>> > > > > > assuming Solr even supports it, is sending the same
> delete
> >> >>>> request
> >> >>>> > > > > directly
> >> >>>> > > > > > to each shard core with distrib=false.
> >> >>>> > > > > >
> >> >>>> > > > > > Here's a very incomplete list about how you can reduce
> Solr
> >> >>>> heap
> >> >>>> > > > > > requirements:
> >> >>>> > > > > >
> >> >>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> >> >>>> > > > > > Reducing_heap_requirements
> >> >>>> > > > > >
> >> >>>> > > > > > Thanks,
> >> >>>> > > > > > Shawn
> >> >>>> > > > > >
> >> >>>> > > > > >
> >> >>>> > > > >
> >> >>>> > > >
> >> >>>> > > >
> >> >>>> > > >
> >> >>>> > > > --
> >> >>>> > > > With Regards
> >> >>>> > > > Aman Tandon
> >> >>>> > > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >>
> >
> >
>
--
Regards,
Shalin Shekhar Mangar.
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Hello,
Couple of follow up questions:
* When the optimize command is run, looks like it creates one big segment
(forceMerge = 1). Will it get split at any point later? Or will that big
segment remain?
* Is there anyway to maintain the number of segments - but still merge to
reclaim the deleted documents space? In other words, can I issue
"forceMerge=20"? If so, how would the command look like? Any examples for
this?
Thanks
Vinay
On 16 April 2014 07:59, Vinay Pothnis <po...@gmail.com> wrote:
> Thank you Erick!
> Yes - I am using the expunge deletes option.
>
> Thanks for the note on disk space for the optimize command. I should have
> enough space for that. What about the heap space requirement? I hope it can
> do the optimize with the memory that is allocated to it.
>
> Thanks
> Vinay
>
>
> On 16 April 2014 04:52, Erick Erickson <er...@gmail.com> wrote:
>
>> The optimize should, indeed, reduce the index size. Be aware that it
>> may consume 2x the disk space. You may also try expungedeletes, see
>> here: https://wiki.apache.org/solr/UpdateXmlMessages
>>
>> Best,
>> Erick
>>
>> On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <po...@gmail.com>
>> wrote:
>> > Another update:
>> >
>> > I removed the replicas - to avoid the replication doing a full copy. I
>> am
>> > able delete sizeable chunks of data.
>> > But the overall index size remains the same even after the deletes. It
>> does
>> > not seem to go down.
>> >
>> > I understand that Solr would do this in background - but I don't seem to
>> > see the decrease in overall index size even after 1-2 hours.
>> > I can see a bunch of ".del" files in the index directory, but the it
>> does
>> > not seem to get cleaned up. Is there anyway to monitor/follow the
>> progress
>> > of index compaction?
>> >
>> > Also, does triggering "optimize" from the admin UI help to compact the
>> > index size on disk?
>> >
>> > Thanks
>> > Vinay
>> >
>> >
>> > On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
>> >
>> >> Some update:
>> >>
>> >> I removed the auto warm configurations for the various caches and
>> reduced
>> >> the cache sizes. I then issued a call to delete a day's worth of data
>> (800K
>> >> documents).
>> >>
>> >> There was no out of memory this time - but some of the nodes went into
>> >> recovery mode. Was able to catch some logs this time around and this is
>> >> what i see:
>> >>
>> >> ****************
>> >> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
>> >> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
>> >> <http://host1:8983/solr> too many updates received since start -
>> >> startingUpdates no longer overlaps with our currentUpdates*
>> >> *INFO [2014-04-14 18:11:00.476]
>> [org.apache.solr.cloud.RecoveryStrategy]
>> >> PeerSync Recovery was not successful - trying replication.
>> >> core=core1_shard1_replica2*
>> >> *INFO [2014-04-14 18:11:00.476]
>> [org.apache.solr.cloud.RecoveryStrategy]
>> >> Starting Replication Recovery. core=core1_shard1_replica2*
>> >> *INFO [2014-04-14 18:11:00.535]
>> [org.apache.solr.cloud.RecoveryStrategy]
>> >> Begin buffering updates. core=core1_shard1_replica2*
>> >> *INFO [2014-04-14 18:11:00.536]
>> [org.apache.solr.cloud.RecoveryStrategy]
>> >> Attempting to replicate from
>> http://host2:8983/solr/core1_shard1_replica1/
>> >> <http://host2:8983/solr/core1_shard1_replica1/>.
>> core=core1_shard1_replica2*
>> >> *INFO [2014-04-14 18:11:00.536]
>> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
>> >> client,
>> >>
>> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
>> >> *INFO [2014-04-14 18:11:01.964]
>> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
>> >> client,
>> >>
>> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
>> >> *INFO [2014-04-14 18:11:01.969] [org.apache.solr.handler.SnapPuller]
>> No
>> >> value set for 'pollInterval'. Timer Task not started.*
>> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
>> >> Master's generation: 1108645*
>> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
>> >> Slave's generation: 1108627*
>> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
>> >> Starting replication process*
>> >> *INFO [2014-04-14 18:11:02.007] [org.apache.solr.handler.SnapPuller]
>> >> Number of files in latest index in master: 814*
>> >> *INFO [2014-04-14 18:11:02.007]
>> >> [org.apache.solr.core.CachingDirectoryFactory] return new directory for
>> >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
>> >> *INFO [2014-04-14 18:11:02.008] [org.apache.solr.handler.SnapPuller]
>> >> Starting download to
>> >> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@
>> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
>> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
>> >> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
>> >>
>> >> ****************
>> >>
>> >>
>> >> So, it looks like the number of updates is too huge for the regular
>> >> replication and then it goes into full copy of index. And since our
>> index
>> >> size is very huge (350G), this is causing the cluster to go into
>> recovery
>> >> mode forever - trying to copy that huge index.
>> >>
>> >> I also read in some thread
>> >>
>> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthatthere is a limit of 100 documents.
>> >>
>> >> I wonder if this has been updated to make that configurable since that
>> >> thread. If not, the only option I see is to do a "trickle" delete of
>> 100
>> >> documents per second or something.
>> >>
>> >> Also - the other suggestion of using "distributed=false" might not help
>> >> because the issue currently is that the replication is going to "full
>> copy".
>> >>
>> >> Any thoughts?
>> >>
>> >> Thanks
>> >> Vinay
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
>> >>
>> >>> Yes, that is our approach. We did try deleting a day's worth of data
>> at a
>> >>> time, and that resulted in OOM as well.
>> >>>
>> >>> Thanks
>> >>> Vinay
>> >>>
>> >>>
>> >>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com> wrote:
>> >>>
>> >>>> Hi;
>> >>>>
>> >>>> I mean you can divide the range (i.e. one week at each delete
>> instead of
>> >>>> one month) and try to check whether you still get an OOM or not.
>> >>>>
>> >>>> Thanks;
>> >>>> Furkan KAMACI
>> >>>>
>> >>>>
>> >>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
>> >>>>
>> >>>> > Aman,
>> >>>> > Yes - Will do!
>> >>>> >
>> >>>> > Furkan,
>> >>>> > How do you mean by 'bulk delete'?
>> >>>> >
>> >>>> > -Thanks
>> >>>> > Vinay
>> >>>> >
>> >>>> >
>> >>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com>
>> wrote:
>> >>>> >
>> >>>> > > Hi;
>> >>>> > >
>> >>>> > > Do you get any problems when you index your data? On the other
>> hand
>> >>>> > > deleting as bulks and reducing the size of documents may help you
>> >>>> not to
>> >>>> > > hit OOM.
>> >>>> > >
>> >>>> > > Thanks;
>> >>>> > > Furkan KAMACI
>> >>>> > >
>> >>>> > >
>> >>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
>> >>>> > >
>> >>>> > > > Vinay please share your experience after trying this solution.
>> >>>> > > >
>> >>>> > > >
>> >>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <
>> pothnis@gmail.com>
>> >>>> > > wrote:
>> >>>> > > >
>> >>>> > > > > The query is something like this:
>> >>>> > > > >
>> >>>> > > > >
>> >>>> > > > > *curl -H 'Content-Type: text/xml' --data
>> >>>> '<delete><query>param1:(val1
>> >>>> > > OR
>> >>>> > > > > val2) AND -param2:(val3 OR val4) AND
>> date_param:[1383955200000 TO
>> >>>> > > > > 1385164800000]</query></delete>'
>> >>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
>> >>>> > > > >
>> >>>> > > > > Trying to restrict the number of documents deleted via the
>> date
>> >>>> > > > parameter.
>> >>>> > > > >
>> >>>> > > > > Had not tried the "distrib=false" option. I could give that a
>> >>>> try.
>> >>>> > > Thanks
>> >>>> > > > > for the link! I will check on the cache sizes and autowarm
>> >>>> values.
>> >>>> > Will
>> >>>> > > > try
>> >>>> > > > > and disable the caches when I am deleting and give that a
>> try.
>> >>>> > > > >
>> >>>> > > > > Thanks Erick and Shawn for your inputs!
>> >>>> > > > >
>> >>>> > > > > -Vinay
>> >>>> > > > >
>> >>>> > > > >
>> >>>> > > > >
>> >>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org>
>> wrote:
>> >>>> > > > >
>> >>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
>> >>>> > > > > >
>> >>>> > > > > >> When we tried to delete the data through a query - say 1
>> >>>> > day/month's
>> >>>> > > > > worth
>> >>>> > > > > >> of data. But after deleting just 1 month's worth of data,
>> the
>> >>>> > master
>> >>>> > > > > node
>> >>>> > > > > >> is going out of memory - heap space.
>> >>>> > > > > >>
>> >>>> > > > > >> Wondering is there any way to incrementally delete the
>> data
>> >>>> > without
>> >>>> > > > > >> affecting the cluster adversely.
>> >>>> > > > > >>
>> >>>> > > > > >
>> >>>> > > > > > I'm curious about the actual query being used here. Can
>> you
>> >>>> share
>> >>>> > > it,
>> >>>> > > > or
>> >>>> > > > > > a redacted version of it? Perhaps there might be a clue
>> there?
>> >>>> > > > > >
>> >>>> > > > > > Is this a fully distributed delete request? One thing you
>> >>>> might
>> >>>> > try,
>> >>>> > > > > > assuming Solr even supports it, is sending the same delete
>> >>>> request
>> >>>> > > > > directly
>> >>>> > > > > > to each shard core with distrib=false.
>> >>>> > > > > >
>> >>>> > > > > > Here's a very incomplete list about how you can reduce Solr
>> >>>> heap
>> >>>> > > > > > requirements:
>> >>>> > > > > >
>> >>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
>> >>>> > > > > > Reducing_heap_requirements
>> >>>> > > > > >
>> >>>> > > > > > Thanks,
>> >>>> > > > > > Shawn
>> >>>> > > > > >
>> >>>> > > > > >
>> >>>> > > > >
>> >>>> > > >
>> >>>> > > >
>> >>>> > > >
>> >>>> > > > --
>> >>>> > > > With Regards
>> >>>> > > > Aman Tandon
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >>
>>
>
>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Thank you Erick!
Yes - I am using the expunge deletes option.
Thanks for the note on disk space for the optimize command. I should have
enough space for that. What about the heap space requirement? I hope it can
do the optimize with the memory that is allocated to it.
Thanks
Vinay
On 16 April 2014 04:52, Erick Erickson <er...@gmail.com> wrote:
> The optimize should, indeed, reduce the index size. Be aware that it
> may consume 2x the disk space. You may also try expungedeletes, see
> here: https://wiki.apache.org/solr/UpdateXmlMessages
>
> Best,
> Erick
>
> On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <po...@gmail.com> wrote:
> > Another update:
> >
> > I removed the replicas - to avoid the replication doing a full copy. I am
> > able delete sizeable chunks of data.
> > But the overall index size remains the same even after the deletes. It
> does
> > not seem to go down.
> >
> > I understand that Solr would do this in background - but I don't seem to
> > see the decrease in overall index size even after 1-2 hours.
> > I can see a bunch of ".del" files in the index directory, but the it does
> > not seem to get cleaned up. Is there anyway to monitor/follow the
> progress
> > of index compaction?
> >
> > Also, does triggering "optimize" from the admin UI help to compact the
> > index size on disk?
> >
> > Thanks
> > Vinay
> >
> >
> > On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
> >
> >> Some update:
> >>
> >> I removed the auto warm configurations for the various caches and
> reduced
> >> the cache sizes. I then issued a call to delete a day's worth of data
> (800K
> >> documents).
> >>
> >> There was no out of memory this time - but some of the nodes went into
> >> recovery mode. Was able to catch some logs this time around and this is
> >> what i see:
> >>
> >> ****************
> >> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
> >> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
> >> <http://host1:8983/solr> too many updates received since start -
> >> startingUpdates no longer overlaps with our currentUpdates*
> >> *INFO [2014-04-14 18:11:00.476]
> [org.apache.solr.cloud.RecoveryStrategy]
> >> PeerSync Recovery was not successful - trying replication.
> >> core=core1_shard1_replica2*
> >> *INFO [2014-04-14 18:11:00.476]
> [org.apache.solr.cloud.RecoveryStrategy]
> >> Starting Replication Recovery. core=core1_shard1_replica2*
> >> *INFO [2014-04-14 18:11:00.535]
> [org.apache.solr.cloud.RecoveryStrategy]
> >> Begin buffering updates. core=core1_shard1_replica2*
> >> *INFO [2014-04-14 18:11:00.536]
> [org.apache.solr.cloud.RecoveryStrategy]
> >> Attempting to replicate from
> http://host2:8983/solr/core1_shard1_replica1/
> >> <http://host2:8983/solr/core1_shard1_replica1/>.
> core=core1_shard1_replica2*
> >> *INFO [2014-04-14 18:11:00.536]
> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
> >> client,
> >>
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
> >> *INFO [2014-04-14 18:11:01.964]
> >> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
> >> client,
> >>
> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
> >> *INFO [2014-04-14 18:11:01.969] [org.apache.solr.handler.SnapPuller]
> No
> >> value set for 'pollInterval'. Timer Task not started.*
> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> >> Master's generation: 1108645*
> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> >> Slave's generation: 1108627*
> >> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> >> Starting replication process*
> >> *INFO [2014-04-14 18:11:02.007] [org.apache.solr.handler.SnapPuller]
> >> Number of files in latest index in master: 814*
> >> *INFO [2014-04-14 18:11:02.007]
> >> [org.apache.solr.core.CachingDirectoryFactory] return new directory for
> >> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
> >> *INFO [2014-04-14 18:11:02.008] [org.apache.solr.handler.SnapPuller]
> >> Starting download to
> >> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@
> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
> >> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
> >>
> >> ****************
> >>
> >>
> >> So, it looks like the number of updates is too huge for the regular
> >> replication and then it goes into full copy of index. And since our
> index
> >> size is very huge (350G), this is causing the cluster to go into
> recovery
> >> mode forever - trying to copy that huge index.
> >>
> >> I also read in some thread
> >>
> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthatthere is a limit of 100 documents.
> >>
> >> I wonder if this has been updated to make that configurable since that
> >> thread. If not, the only option I see is to do a "trickle" delete of 100
> >> documents per second or something.
> >>
> >> Also - the other suggestion of using "distributed=false" might not help
> >> because the issue currently is that the replication is going to "full
> copy".
> >>
> >> Any thoughts?
> >>
> >> Thanks
> >> Vinay
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
> >>
> >>> Yes, that is our approach. We did try deleting a day's worth of data
> at a
> >>> time, and that resulted in OOM as well.
> >>>
> >>> Thanks
> >>> Vinay
> >>>
> >>>
> >>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com> wrote:
> >>>
> >>>> Hi;
> >>>>
> >>>> I mean you can divide the range (i.e. one week at each delete instead
> of
> >>>> one month) and try to check whether you still get an OOM or not.
> >>>>
> >>>> Thanks;
> >>>> Furkan KAMACI
> >>>>
> >>>>
> >>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
> >>>>
> >>>> > Aman,
> >>>> > Yes - Will do!
> >>>> >
> >>>> > Furkan,
> >>>> > How do you mean by 'bulk delete'?
> >>>> >
> >>>> > -Thanks
> >>>> > Vinay
> >>>> >
> >>>> >
> >>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com>
> wrote:
> >>>> >
> >>>> > > Hi;
> >>>> > >
> >>>> > > Do you get any problems when you index your data? On the other
> hand
> >>>> > > deleting as bulks and reducing the size of documents may help you
> >>>> not to
> >>>> > > hit OOM.
> >>>> > >
> >>>> > > Thanks;
> >>>> > > Furkan KAMACI
> >>>> > >
> >>>> > >
> >>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
> >>>> > >
> >>>> > > > Vinay please share your experience after trying this solution.
> >>>> > > >
> >>>> > > >
> >>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <
> pothnis@gmail.com>
> >>>> > > wrote:
> >>>> > > >
> >>>> > > > > The query is something like this:
> >>>> > > > >
> >>>> > > > >
> >>>> > > > > *curl -H 'Content-Type: text/xml' --data
> >>>> '<delete><query>param1:(val1
> >>>> > > OR
> >>>> > > > > val2) AND -param2:(val3 OR val4) AND
> date_param:[1383955200000 TO
> >>>> > > > > 1385164800000]</query></delete>'
> >>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
> >>>> > > > >
> >>>> > > > > Trying to restrict the number of documents deleted via the
> date
> >>>> > > > parameter.
> >>>> > > > >
> >>>> > > > > Had not tried the "distrib=false" option. I could give that a
> >>>> try.
> >>>> > > Thanks
> >>>> > > > > for the link! I will check on the cache sizes and autowarm
> >>>> values.
> >>>> > Will
> >>>> > > > try
> >>>> > > > > and disable the caches when I am deleting and give that a try.
> >>>> > > > >
> >>>> > > > > Thanks Erick and Shawn for your inputs!
> >>>> > > > >
> >>>> > > > > -Vinay
> >>>> > > > >
> >>>> > > > >
> >>>> > > > >
> >>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org>
> wrote:
> >>>> > > > >
> >>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> >>>> > > > > >
> >>>> > > > > >> When we tried to delete the data through a query - say 1
> >>>> > day/month's
> >>>> > > > > worth
> >>>> > > > > >> of data. But after deleting just 1 month's worth of data,
> the
> >>>> > master
> >>>> > > > > node
> >>>> > > > > >> is going out of memory - heap space.
> >>>> > > > > >>
> >>>> > > > > >> Wondering is there any way to incrementally delete the data
> >>>> > without
> >>>> > > > > >> affecting the cluster adversely.
> >>>> > > > > >>
> >>>> > > > > >
> >>>> > > > > > I'm curious about the actual query being used here. Can you
> >>>> share
> >>>> > > it,
> >>>> > > > or
> >>>> > > > > > a redacted version of it? Perhaps there might be a clue
> there?
> >>>> > > > > >
> >>>> > > > > > Is this a fully distributed delete request? One thing you
> >>>> might
> >>>> > try,
> >>>> > > > > > assuming Solr even supports it, is sending the same delete
> >>>> request
> >>>> > > > > directly
> >>>> > > > > > to each shard core with distrib=false.
> >>>> > > > > >
> >>>> > > > > > Here's a very incomplete list about how you can reduce Solr
> >>>> heap
> >>>> > > > > > requirements:
> >>>> > > > > >
> >>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> >>>> > > > > > Reducing_heap_requirements
> >>>> > > > > >
> >>>> > > > > > Thanks,
> >>>> > > > > > Shawn
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > > With Regards
> >>>> > > > Aman Tandon
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
>
Re: deleting large amount data from solr cloud
Posted by Erick Erickson <er...@gmail.com>.
The optimize should, indeed, reduce the index size. Be aware that it
may consume 2x the disk space. You may also try expungedeletes, see
here: https://wiki.apache.org/solr/UpdateXmlMessages
Best,
Erick
On Wed, Apr 16, 2014 at 12:47 AM, Vinay Pothnis <po...@gmail.com> wrote:
> Another update:
>
> I removed the replicas - to avoid the replication doing a full copy. I am
> able delete sizeable chunks of data.
> But the overall index size remains the same even after the deletes. It does
> not seem to go down.
>
> I understand that Solr would do this in background - but I don't seem to
> see the decrease in overall index size even after 1-2 hours.
> I can see a bunch of ".del" files in the index directory, but the it does
> not seem to get cleaned up. Is there anyway to monitor/follow the progress
> of index compaction?
>
> Also, does triggering "optimize" from the admin UI help to compact the
> index size on disk?
>
> Thanks
> Vinay
>
>
> On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
>
>> Some update:
>>
>> I removed the auto warm configurations for the various caches and reduced
>> the cache sizes. I then issued a call to delete a day's worth of data (800K
>> documents).
>>
>> There was no out of memory this time - but some of the nodes went into
>> recovery mode. Was able to catch some logs this time around and this is
>> what i see:
>>
>> ****************
>> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
>> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
>> <http://host1:8983/solr> too many updates received since start -
>> startingUpdates no longer overlaps with our currentUpdates*
>> *INFO [2014-04-14 18:11:00.476] [org.apache.solr.cloud.RecoveryStrategy]
>> PeerSync Recovery was not successful - trying replication.
>> core=core1_shard1_replica2*
>> *INFO [2014-04-14 18:11:00.476] [org.apache.solr.cloud.RecoveryStrategy]
>> Starting Replication Recovery. core=core1_shard1_replica2*
>> *INFO [2014-04-14 18:11:00.535] [org.apache.solr.cloud.RecoveryStrategy]
>> Begin buffering updates. core=core1_shard1_replica2*
>> *INFO [2014-04-14 18:11:00.536] [org.apache.solr.cloud.RecoveryStrategy]
>> Attempting to replicate from http://host2:8983/solr/core1_shard1_replica1/
>> <http://host2:8983/solr/core1_shard1_replica1/>. core=core1_shard1_replica2*
>> *INFO [2014-04-14 18:11:00.536]
>> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
>> client,
>> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
>> *INFO [2014-04-14 18:11:01.964]
>> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
>> client,
>> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
>> *INFO [2014-04-14 18:11:01.969] [org.apache.solr.handler.SnapPuller] No
>> value set for 'pollInterval'. Timer Task not started.*
>> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
>> Master's generation: 1108645*
>> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
>> Slave's generation: 1108627*
>> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
>> Starting replication process*
>> *INFO [2014-04-14 18:11:02.007] [org.apache.solr.handler.SnapPuller]
>> Number of files in latest index in master: 814*
>> *INFO [2014-04-14 18:11:02.007]
>> [org.apache.solr.core.CachingDirectoryFactory] return new directory for
>> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
>> *INFO [2014-04-14 18:11:02.008] [org.apache.solr.handler.SnapPuller]
>> Starting download to
>> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
>> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
>> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
>>
>> ****************
>>
>>
>> So, it looks like the number of updates is too huge for the regular
>> replication and then it goes into full copy of index. And since our index
>> size is very huge (350G), this is causing the cluster to go into recovery
>> mode forever - trying to copy that huge index.
>>
>> I also read in some thread
>> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthat there is a limit of 100 documents.
>>
>> I wonder if this has been updated to make that configurable since that
>> thread. If not, the only option I see is to do a "trickle" delete of 100
>> documents per second or something.
>>
>> Also - the other suggestion of using "distributed=false" might not help
>> because the issue currently is that the replication is going to "full copy".
>>
>> Any thoughts?
>>
>> Thanks
>> Vinay
>>
>>
>>
>>
>>
>>
>>
>> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
>>
>>> Yes, that is our approach. We did try deleting a day's worth of data at a
>>> time, and that resulted in OOM as well.
>>>
>>> Thanks
>>> Vinay
>>>
>>>
>>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com> wrote:
>>>
>>>> Hi;
>>>>
>>>> I mean you can divide the range (i.e. one week at each delete instead of
>>>> one month) and try to check whether you still get an OOM or not.
>>>>
>>>> Thanks;
>>>> Furkan KAMACI
>>>>
>>>>
>>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
>>>>
>>>> > Aman,
>>>> > Yes - Will do!
>>>> >
>>>> > Furkan,
>>>> > How do you mean by 'bulk delete'?
>>>> >
>>>> > -Thanks
>>>> > Vinay
>>>> >
>>>> >
>>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com> wrote:
>>>> >
>>>> > > Hi;
>>>> > >
>>>> > > Do you get any problems when you index your data? On the other hand
>>>> > > deleting as bulks and reducing the size of documents may help you
>>>> not to
>>>> > > hit OOM.
>>>> > >
>>>> > > Thanks;
>>>> > > Furkan KAMACI
>>>> > >
>>>> > >
>>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
>>>> > >
>>>> > > > Vinay please share your experience after trying this solution.
>>>> > > >
>>>> > > >
>>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com>
>>>> > > wrote:
>>>> > > >
>>>> > > > > The query is something like this:
>>>> > > > >
>>>> > > > >
>>>> > > > > *curl -H 'Content-Type: text/xml' --data
>>>> '<delete><query>param1:(val1
>>>> > > OR
>>>> > > > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
>>>> > > > > 1385164800000]</query></delete>'
>>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
>>>> > > > >
>>>> > > > > Trying to restrict the number of documents deleted via the date
>>>> > > > parameter.
>>>> > > > >
>>>> > > > > Had not tried the "distrib=false" option. I could give that a
>>>> try.
>>>> > > Thanks
>>>> > > > > for the link! I will check on the cache sizes and autowarm
>>>> values.
>>>> > Will
>>>> > > > try
>>>> > > > > and disable the caches when I am deleting and give that a try.
>>>> > > > >
>>>> > > > > Thanks Erick and Shawn for your inputs!
>>>> > > > >
>>>> > > > > -Vinay
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
>>>> > > > >
>>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
>>>> > > > > >
>>>> > > > > >> When we tried to delete the data through a query - say 1
>>>> > day/month's
>>>> > > > > worth
>>>> > > > > >> of data. But after deleting just 1 month's worth of data, the
>>>> > master
>>>> > > > > node
>>>> > > > > >> is going out of memory - heap space.
>>>> > > > > >>
>>>> > > > > >> Wondering is there any way to incrementally delete the data
>>>> > without
>>>> > > > > >> affecting the cluster adversely.
>>>> > > > > >>
>>>> > > > > >
>>>> > > > > > I'm curious about the actual query being used here. Can you
>>>> share
>>>> > > it,
>>>> > > > or
>>>> > > > > > a redacted version of it? Perhaps there might be a clue there?
>>>> > > > > >
>>>> > > > > > Is this a fully distributed delete request? One thing you
>>>> might
>>>> > try,
>>>> > > > > > assuming Solr even supports it, is sending the same delete
>>>> request
>>>> > > > > directly
>>>> > > > > > to each shard core with distrib=false.
>>>> > > > > >
>>>> > > > > > Here's a very incomplete list about how you can reduce Solr
>>>> heap
>>>> > > > > > requirements:
>>>> > > > > >
>>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
>>>> > > > > > Reducing_heap_requirements
>>>> > > > > >
>>>> > > > > > Thanks,
>>>> > > > > > Shawn
>>>> > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > With Regards
>>>> > > > Aman Tandon
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Another update:
I removed the replicas - to avoid the replication doing a full copy. I am
able delete sizeable chunks of data.
But the overall index size remains the same even after the deletes. It does
not seem to go down.
I understand that Solr would do this in background - but I don't seem to
see the decrease in overall index size even after 1-2 hours.
I can see a bunch of ".del" files in the index directory, but the it does
not seem to get cleaned up. Is there anyway to monitor/follow the progress
of index compaction?
Also, does triggering "optimize" from the admin UI help to compact the
index size on disk?
Thanks
Vinay
On 14 April 2014 12:19, Vinay Pothnis <po...@gmail.com> wrote:
> Some update:
>
> I removed the auto warm configurations for the various caches and reduced
> the cache sizes. I then issued a call to delete a day's worth of data (800K
> documents).
>
> There was no out of memory this time - but some of the nodes went into
> recovery mode. Was able to catch some logs this time around and this is
> what i see:
>
> ****************
> *WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
> PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
> <http://host1:8983/solr> too many updates received since start -
> startingUpdates no longer overlaps with our currentUpdates*
> *INFO [2014-04-14 18:11:00.476] [org.apache.solr.cloud.RecoveryStrategy]
> PeerSync Recovery was not successful - trying replication.
> core=core1_shard1_replica2*
> *INFO [2014-04-14 18:11:00.476] [org.apache.solr.cloud.RecoveryStrategy]
> Starting Replication Recovery. core=core1_shard1_replica2*
> *INFO [2014-04-14 18:11:00.535] [org.apache.solr.cloud.RecoveryStrategy]
> Begin buffering updates. core=core1_shard1_replica2*
> *INFO [2014-04-14 18:11:00.536] [org.apache.solr.cloud.RecoveryStrategy]
> Attempting to replicate from http://host2:8983/solr/core1_shard1_replica1/
> <http://host2:8983/solr/core1_shard1_replica1/>. core=core1_shard1_replica2*
> *INFO [2014-04-14 18:11:00.536]
> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
> *INFO [2014-04-14 18:11:01.964]
> [org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
> client,
> config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
> *INFO [2014-04-14 18:11:01.969] [org.apache.solr.handler.SnapPuller] No
> value set for 'pollInterval'. Timer Task not started.*
> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> Master's generation: 1108645*
> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> Slave's generation: 1108627*
> *INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
> Starting replication process*
> *INFO [2014-04-14 18:11:02.007] [org.apache.solr.handler.SnapPuller]
> Number of files in latest index in master: 814*
> *INFO [2014-04-14 18:11:02.007]
> [org.apache.solr.core.CachingDirectoryFactory] return new directory for
> /opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
> *INFO [2014-04-14 18:11:02.008] [org.apache.solr.handler.SnapPuller]
> Starting download to
> NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
> maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
>
> ****************
>
>
> So, it looks like the number of updates is too huge for the regular
> replication and then it goes into full copy of index. And since our index
> size is very huge (350G), this is causing the cluster to go into recovery
> mode forever - trying to copy that huge index.
>
> I also read in some thread
> http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthat there is a limit of 100 documents.
>
> I wonder if this has been updated to make that configurable since that
> thread. If not, the only option I see is to do a "trickle" delete of 100
> documents per second or something.
>
> Also - the other suggestion of using "distributed=false" might not help
> because the issue currently is that the replication is going to "full copy".
>
> Any thoughts?
>
> Thanks
> Vinay
>
>
>
>
>
>
>
> On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
>
>> Yes, that is our approach. We did try deleting a day's worth of data at a
>> time, and that resulted in OOM as well.
>>
>> Thanks
>> Vinay
>>
>>
>> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com> wrote:
>>
>>> Hi;
>>>
>>> I mean you can divide the range (i.e. one week at each delete instead of
>>> one month) and try to check whether you still get an OOM or not.
>>>
>>> Thanks;
>>> Furkan KAMACI
>>>
>>>
>>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
>>>
>>> > Aman,
>>> > Yes - Will do!
>>> >
>>> > Furkan,
>>> > How do you mean by 'bulk delete'?
>>> >
>>> > -Thanks
>>> > Vinay
>>> >
>>> >
>>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com> wrote:
>>> >
>>> > > Hi;
>>> > >
>>> > > Do you get any problems when you index your data? On the other hand
>>> > > deleting as bulks and reducing the size of documents may help you
>>> not to
>>> > > hit OOM.
>>> > >
>>> > > Thanks;
>>> > > Furkan KAMACI
>>> > >
>>> > >
>>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
>>> > >
>>> > > > Vinay please share your experience after trying this solution.
>>> > > >
>>> > > >
>>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com>
>>> > > wrote:
>>> > > >
>>> > > > > The query is something like this:
>>> > > > >
>>> > > > >
>>> > > > > *curl -H 'Content-Type: text/xml' --data
>>> '<delete><query>param1:(val1
>>> > > OR
>>> > > > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
>>> > > > > 1385164800000]</query></delete>'
>>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
>>> > > > >
>>> > > > > Trying to restrict the number of documents deleted via the date
>>> > > > parameter.
>>> > > > >
>>> > > > > Had not tried the "distrib=false" option. I could give that a
>>> try.
>>> > > Thanks
>>> > > > > for the link! I will check on the cache sizes and autowarm
>>> values.
>>> > Will
>>> > > > try
>>> > > > > and disable the caches when I am deleting and give that a try.
>>> > > > >
>>> > > > > Thanks Erick and Shawn for your inputs!
>>> > > > >
>>> > > > > -Vinay
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
>>> > > > >
>>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
>>> > > > > >
>>> > > > > >> When we tried to delete the data through a query - say 1
>>> > day/month's
>>> > > > > worth
>>> > > > > >> of data. But after deleting just 1 month's worth of data, the
>>> > master
>>> > > > > node
>>> > > > > >> is going out of memory - heap space.
>>> > > > > >>
>>> > > > > >> Wondering is there any way to incrementally delete the data
>>> > without
>>> > > > > >> affecting the cluster adversely.
>>> > > > > >>
>>> > > > > >
>>> > > > > > I'm curious about the actual query being used here. Can you
>>> share
>>> > > it,
>>> > > > or
>>> > > > > > a redacted version of it? Perhaps there might be a clue there?
>>> > > > > >
>>> > > > > > Is this a fully distributed delete request? One thing you
>>> might
>>> > try,
>>> > > > > > assuming Solr even supports it, is sending the same delete
>>> request
>>> > > > > directly
>>> > > > > > to each shard core with distrib=false.
>>> > > > > >
>>> > > > > > Here's a very incomplete list about how you can reduce Solr
>>> heap
>>> > > > > > requirements:
>>> > > > > >
>>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
>>> > > > > > Reducing_heap_requirements
>>> > > > > >
>>> > > > > > Thanks,
>>> > > > > > Shawn
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > With Regards
>>> > > > Aman Tandon
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Some update:
I removed the auto warm configurations for the various caches and reduced
the cache sizes. I then issued a call to delete a day's worth of data (800K
documents).
There was no out of memory this time - but some of the nodes went into
recovery mode. Was able to catch some logs this time around and this is
what i see:
****************
*WARN [2014-04-14 18:11:00.381] [org.apache.solr.update.PeerSync]
PeerSync: core=core1_shard1_replica2 url=http://host1:8983/solr
<http://host1:8983/solr> too many updates received since start -
startingUpdates no longer overlaps with our currentUpdates*
*INFO [2014-04-14 18:11:00.476] [org.apache.solr.cloud.RecoveryStrategy]
PeerSync Recovery was not successful - trying replication.
core=core1_shard1_replica2*
*INFO [2014-04-14 18:11:00.476] [org.apache.solr.cloud.RecoveryStrategy]
Starting Replication Recovery. core=core1_shard1_replica2*
*INFO [2014-04-14 18:11:00.535] [org.apache.solr.cloud.RecoveryStrategy]
Begin buffering updates. core=core1_shard1_replica2*
*INFO [2014-04-14 18:11:00.536] [org.apache.solr.cloud.RecoveryStrategy]
Attempting to replicate from http://host2:8983/solr/core1_shard1_replica1/
<http://host2:8983/solr/core1_shard1_replica1/>. core=core1_shard1_replica2*
*INFO [2014-04-14 18:11:00.536]
[org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false*
*INFO [2014-04-14 18:11:01.964]
[org.apache.solr.client.solrj.impl.HttpClientUtil] Creating new http
client,
config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000*
*INFO [2014-04-14 18:11:01.969] [org.apache.solr.handler.SnapPuller] No
value set for 'pollInterval'. Timer Task not started.*
*INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
Master's generation: 1108645*
*INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
Slave's generation: 1108627*
*INFO [2014-04-14 18:11:01.973] [org.apache.solr.handler.SnapPuller]
Starting replication process*
*INFO [2014-04-14 18:11:02.007] [org.apache.solr.handler.SnapPuller]
Number of files in latest index in master: 814*
*INFO [2014-04-14 18:11:02.007]
[org.apache.solr.core.CachingDirectoryFactory] return new directory for
/opt/data/solr/core1_shard1_replica2/data/index.20140414181102007*
*INFO [2014-04-14 18:11:02.008] [org.apache.solr.handler.SnapPuller]
Starting download to
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/opt/data/solr/core1_shard1_replica2/data/index.20140414181102007
lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f6570fe;
maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=true*
****************
So, it looks like the number of updates is too huge for the regular
replication and then it goes into full copy of index. And since our index
size is very huge (350G), this is causing the cluster to go into recovery
mode forever - trying to copy that huge index.
I also read in some thread
http://lucene.472066.n3.nabble.com/Recovery-too-many-updates-received-since-start-td3935281.htmlthat
there is a limit of 100 documents.
I wonder if this has been updated to make that configurable since that
thread. If not, the only option I see is to do a "trickle" delete of 100
documents per second or something.
Also - the other suggestion of using "distributed=false" might not help
because the issue currently is that the replication is going to "full copy".
Any thoughts?
Thanks
Vinay
On 14 April 2014 07:54, Vinay Pothnis <po...@gmail.com> wrote:
> Yes, that is our approach. We did try deleting a day's worth of data at a
> time, and that resulted in OOM as well.
>
> Thanks
> Vinay
>
>
> On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com> wrote:
>
>> Hi;
>>
>> I mean you can divide the range (i.e. one week at each delete instead of
>> one month) and try to check whether you still get an OOM or not.
>>
>> Thanks;
>> Furkan KAMACI
>>
>>
>> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
>>
>> > Aman,
>> > Yes - Will do!
>> >
>> > Furkan,
>> > How do you mean by 'bulk delete'?
>> >
>> > -Thanks
>> > Vinay
>> >
>> >
>> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com> wrote:
>> >
>> > > Hi;
>> > >
>> > > Do you get any problems when you index your data? On the other hand
>> > > deleting as bulks and reducing the size of documents may help you not
>> to
>> > > hit OOM.
>> > >
>> > > Thanks;
>> > > Furkan KAMACI
>> > >
>> > >
>> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
>> > >
>> > > > Vinay please share your experience after trying this solution.
>> > > >
>> > > >
>> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > The query is something like this:
>> > > > >
>> > > > >
>> > > > > *curl -H 'Content-Type: text/xml' --data
>> '<delete><query>param1:(val1
>> > > OR
>> > > > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
>> > > > > 1385164800000]</query></delete>'
>> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
>> > > > >
>> > > > > Trying to restrict the number of documents deleted via the date
>> > > > parameter.
>> > > > >
>> > > > > Had not tried the "distrib=false" option. I could give that a try.
>> > > Thanks
>> > > > > for the link! I will check on the cache sizes and autowarm values.
>> > Will
>> > > > try
>> > > > > and disable the caches when I am deleting and give that a try.
>> > > > >
>> > > > > Thanks Erick and Shawn for your inputs!
>> > > > >
>> > > > > -Vinay
>> > > > >
>> > > > >
>> > > > >
>> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
>> > > > >
>> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
>> > > > > >
>> > > > > >> When we tried to delete the data through a query - say 1
>> > day/month's
>> > > > > worth
>> > > > > >> of data. But after deleting just 1 month's worth of data, the
>> > master
>> > > > > node
>> > > > > >> is going out of memory - heap space.
>> > > > > >>
>> > > > > >> Wondering is there any way to incrementally delete the data
>> > without
>> > > > > >> affecting the cluster adversely.
>> > > > > >>
>> > > > > >
>> > > > > > I'm curious about the actual query being used here. Can you
>> share
>> > > it,
>> > > > or
>> > > > > > a redacted version of it? Perhaps there might be a clue there?
>> > > > > >
>> > > > > > Is this a fully distributed delete request? One thing you might
>> > try,
>> > > > > > assuming Solr even supports it, is sending the same delete
>> request
>> > > > > directly
>> > > > > > to each shard core with distrib=false.
>> > > > > >
>> > > > > > Here's a very incomplete list about how you can reduce Solr heap
>> > > > > > requirements:
>> > > > > >
>> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
>> > > > > > Reducing_heap_requirements
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Shawn
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > With Regards
>> > > > Aman Tandon
>> > > >
>> > >
>> >
>>
>
>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Yes, that is our approach. We did try deleting a day's worth of data at a
time, and that resulted in OOM as well.
Thanks
Vinay
On 14 April 2014 00:27, Furkan KAMACI <fu...@gmail.com> wrote:
> Hi;
>
> I mean you can divide the range (i.e. one week at each delete instead of
> one month) and try to check whether you still get an OOM or not.
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
>
> > Aman,
> > Yes - Will do!
> >
> > Furkan,
> > How do you mean by 'bulk delete'?
> >
> > -Thanks
> > Vinay
> >
> >
> > On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com> wrote:
> >
> > > Hi;
> > >
> > > Do you get any problems when you index your data? On the other hand
> > > deleting as bulks and reducing the size of documents may help you not
> to
> > > hit OOM.
> > >
> > > Thanks;
> > > Furkan KAMACI
> > >
> > >
> > > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
> > >
> > > > Vinay please share your experience after trying this solution.
> > > >
> > > >
> > > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com>
> > > wrote:
> > > >
> > > > > The query is something like this:
> > > > >
> > > > >
> > > > > *curl -H 'Content-Type: text/xml' --data
> '<delete><query>param1:(val1
> > > OR
> > > > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
> > > > > 1385164800000]</query></delete>'
> > > > > 'http://host:port/solr/coll-name1/update?commit=true'*
> > > > >
> > > > > Trying to restrict the number of documents deleted via the date
> > > > parameter.
> > > > >
> > > > > Had not tried the "distrib=false" option. I could give that a try.
> > > Thanks
> > > > > for the link! I will check on the cache sizes and autowarm values.
> > Will
> > > > try
> > > > > and disable the caches when I am deleting and give that a try.
> > > > >
> > > > > Thanks Erick and Shawn for your inputs!
> > > > >
> > > > > -Vinay
> > > > >
> > > > >
> > > > >
> > > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
> > > > >
> > > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> > > > > >
> > > > > >> When we tried to delete the data through a query - say 1
> > day/month's
> > > > > worth
> > > > > >> of data. But after deleting just 1 month's worth of data, the
> > master
> > > > > node
> > > > > >> is going out of memory - heap space.
> > > > > >>
> > > > > >> Wondering is there any way to incrementally delete the data
> > without
> > > > > >> affecting the cluster adversely.
> > > > > >>
> > > > > >
> > > > > > I'm curious about the actual query being used here. Can you
> share
> > > it,
> > > > or
> > > > > > a redacted version of it? Perhaps there might be a clue there?
> > > > > >
> > > > > > Is this a fully distributed delete request? One thing you might
> > try,
> > > > > > assuming Solr even supports it, is sending the same delete
> request
> > > > > directly
> > > > > > to each shard core with distrib=false.
> > > > > >
> > > > > > Here's a very incomplete list about how you can reduce Solr heap
> > > > > > requirements:
> > > > > >
> > > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > > > > > Reducing_heap_requirements
> > > > > >
> > > > > > Thanks,
> > > > > > Shawn
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > With Regards
> > > > Aman Tandon
> > > >
> > >
> >
>
Re: deleting large amount data from solr cloud
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;
I mean you can divide the range (i.e. one week at each delete instead of
one month) and try to check whether you still get an OOM or not.
Thanks;
Furkan KAMACI
2014-04-14 7:09 GMT+03:00 Vinay Pothnis <po...@gmail.com>:
> Aman,
> Yes - Will do!
>
> Furkan,
> How do you mean by 'bulk delete'?
>
> -Thanks
> Vinay
>
>
> On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com> wrote:
>
> > Hi;
> >
> > Do you get any problems when you index your data? On the other hand
> > deleting as bulks and reducing the size of documents may help you not to
> > hit OOM.
> >
> > Thanks;
> > Furkan KAMACI
> >
> >
> > 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
> >
> > > Vinay please share your experience after trying this solution.
> > >
> > >
> > > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com>
> > wrote:
> > >
> > > > The query is something like this:
> > > >
> > > >
> > > > *curl -H 'Content-Type: text/xml' --data '<delete><query>param1:(val1
> > OR
> > > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
> > > > 1385164800000]</query></delete>'
> > > > 'http://host:port/solr/coll-name1/update?commit=true'*
> > > >
> > > > Trying to restrict the number of documents deleted via the date
> > > parameter.
> > > >
> > > > Had not tried the "distrib=false" option. I could give that a try.
> > Thanks
> > > > for the link! I will check on the cache sizes and autowarm values.
> Will
> > > try
> > > > and disable the caches when I am deleting and give that a try.
> > > >
> > > > Thanks Erick and Shawn for your inputs!
> > > >
> > > > -Vinay
> > > >
> > > >
> > > >
> > > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
> > > >
> > > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> > > > >
> > > > >> When we tried to delete the data through a query - say 1
> day/month's
> > > > worth
> > > > >> of data. But after deleting just 1 month's worth of data, the
> master
> > > > node
> > > > >> is going out of memory - heap space.
> > > > >>
> > > > >> Wondering is there any way to incrementally delete the data
> without
> > > > >> affecting the cluster adversely.
> > > > >>
> > > > >
> > > > > I'm curious about the actual query being used here. Can you share
> > it,
> > > or
> > > > > a redacted version of it? Perhaps there might be a clue there?
> > > > >
> > > > > Is this a fully distributed delete request? One thing you might
> try,
> > > > > assuming Solr even supports it, is sending the same delete request
> > > > directly
> > > > > to each shard core with distrib=false.
> > > > >
> > > > > Here's a very incomplete list about how you can reduce Solr heap
> > > > > requirements:
> > > > >
> > > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > > > > Reducing_heap_requirements
> > > > >
> > > > > Thanks,
> > > > > Shawn
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > With Regards
> > > Aman Tandon
> > >
> >
>
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
Aman,
Yes - Will do!
Furkan,
How do you mean by 'bulk delete'?
-Thanks
Vinay
On 12 April 2014 14:49, Furkan KAMACI <fu...@gmail.com> wrote:
> Hi;
>
> Do you get any problems when you index your data? On the other hand
> deleting as bulks and reducing the size of documents may help you not to
> hit OOM.
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
>
> > Vinay please share your experience after trying this solution.
> >
> >
> > On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com>
> wrote:
> >
> > > The query is something like this:
> > >
> > >
> > > *curl -H 'Content-Type: text/xml' --data '<delete><query>param1:(val1
> OR
> > > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
> > > 1385164800000]</query></delete>'
> > > 'http://host:port/solr/coll-name1/update?commit=true'*
> > >
> > > Trying to restrict the number of documents deleted via the date
> > parameter.
> > >
> > > Had not tried the "distrib=false" option. I could give that a try.
> Thanks
> > > for the link! I will check on the cache sizes and autowarm values. Will
> > try
> > > and disable the caches when I am deleting and give that a try.
> > >
> > > Thanks Erick and Shawn for your inputs!
> > >
> > > -Vinay
> > >
> > >
> > >
> > > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
> > >
> > > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> > > >
> > > >> When we tried to delete the data through a query - say 1 day/month's
> > > worth
> > > >> of data. But after deleting just 1 month's worth of data, the master
> > > node
> > > >> is going out of memory - heap space.
> > > >>
> > > >> Wondering is there any way to incrementally delete the data without
> > > >> affecting the cluster adversely.
> > > >>
> > > >
> > > > I'm curious about the actual query being used here. Can you share
> it,
> > or
> > > > a redacted version of it? Perhaps there might be a clue there?
> > > >
> > > > Is this a fully distributed delete request? One thing you might try,
> > > > assuming Solr even supports it, is sending the same delete request
> > > directly
> > > > to each shard core with distrib=false.
> > > >
> > > > Here's a very incomplete list about how you can reduce Solr heap
> > > > requirements:
> > > >
> > > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > > > Reducing_heap_requirements
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > With Regards
> > Aman Tandon
> >
>
Re: deleting large amount data from solr cloud
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;
Do you get any problems when you index your data? On the other hand
deleting as bulks and reducing the size of documents may help you not to
hit OOM.
Thanks;
Furkan KAMACI
2014-04-12 8:22 GMT+03:00 Aman Tandon <am...@gmail.com>:
> Vinay please share your experience after trying this solution.
>
>
> On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com> wrote:
>
> > The query is something like this:
> >
> >
> > *curl -H 'Content-Type: text/xml' --data '<delete><query>param1:(val1 OR
> > val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
> > 1385164800000]</query></delete>'
> > 'http://host:port/solr/coll-name1/update?commit=true'*
> >
> > Trying to restrict the number of documents deleted via the date
> parameter.
> >
> > Had not tried the "distrib=false" option. I could give that a try. Thanks
> > for the link! I will check on the cache sizes and autowarm values. Will
> try
> > and disable the caches when I am deleting and give that a try.
> >
> > Thanks Erick and Shawn for your inputs!
> >
> > -Vinay
> >
> >
> >
> > On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
> >
> > > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> > >
> > >> When we tried to delete the data through a query - say 1 day/month's
> > worth
> > >> of data. But after deleting just 1 month's worth of data, the master
> > node
> > >> is going out of memory - heap space.
> > >>
> > >> Wondering is there any way to incrementally delete the data without
> > >> affecting the cluster adversely.
> > >>
> > >
> > > I'm curious about the actual query being used here. Can you share it,
> or
> > > a redacted version of it? Perhaps there might be a clue there?
> > >
> > > Is this a fully distributed delete request? One thing you might try,
> > > assuming Solr even supports it, is sending the same delete request
> > directly
> > > to each shard core with distrib=false.
> > >
> > > Here's a very incomplete list about how you can reduce Solr heap
> > > requirements:
> > >
> > > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > > Reducing_heap_requirements
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
>
>
> --
> With Regards
> Aman Tandon
>
Re: deleting large amount data from solr cloud
Posted by Aman Tandon <am...@gmail.com>.
Vinay please share your experience after trying this solution.
On Sat, Apr 12, 2014 at 4:12 AM, Vinay Pothnis <po...@gmail.com> wrote:
> The query is something like this:
>
>
> *curl -H 'Content-Type: text/xml' --data '<delete><query>param1:(val1 OR
> val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
> 1385164800000]</query></delete>'
> 'http://host:port/solr/coll-name1/update?commit=true'*
>
> Trying to restrict the number of documents deleted via the date parameter.
>
> Had not tried the "distrib=false" option. I could give that a try. Thanks
> for the link! I will check on the cache sizes and autowarm values. Will try
> and disable the caches when I am deleting and give that a try.
>
> Thanks Erick and Shawn for your inputs!
>
> -Vinay
>
>
>
> On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
>
> > On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> >
> >> When we tried to delete the data through a query - say 1 day/month's
> worth
> >> of data. But after deleting just 1 month's worth of data, the master
> node
> >> is going out of memory - heap space.
> >>
> >> Wondering is there any way to incrementally delete the data without
> >> affecting the cluster adversely.
> >>
> >
> > I'm curious about the actual query being used here. Can you share it, or
> > a redacted version of it? Perhaps there might be a clue there?
> >
> > Is this a fully distributed delete request? One thing you might try,
> > assuming Solr even supports it, is sending the same delete request
> directly
> > to each shard core with distrib=false.
> >
> > Here's a very incomplete list about how you can reduce Solr heap
> > requirements:
> >
> > http://wiki.apache.org/solr/SolrPerformanceProblems#
> > Reducing_heap_requirements
> >
> > Thanks,
> > Shawn
> >
> >
>
--
With Regards
Aman Tandon
Re: deleting large amount data from solr cloud
Posted by Vinay Pothnis <po...@gmail.com>.
The query is something like this:
*curl -H 'Content-Type: text/xml' --data '<delete><query>param1:(val1 OR
val2) AND -param2:(val3 OR val4) AND date_param:[1383955200000 TO
1385164800000]</query></delete>'
'http://host:port/solr/coll-name1/update?commit=true'*
Trying to restrict the number of documents deleted via the date parameter.
Had not tried the "distrib=false" option. I could give that a try. Thanks
for the link! I will check on the cache sizes and autowarm values. Will try
and disable the caches when I am deleting and give that a try.
Thanks Erick and Shawn for your inputs!
-Vinay
On 11 April 2014 15:28, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
>
>> When we tried to delete the data through a query - say 1 day/month's worth
>> of data. But after deleting just 1 month's worth of data, the master node
>> is going out of memory - heap space.
>>
>> Wondering is there any way to incrementally delete the data without
>> affecting the cluster adversely.
>>
>
> I'm curious about the actual query being used here. Can you share it, or
> a redacted version of it? Perhaps there might be a clue there?
>
> Is this a fully distributed delete request? One thing you might try,
> assuming Solr even supports it, is sending the same delete request directly
> to each shard core with distrib=false.
>
> Here's a very incomplete list about how you can reduce Solr heap
> requirements:
>
> http://wiki.apache.org/solr/SolrPerformanceProblems#
> Reducing_heap_requirements
>
> Thanks,
> Shawn
>
>
Re: deleting large amount data from solr cloud
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/10/2014 7:25 PM, Vinay Pothnis wrote:
> When we tried to delete the data through a query - say 1 day/month's worth
> of data. But after deleting just 1 month's worth of data, the master node
> is going out of memory - heap space.
>
> Wondering is there any way to incrementally delete the data without
> affecting the cluster adversely.
I'm curious about the actual query being used here. Can you share it,
or a redacted version of it? Perhaps there might be a clue there?
Is this a fully distributed delete request? One thing you might try,
assuming Solr even supports it, is sending the same delete request
directly to each shard core with distrib=false.
Here's a very incomplete list about how you can reduce Solr heap
requirements:
http://wiki.apache.org/solr/SolrPerformanceProblems#Reducing_heap_requirements
Thanks,
Shawn