You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Midas A <te...@gmail.com> on 2015/06/10 10:15:53 UTC
Indexing issue - index get deleted
We are running full import and delta import crons .
Fulll index once a day
delta index : every 10 mins
last night my index automatically deleted(numdocs=0).
attaching logs for review .
please suggest to resolve the issue.
Re: Indexing issue - index get deleted
Posted by Alessandro Benedetti <be...@gmail.com>.
Let me answer in line, to get more info :
2015-06-10 10:59 GMT+01:00 Midas A <te...@gmail.com>:
> Hi Alessandro,
>
> Please find the answers inline and help me out to figure out this problem.
>
> 1) Solr version : *4.2.1*
> 2) Solr architecture :* Master -slave/ Replication with requestHandler*
>
>
Where happened the issue ?
Have you read this :
The SQL Entity Processor
The SqlEntityProcessor is the default processor. The associated data source
<https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler#UploadingStructuredDataStoreDatawiththeDataImportHandler-JdbcDataSource>
should
be a JDBC URL.
The entity attributes specific to this processor are shown in the table
below.
Attribute
Use
query
Required. The SQL query used to select rows.
deltaQuery
SQL query used if the operation is delta-import. This query selects the
primary keys of the rows which will be parts of the delta-update. The pks
will be available to the deltaImportQuery through the variable
${dataimporter.delta.<column-name>}.
parentDeltaQuery
SQL query used if the operation is delta-import.
deletedPkQuery
SQL query used if the operation is delta-import.
deltaImportQuery
SQL query used if the operation is delta-import. If this is not present,
DIH tries to construct the import query by(after identifying the delta)
modifying the 'query' (this is error prone). There is a namespace
${dataimporter.delta.<column-name>} which can be used in this query. For
example, select * from tbl where id=${dataimporter.delta.id}.
It is from Solr official wiki.
You should be sure you adhere to the proper configurations.
> 3) Kind of data source indexed : *Mysql *
>
what about your delta query ? that one is the responsible for the delta
indexing
> 4) What happened to the datasource ? any change in there ? : *No change *
>
Nothing relevant happened there ? any deletion or weird update to the
database ?
> 5) Was the index actually deleted ? All docs deleted ? Index file segments
> deleted ? Index corrupted ? : *all docs deleted , segment files are there.
> index file is also there .*
>
So a deletion + commit happened, but still no merge purging the index
deleted content ?
> 6) What about system resources ?
> * JVM: 30 GB*
> * RAM: 48 GB*
>
> *CPU : 8 core*
>
eheheh not interested in your current resources, I have no indication of
the size of your data, My question was more related to check if the system
was healthy from the system resource point of view.
Cheers
>
> On Wed, Jun 10, 2015 at 2:13 PM, Alessandro Benedetti <
> benedetti.alex85@gmail.com> wrote:
>
> > Let me try to help you, first of all I would like to encourage people to
> > post more information about their scenario than "This is my log, index
> > deleted, help me" :)
> >
> > This kind of Info can be really useful :
> >
> > 1) Solr version
> > 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual
> > Sharding ? Manual Replication ? where the problem happened ? )
> > 3) Kind of data source indexed
> > 4) What happened to the datasource ? any change in there ?
> > 5) Was the index actually deleted ? All docs deleted ? Index file
> segments
> > deleted ? Index corrupted ?
> > 6) What about system resources ?
> >
> > These questions are only few example one that everyone should always post
> > along their mysterious problem !
> >
> > Hope this helps,
> >
> > Cheers
> >
> >
> > 2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
> >
> > >
> > > We are running full import and delta import crons .
> > >
> > > Fulll index once a day
> > >
> > > delta index : every 10 mins
> > >
> > >
> > > last night my index automatically deleted(numdocs=0).
> > >
> > > attaching logs for review .
> > >
> > > please suggest to resolve the issue.
> > >
> > >
> >
> >
> > --
> > --------------------------
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
> >
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: Indexing issue - index get deleted
Posted by Alessandro Benedetti <be...@gmail.com>.
Hi Chris,
Amazing Analysis !
I did actually not investigated the log, because I was first trying to get
more information from the user.
"We are running full import and delta import crons .
Fulll index once a day
delta index : every 10 mins
last night my index automatically deleted(numdocs=0).
attaching logs for review ."
Reading better the user initial mail , he does a full import as well ( and
at this point, cleaning the Index) .
Not sure is there any practical reason to do that, the user will clarify
that to us.
So after the clean happened, something prevented the full import to
proceed, and we had the weird behaviour monitored in the logs.
Really curious of understanding this better :)
2015-06-11 1:36 GMT+01:00 Chris Hostetter <ho...@fucit.org>:
>
> : The guys was using delta import anyway, so maybe the problem is
> : different and not related to the clean.
>
> that's not what the logs say.
>
> Here's what i see...
>
> Log begins with server startup @ "Jun 10, 2015 11:14:56 AM"
>
> The DeletionPolicy for the "shopclue_prod" core is initialized at "Jun
> 10, 2015 11:15:04 AM" and we see a few interesting things here we note
> for the future as we keep reading...
>
> 1) There is currently "commits:num=1" commits on disk
> 2) the current index dir in use is "index.20150311161021822"
> 3) the current segment & generation are "segFN=segments_1a,generation=46"
>
> Immediately after this, we see some searcher warming using a searcher with
> this same segments file, and then this searcher is registered ("Jun 10,
> 2015 11:15:05 AM") and the core is registered.
>
> Next we see some replication polling, and we see what look like some
> simple monitoring requests for "q=*" which return hits=85898 being
> repeated over and over.
>
> At "Jun 10, 2015 11:16:30 AM" we see some requests for /dataimport that
> look like they are coming from the UI. and then at "Jun 10, 2015 11:17:01
> AM" we see a request for a full import started.
>
> We have no idea what the data import configuration file looks like, so we
> have no idea if clean=false is being used or not. it's certianly not
> specified in the URL.
>
> We see some more monitoring URLs returning hits=85898 and some more
> /repliation status calls, and then @ "Jun 10, 2015 11:18:02 AM" we see the
> first commit executed since hte server started up.
>
> there's no indication that this commit came from an external request (eg
> "/update") so probably was made by some internal request. One
> possiblility is that it came from DIH finishing -- but i doubt it, i'm
> fairly sure that would have involved more logging then this. A more
> probably scenerio is that it came from an autoCommit setting -- the fact
> that it is almost exactly 60 seconds after DIH started -- and almost
> exactly 60 seconds after DIH may have done a deleteAll query due to
> clean=true -- makes it seem very likely that this was a 1 minute
> autoCommit)
>
> (but since we don't have either hte data import config, or the
> solrconfig.xml, we have no way of knowing -- it's all just guess work.)
>
> Very importantly, note that this commit is not opening a new searcher...
>
> Jun 10, 2015 11:18:02 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start
> commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> Here are some other interesting things to note from the logging
> that comes from the DeletionPolicy when this commit happens...
>
> 1) it now notes that there are commits:num=2 on disk
> 2) the current index dir hasn't changed (index.20150311161021822) so
> some weird replication command didn't swap the world out from under us
> 3) the newest segment/generation are "segFN=segments_1b,generation=47"
> 4) the newest commit has no other files in it besides the segments file.
>
> this means, with out a doubt, there are no documents in this commits view
> of the index. they have all been deleted by something.
>
>
> At this point the *old* searcher (for commit generation 46) is still in
> use however -- nothing has done an openSearcher=true.
>
> we see more /dataimport status requests, and other requests that appear to
> come from the Solr UI, and more monitoring queries that still return
> hits=85898 because the same searcher is in use.
>
> At "Jun 10, 2015 11:27:04 AM" we see another commit happen -- again, no
> indication that this came from an outside /update request, so it might be
> from DIH, or it might be from an autoCommit setting. the fact that it is
> nearly exactly 10 minutes after DIH started (and probably did a clean=true
> deleteAll query) makes it seem extremely likely this is an autoSoftCommit
> setting kicking in.
>
> Very importantly, note that this softCommit *does* open a new searcher...
>
> Jun 10, 2015 11:27:04 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start
>
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
>
>
> In less then a second, this new searcher is warmed up and the next time we
> see a q=* monitoring query get logged, it returns hits=0.
>
> Note that at no point in the logs, after the DataImporter is started, do
> we see it log anything other then that it has initiated the request to
> MySQL -- we do see some logs starting ~ "Jun 10, 2015 11:41:19 AM"
> indicating that someone was using the Web UI to look at the dataimport
> handler's status report. it would be really nice to know what that person
> saw at that point -- because my guess is DIH was still running and was
> staled waiting for MySql, and hadn't even started adding docs to Solr (if
> it had, i'm certian there would have been some log of it).
>
> So instead, the combination of a (probable) DIH clean=true option and a
> (near certainty) autoCommit=60sec and autoSoftCommit=10min ment that a new
> commit was created after the clean, and that commit was opened as a new
> searcher before the DIH ever finished.
>
> Or at least ... that's my best guess based solely on the Solr logs, w/o
> any concrete info on what the configs look like, or what the User observed
> at the time when looking at the DIH status.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: Indexing issue - index get deleted
Posted by Midas A <te...@gmail.com>.
Thanks . for replying ..
please find the data-config
On Thu, Jun 11, 2015 at 6:06 AM, Chris Hostetter <ho...@fucit.org>
wrote:
>
> : The guys was using delta import anyway, so maybe the problem is
> : different and not related to the clean.
>
> that's not what the logs say.
>
> Here's what i see...
>
> Log begins with server startup @ "Jun 10, 2015 11:14:56 AM"
>
> The DeletionPolicy for the "shopclue_prod" core is initialized at "Jun
> 10, 2015 11:15:04 AM" and we see a few interesting things here we note
> for the future as we keep reading...
>
> 1) There is currently "commits:num=1" commits on disk
> 2) the current index dir in use is "index.20150311161021822"
> 3) the current segment & generation are "segFN=segments_1a,generation=46"
>
> Immediately after this, we see some searcher warming using a searcher with
> this same segments file, and then this searcher is registered ("Jun 10,
> 2015 11:15:05 AM") and the core is registered.
>
> Next we see some replication polling, and we see what look like some
> simple monitoring requests for "q=*" which return hits=85898 being
> repeated over and over.
>
> At "Jun 10, 2015 11:16:30 AM" we see some requests for /dataimport that
> look like they are coming from the UI. and then at "Jun 10, 2015 11:17:01
> AM" we see a request for a full import started.
>
> We have no idea what the data import configuration file looks like, so we
> have no idea if clean=false is being used or not. it's certianly not
> specified in the URL.
>
> We see some more monitoring URLs returning hits=85898 and some more
> /repliation status calls, and then @ "Jun 10, 2015 11:18:02 AM" we see the
> first commit executed since hte server started up.
>
> there's no indication that this commit came from an external request (eg
> "/update") so probably was made by some internal request. One
> possiblility is that it came from DIH finishing -- but i doubt it, i'm
> fairly sure that would have involved more logging then this. A more
> probably scenerio is that it came from an autoCommit setting -- the fact
> that it is almost exactly 60 seconds after DIH started -- and almost
> exactly 60 seconds after DIH may have done a deleteAll query due to
> clean=true -- makes it seem very likely that this was a 1 minute
> autoCommit)
>
> (but since we don't have either hte data import config, or the
> solrconfig.xml, we have no way of knowing -- it's all just guess work.)
>
> Very importantly, note that this commit is not opening a new searcher...
>
> Jun 10, 2015 11:18:02 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start
> commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> Here are some other interesting things to note from the logging
> that comes from the DeletionPolicy when this commit happens...
>
> 1) it now notes that there are commits:num=2 on disk
> 2) the current index dir hasn't changed (index.20150311161021822) so
> some weird replication command didn't swap the world out from under us
> 3) the newest segment/generation are "segFN=segments_1b,generation=47"
> 4) the newest commit has no other files in it besides the segments file.
>
> this means, with out a doubt, there are no documents in this commits view
> of the index. they have all been deleted by something.
>
>
> At this point the *old* searcher (for commit generation 46) is still in
> use however -- nothing has done an openSearcher=true.
>
> we see more /dataimport status requests, and other requests that appear to
> come from the Solr UI, and more monitoring queries that still return
> hits=85898 because the same searcher is in use.
>
> At "Jun 10, 2015 11:27:04 AM" we see another commit happen -- again, no
> indication that this came from an outside /update request, so it might be
> from DIH, or it might be from an autoCommit setting. the fact that it is
> nearly exactly 10 minutes after DIH started (and probably did a clean=true
> deleteAll query) makes it seem extremely likely this is an autoSoftCommit
> setting kicking in.
>
> Very importantly, note that this softCommit *does* open a new searcher...
>
> Jun 10, 2015 11:27:04 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start
>
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
>
>
> In less then a second, this new searcher is warmed up and the next time we
> see a q=* monitoring query get logged, it returns hits=0.
>
> Note that at no point in the logs, after the DataImporter is started, do
> we see it log anything other then that it has initiated the request to
> MySQL -- we do see some logs starting ~ "Jun 10, 2015 11:41:19 AM"
> indicating that someone was using the Web UI to look at the dataimport
> handler's status report. it would be really nice to know what that person
> saw at that point -- because my guess is DIH was still running and was
> staled waiting for MySql, and hadn't even started adding docs to Solr (if
> it had, i'm certian there would have been some log of it).
>
> So instead, the combination of a (probable) DIH clean=true option and a
> (near certainty) autoCommit=60sec and autoSoftCommit=10min ment that a new
> commit was created after the clean, and that commit was opened as a new
> searcher before the DIH ever finished.
>
> Or at least ... that's my best guess based solely on the Solr logs, w/o
> any concrete info on what the configs look like, or what the User observed
> at the time when looking at the DIH status.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Indexing issue - index get deleted
Posted by Chris Hostetter <ho...@fucit.org>.
: The guys was using delta import anyway, so maybe the problem is
: different and not related to the clean.
that's not what the logs say.
Here's what i see...
Log begins with server startup @ "Jun 10, 2015 11:14:56 AM"
The DeletionPolicy for the "shopclue_prod" core is initialized at "Jun
10, 2015 11:15:04 AM" and we see a few interesting things here we note
for the future as we keep reading...
1) There is currently "commits:num=1" commits on disk
2) the current index dir in use is "index.20150311161021822"
3) the current segment & generation are "segFN=segments_1a,generation=46"
Immediately after this, we see some searcher warming using a searcher with
this same segments file, and then this searcher is registered ("Jun 10,
2015 11:15:05 AM") and the core is registered.
Next we see some replication polling, and we see what look like some
simple monitoring requests for "q=*" which return hits=85898 being
repeated over and over.
At "Jun 10, 2015 11:16:30 AM" we see some requests for /dataimport that
look like they are coming from the UI. and then at "Jun 10, 2015 11:17:01
AM" we see a request for a full import started.
We have no idea what the data import configuration file looks like, so we
have no idea if clean=false is being used or not. it's certianly not
specified in the URL.
We see some more monitoring URLs returning hits=85898 and some more
/repliation status calls, and then @ "Jun 10, 2015 11:18:02 AM" we see the
first commit executed since hte server started up.
there's no indication that this commit came from an external request (eg
"/update") so probably was made by some internal request. One
possiblility is that it came from DIH finishing -- but i doubt it, i'm
fairly sure that would have involved more logging then this. A more
probably scenerio is that it came from an autoCommit setting -- the fact
that it is almost exactly 60 seconds after DIH started -- and almost
exactly 60 seconds after DIH may have done a deleteAll query due to
clean=true -- makes it seem very likely that this was a 1 minute
autoCommit)
(but since we don't have either hte data import config, or the
solrconfig.xml, we have no way of knowing -- it's all just guess work.)
Very importantly, note that this commit is not opening a new searcher...
Jun 10, 2015 11:18:02 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
Here are some other interesting things to note from the logging
that comes from the DeletionPolicy when this commit happens...
1) it now notes that there are commits:num=2 on disk
2) the current index dir hasn't changed (index.20150311161021822) so
some weird replication command didn't swap the world out from under us
3) the newest segment/generation are "segFN=segments_1b,generation=47"
4) the newest commit has no other files in it besides the segments file.
this means, with out a doubt, there are no documents in this commits view
of the index. they have all been deleted by something.
At this point the *old* searcher (for commit generation 46) is still in
use however -- nothing has done an openSearcher=true.
we see more /dataimport status requests, and other requests that appear to
come from the Solr UI, and more monitoring queries that still return
hits=85898 because the same searcher is in use.
At "Jun 10, 2015 11:27:04 AM" we see another commit happen -- again, no
indication that this came from an outside /update request, so it might be
from DIH, or it might be from an autoCommit setting. the fact that it is
nearly exactly 10 minutes after DIH started (and probably did a clean=true
deleteAll query) makes it seem extremely likely this is an autoSoftCommit
setting kicking in.
Very importantly, note that this softCommit *does* open a new searcher...
Jun 10, 2015 11:27:04 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
In less then a second, this new searcher is warmed up and the next time we
see a q=* monitoring query get logged, it returns hits=0.
Note that at no point in the logs, after the DataImporter is started, do
we see it log anything other then that it has initiated the request to
MySQL -- we do see some logs starting ~ "Jun 10, 2015 11:41:19 AM"
indicating that someone was using the Web UI to look at the dataimport
handler's status report. it would be really nice to know what that person
saw at that point -- because my guess is DIH was still running and was
staled waiting for MySql, and hadn't even started adding docs to Solr (if
it had, i'm certian there would have been some log of it).
So instead, the combination of a (probable) DIH clean=true option and a
(near certainty) autoCommit=60sec and autoSoftCommit=10min ment that a new
commit was created after the clean, and that commit was opened as a new
searcher before the DIH ever finished.
Or at least ... that's my best guess based solely on the Solr logs, w/o
any concrete info on what the configs look like, or what the User observed
at the time when looking at the DIH status.
-Hoss
http://www.lucidworks.com/
Re: Indexing issue - index get deleted
Posted by Alessandro Benedetti <be...@gmail.com>.
Just taking a look to the code :
"
if (requestParams.containsKey("clean")) {
clean = StrUtils.parseBool( (String) requestParams.get("clean"), true);
} else if (DataImporter.DELTA_IMPORT_CMD.equals(command) ||
DataImporter.IMPORT_CMD.equals(command)) {
clean = false;
} else {
clean = debug ? false : true;
} "
Which make sense, as I would be surprised to see a delta import with a
default cleaning.
The guys was using delta import anyway, so maybe the problem is
different and not related to the clean.
But he needs definitely to give us more information .
Cheers
2015-06-10 12:11 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
> I was only speaking about full import regarding the default of
> clean=true. However, looking at the source code, it doesn't seem to
> differentiate especially between a full and a delta in relation to the
> default of clean=true, which would be pretty crappy. However, I'd need
> to try it.
>
> Upayavira
>
> On Wed, Jun 10, 2015, at 11:57 AM, Alessandro Benedetti wrote:
> > Wow, Upaya, I didn't know that clean was default=true in the delta import
> > as well!
> > I did know it was default in the full import, but I agree with you that
> > having a default to true for delta import is very dangerous !
> >
> > But assuming the user was using the delta import so far, if cleaning
> > every
> > time, how was possible to have a coherent index ?
> >
> > Using a delta import with clean=true should produce a non consistent
> > index
> > with only a subset ( the latest modified) of the entire data set !
> >
> > Cheers
> >
> > 2015-06-10 11:46 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
> >
> > > Note the clean= parameter to the DIH. It defaults to true. It will wipe
> > > your index before it runs. Perhaps it succeeded at wiping, but failed
> to
> > > connect to your database. Hence an empty DB?
> > >
> > > clean=true is, IMO, a very dangerous default option.
> > >
> > > Upayavira
> > >
> > > On Wed, Jun 10, 2015, at 10:59 AM, Midas A wrote:
> > > > Hi Alessandro,
> > > >
> > > > Please find the answers inline and help me out to figure out this
> > > > problem.
> > > >
> > > > 1) Solr version : *4.2.1*
> > > > 2) Solr architecture :* Master -slave/ Replication with
> requestHandler*
> > > >
> > > > 3) Kind of data source indexed : *Mysql *
> > > > 4) What happened to the datasource ? any change in there ? : *No
> change *
> > > > 5) Was the index actually deleted ? All docs deleted ? Index file
> > > > segments
> > > > deleted ? Index corrupted ? : *all docs deleted , segment files are
> > > > there.
> > > > index file is also there .*
> > > > 6) What about system resources ?
> > > > * JVM: 30 GB*
> > > > * RAM: 48 GB*
> > > >
> > > > *CPU : 8 core*
> > > >
> > > >
> > > > On Wed, Jun 10, 2015 at 2:13 PM, Alessandro Benedetti <
> > > > benedetti.alex85@gmail.com> wrote:
> > > >
> > > > > Let me try to help you, first of all I would like to encourage
> people
> > > to
> > > > > post more information about their scenario than "This is my log,
> index
> > > > > deleted, help me" :)
> > > > >
> > > > > This kind of Info can be really useful :
> > > > >
> > > > > 1) Solr version
> > > > > 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ?
> Manual
> > > > > Sharding ? Manual Replication ? where the problem happened ? )
> > > > > 3) Kind of data source indexed
> > > > > 4) What happened to the datasource ? any change in there ?
> > > > > 5) Was the index actually deleted ? All docs deleted ? Index file
> > > segments
> > > > > deleted ? Index corrupted ?
> > > > > 6) What about system resources ?
> > > > >
> > > > > These questions are only few example one that everyone should
> always
> > > post
> > > > > along their mysterious problem !
> > > > >
> > > > > Hope this helps,
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > 2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
> > > > >
> > > > > >
> > > > > > We are running full import and delta import crons .
> > > > > >
> > > > > > Fulll index once a day
> > > > > >
> > > > > > delta index : every 10 mins
> > > > > >
> > > > > >
> > > > > > last night my index automatically deleted(numdocs=0).
> > > > > >
> > > > > > attaching logs for review .
> > > > > >
> > > > > > please suggest to resolve the issue.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > --------------------------
> > > > >
> > > > > Benedetti Alessandro
> > > > > Visiting card : http://about.me/alessandro_benedetti
> > > > >
> > > > > "Tyger, tyger burning bright
> > > > > In the forests of the night,
> > > > > What immortal hand or eye
> > > > > Could frame thy fearful symmetry?"
> > > > >
> > > > > William Blake - Songs of Experience -1794 England
> > > > >
> > >
> >
> >
> >
> > --
> > --------------------------
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: Indexing issue - index get deleted
Posted by Upayavira <uv...@odoko.co.uk>.
I was only speaking about full import regarding the default of
clean=true. However, looking at the source code, it doesn't seem to
differentiate especially between a full and a delta in relation to the
default of clean=true, which would be pretty crappy. However, I'd need
to try it.
Upayavira
On Wed, Jun 10, 2015, at 11:57 AM, Alessandro Benedetti wrote:
> Wow, Upaya, I didn't know that clean was default=true in the delta import
> as well!
> I did know it was default in the full import, but I agree with you that
> having a default to true for delta import is very dangerous !
>
> But assuming the user was using the delta import so far, if cleaning
> every
> time, how was possible to have a coherent index ?
>
> Using a delta import with clean=true should produce a non consistent
> index
> with only a subset ( the latest modified) of the entire data set !
>
> Cheers
>
> 2015-06-10 11:46 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
>
> > Note the clean= parameter to the DIH. It defaults to true. It will wipe
> > your index before it runs. Perhaps it succeeded at wiping, but failed to
> > connect to your database. Hence an empty DB?
> >
> > clean=true is, IMO, a very dangerous default option.
> >
> > Upayavira
> >
> > On Wed, Jun 10, 2015, at 10:59 AM, Midas A wrote:
> > > Hi Alessandro,
> > >
> > > Please find the answers inline and help me out to figure out this
> > > problem.
> > >
> > > 1) Solr version : *4.2.1*
> > > 2) Solr architecture :* Master -slave/ Replication with requestHandler*
> > >
> > > 3) Kind of data source indexed : *Mysql *
> > > 4) What happened to the datasource ? any change in there ? : *No change *
> > > 5) Was the index actually deleted ? All docs deleted ? Index file
> > > segments
> > > deleted ? Index corrupted ? : *all docs deleted , segment files are
> > > there.
> > > index file is also there .*
> > > 6) What about system resources ?
> > > * JVM: 30 GB*
> > > * RAM: 48 GB*
> > >
> > > *CPU : 8 core*
> > >
> > >
> > > On Wed, Jun 10, 2015 at 2:13 PM, Alessandro Benedetti <
> > > benedetti.alex85@gmail.com> wrote:
> > >
> > > > Let me try to help you, first of all I would like to encourage people
> > to
> > > > post more information about their scenario than "This is my log, index
> > > > deleted, help me" :)
> > > >
> > > > This kind of Info can be really useful :
> > > >
> > > > 1) Solr version
> > > > 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual
> > > > Sharding ? Manual Replication ? where the problem happened ? )
> > > > 3) Kind of data source indexed
> > > > 4) What happened to the datasource ? any change in there ?
> > > > 5) Was the index actually deleted ? All docs deleted ? Index file
> > segments
> > > > deleted ? Index corrupted ?
> > > > 6) What about system resources ?
> > > >
> > > > These questions are only few example one that everyone should always
> > post
> > > > along their mysterious problem !
> > > >
> > > > Hope this helps,
> > > >
> > > > Cheers
> > > >
> > > >
> > > > 2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
> > > >
> > > > >
> > > > > We are running full import and delta import crons .
> > > > >
> > > > > Fulll index once a day
> > > > >
> > > > > delta index : every 10 mins
> > > > >
> > > > >
> > > > > last night my index automatically deleted(numdocs=0).
> > > > >
> > > > > attaching logs for review .
> > > > >
> > > > > please suggest to resolve the issue.
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > --------------------------
> > > >
> > > > Benedetti Alessandro
> > > > Visiting card : http://about.me/alessandro_benedetti
> > > >
> > > > "Tyger, tyger burning bright
> > > > In the forests of the night,
> > > > What immortal hand or eye
> > > > Could frame thy fearful symmetry?"
> > > >
> > > > William Blake - Songs of Experience -1794 England
> > > >
> >
>
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
Re: Indexing issue - index get deleted
Posted by Alessandro Benedetti <be...@gmail.com>.
Wow, Upaya, I didn't know that clean was default=true in the delta import
as well!
I did know it was default in the full import, but I agree with you that
having a default to true for delta import is very dangerous !
But assuming the user was using the delta import so far, if cleaning every
time, how was possible to have a coherent index ?
Using a delta import with clean=true should produce a non consistent index
with only a subset ( the latest modified) of the entire data set !
Cheers
2015-06-10 11:46 GMT+01:00 Upayavira <uv...@odoko.co.uk>:
> Note the clean= parameter to the DIH. It defaults to true. It will wipe
> your index before it runs. Perhaps it succeeded at wiping, but failed to
> connect to your database. Hence an empty DB?
>
> clean=true is, IMO, a very dangerous default option.
>
> Upayavira
>
> On Wed, Jun 10, 2015, at 10:59 AM, Midas A wrote:
> > Hi Alessandro,
> >
> > Please find the answers inline and help me out to figure out this
> > problem.
> >
> > 1) Solr version : *4.2.1*
> > 2) Solr architecture :* Master -slave/ Replication with requestHandler*
> >
> > 3) Kind of data source indexed : *Mysql *
> > 4) What happened to the datasource ? any change in there ? : *No change *
> > 5) Was the index actually deleted ? All docs deleted ? Index file
> > segments
> > deleted ? Index corrupted ? : *all docs deleted , segment files are
> > there.
> > index file is also there .*
> > 6) What about system resources ?
> > * JVM: 30 GB*
> > * RAM: 48 GB*
> >
> > *CPU : 8 core*
> >
> >
> > On Wed, Jun 10, 2015 at 2:13 PM, Alessandro Benedetti <
> > benedetti.alex85@gmail.com> wrote:
> >
> > > Let me try to help you, first of all I would like to encourage people
> to
> > > post more information about their scenario than "This is my log, index
> > > deleted, help me" :)
> > >
> > > This kind of Info can be really useful :
> > >
> > > 1) Solr version
> > > 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual
> > > Sharding ? Manual Replication ? where the problem happened ? )
> > > 3) Kind of data source indexed
> > > 4) What happened to the datasource ? any change in there ?
> > > 5) Was the index actually deleted ? All docs deleted ? Index file
> segments
> > > deleted ? Index corrupted ?
> > > 6) What about system resources ?
> > >
> > > These questions are only few example one that everyone should always
> post
> > > along their mysterious problem !
> > >
> > > Hope this helps,
> > >
> > > Cheers
> > >
> > >
> > > 2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
> > >
> > > >
> > > > We are running full import and delta import crons .
> > > >
> > > > Fulll index once a day
> > > >
> > > > delta index : every 10 mins
> > > >
> > > >
> > > > last night my index automatically deleted(numdocs=0).
> > > >
> > > > attaching logs for review .
> > > >
> > > > please suggest to resolve the issue.
> > > >
> > > >
> > >
> > >
> > > --
> > > --------------------------
> > >
> > > Benedetti Alessandro
> > > Visiting card : http://about.me/alessandro_benedetti
> > >
> > > "Tyger, tyger burning bright
> > > In the forests of the night,
> > > What immortal hand or eye
> > > Could frame thy fearful symmetry?"
> > >
> > > William Blake - Songs of Experience -1794 England
> > >
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: Indexing issue - index get deleted
Posted by Upayavira <uv...@odoko.co.uk>.
Note the clean= parameter to the DIH. It defaults to true. It will wipe
your index before it runs. Perhaps it succeeded at wiping, but failed to
connect to your database. Hence an empty DB?
clean=true is, IMO, a very dangerous default option.
Upayavira
On Wed, Jun 10, 2015, at 10:59 AM, Midas A wrote:
> Hi Alessandro,
>
> Please find the answers inline and help me out to figure out this
> problem.
>
> 1) Solr version : *4.2.1*
> 2) Solr architecture :* Master -slave/ Replication with requestHandler*
>
> 3) Kind of data source indexed : *Mysql *
> 4) What happened to the datasource ? any change in there ? : *No change *
> 5) Was the index actually deleted ? All docs deleted ? Index file
> segments
> deleted ? Index corrupted ? : *all docs deleted , segment files are
> there.
> index file is also there .*
> 6) What about system resources ?
> * JVM: 30 GB*
> * RAM: 48 GB*
>
> *CPU : 8 core*
>
>
> On Wed, Jun 10, 2015 at 2:13 PM, Alessandro Benedetti <
> benedetti.alex85@gmail.com> wrote:
>
> > Let me try to help you, first of all I would like to encourage people to
> > post more information about their scenario than "This is my log, index
> > deleted, help me" :)
> >
> > This kind of Info can be really useful :
> >
> > 1) Solr version
> > 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual
> > Sharding ? Manual Replication ? where the problem happened ? )
> > 3) Kind of data source indexed
> > 4) What happened to the datasource ? any change in there ?
> > 5) Was the index actually deleted ? All docs deleted ? Index file segments
> > deleted ? Index corrupted ?
> > 6) What about system resources ?
> >
> > These questions are only few example one that everyone should always post
> > along their mysterious problem !
> >
> > Hope this helps,
> >
> > Cheers
> >
> >
> > 2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
> >
> > >
> > > We are running full import and delta import crons .
> > >
> > > Fulll index once a day
> > >
> > > delta index : every 10 mins
> > >
> > >
> > > last night my index automatically deleted(numdocs=0).
> > >
> > > attaching logs for review .
> > >
> > > please suggest to resolve the issue.
> > >
> > >
> >
> >
> > --
> > --------------------------
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
> >
Re: Indexing issue - index get deleted
Posted by Midas A <te...@gmail.com>.
Hi Alessandro,
Please find the answers inline and help me out to figure out this problem.
1) Solr version : *4.2.1*
2) Solr architecture :* Master -slave/ Replication with requestHandler*
3) Kind of data source indexed : *Mysql *
4) What happened to the datasource ? any change in there ? : *No change *
5) Was the index actually deleted ? All docs deleted ? Index file segments
deleted ? Index corrupted ? : *all docs deleted , segment files are there.
index file is also there .*
6) What about system resources ?
* JVM: 30 GB*
* RAM: 48 GB*
*CPU : 8 core*
On Wed, Jun 10, 2015 at 2:13 PM, Alessandro Benedetti <
benedetti.alex85@gmail.com> wrote:
> Let me try to help you, first of all I would like to encourage people to
> post more information about their scenario than "This is my log, index
> deleted, help me" :)
>
> This kind of Info can be really useful :
>
> 1) Solr version
> 2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual
> Sharding ? Manual Replication ? where the problem happened ? )
> 3) Kind of data source indexed
> 4) What happened to the datasource ? any change in there ?
> 5) Was the index actually deleted ? All docs deleted ? Index file segments
> deleted ? Index corrupted ?
> 6) What about system resources ?
>
> These questions are only few example one that everyone should always post
> along their mysterious problem !
>
> Hope this helps,
>
> Cheers
>
>
> 2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
>
> >
> > We are running full import and delta import crons .
> >
> > Fulll index once a day
> >
> > delta index : every 10 mins
> >
> >
> > last night my index automatically deleted(numdocs=0).
> >
> > attaching logs for review .
> >
> > please suggest to resolve the issue.
> >
> >
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>
Re: Indexing issue - index get deleted
Posted by Alessandro Benedetti <be...@gmail.com>.
Let me try to help you, first of all I would like to encourage people to
post more information about their scenario than "This is my log, index
deleted, help me" :)
This kind of Info can be really useful :
1) Solr version
2) Solr architecture ( Solr Cloud ? Solr Cloud configuration ? Manual
Sharding ? Manual Replication ? where the problem happened ? )
3) Kind of data source indexed
4) What happened to the datasource ? any change in there ?
5) Was the index actually deleted ? All docs deleted ? Index file segments
deleted ? Index corrupted ?
6) What about system resources ?
These questions are only few example one that everyone should always post
along their mysterious problem !
Hope this helps,
Cheers
2015-06-10 9:15 GMT+01:00 Midas A <te...@gmail.com>:
>
> We are running full import and delta import crons .
>
> Fulll index once a day
>
> delta index : every 10 mins
>
>
> last night my index automatically deleted(numdocs=0).
>
> attaching logs for review .
>
> please suggest to resolve the issue.
>
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England