DIH causing "shutdown hook executing"?

[Phong Dais <ph...@gmail.com>]

Hi,

I am in the process of trying to index about 50 mil documents using the data
import handler.
For some reason, about 2 days into the import, I see this message "shutdown
hook executing" in the log and the solr web server instance exits
"gracefully".
I do not see any errors in the entire log.  This has happened twice now,
usually 5 mil or so documents into the import process.

Does anyone out there knows what this message mean?  It's an INFO log
message so I don't think it is caused by any error.
Does this problem occur because the os is asking the server to shut down
(for whatever reason) or is there something wrong with the server causing it
to shutdown?

Thanks for any help,
Phong

Re: DIH causing "shutdown hook executing"?

[Erick Erickson <er...@gmail.com>]

Try without autocommit or bump the limit up considerably to see
if it changes the behavior. You should not be getting
this kind of performance hit after the first million  docs, so, it's
probably worth exploring.

See if you can find anything in your logs that indicates what's
hogging the critical resource maybe?

Best
Erick

On Mon, Nov 29, 2010 at 3:08 PM, Phong Dais <ph...@gmail.com> wrote:

> It is entirely possible that the server is asking solr to shutdown.  I'll
> have to ask the admin.
> I'm running Solr-1.4 inside of Jetty.  I definitely have enough disk space.
> I think I did notice solr shutting down while it was idle.  I just
> disregarded it as a fluke...  Perhaps there's something going on.
> I will try to run this inside of tomcat and see what happens.
>
> Not sure if this is related but I had to change the <lockType> to single
> instead of the default "native".
> With native, I get a lock time out when starting up solr.  I also have
> <maxDocs> set to 10000.  I did not want to have millions of uncommitted
> docs.
> I'm running under Linux RedHat.
>
> Regarding speed, the first million or so documents is done very quickly
> (maybe 3 hrs) but after that, things slows down tremendously.
>
> Thanks for the advice regarding solrj.  I'll definitely look into that.
>
> P.
>
>
> On Mon, Nov 29, 2010 at 2:39 PM, Erick Erickson <erickerickson@gmail.com
> >wrote:
>
> > You're right, the OS is asking the server to shut down.  In the default
> > example under Jetty, this is a result of issuing a crtl-c. Is it possible
> > that something is asking your server to quit? What servlet container
> > are you running under? Does the Solr server run for more than this
> > period if you're NOT indexing? And are you sure you have enough
> > resources, especially disk space?
> >
> > On another note, I'm surprised that it's taking 2 days to index 5m
> > documents.
> > That's less than 30 docs/second and Solr should handle a considerably
> > greater load than that. For whatever that's worth...
> >
> > And what version of Solr are you using? You may want to consider
> > writing something in SolrJ to do your indexing, it'll provide you more
> > flexible control over indexing than DIH..
> >
> > Best
> > Erick
> >
> > On Mon, Nov 29, 2010 at 1:20 PM, Phong Dais <ph...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I am in the process of trying to index about 50 mil documents using the
> > > data
> > > import handler.
> > > For some reason, about 2 days into the import, I see this message
> > "shutdown
> > > hook executing" in the log and the solr web server instance exits
> > > "gracefully".
> > > I do not see any errors in the entire log.  This has happened twice
> now,
> > > usually 5 mil or so documents into the import process.
> > >
> > > Does anyone out there knows what this message mean?  It's an INFO log
> > > message so I don't think it is caused by any error.
> > > Does this problem occur because the os is asking the server to shut
> down
> > > (for whatever reason) or is there something wrong with the server
> causing
> > > it
> > > to shutdown?
> > >
> > > Thanks for any help,
> > > Phong
> > >
> >
>

Re: DIH causing "shutdown hook executing"?

[Phong Dais <ph...@gmail.com>]

It is entirely possible that the server is asking solr to shutdown.  I'll
have to ask the admin.
I'm running Solr-1.4 inside of Jetty.  I definitely have enough disk space.
I think I did notice solr shutting down while it was idle.  I just
disregarded it as a fluke...  Perhaps there's something going on.
I will try to run this inside of tomcat and see what happens.

Not sure if this is related but I had to change the <lockType> to single
instead of the default "native".
With native, I get a lock time out when starting up solr.  I also have
<maxDocs> set to 10000.  I did not want to have millions of uncommitted
docs.
I'm running under Linux RedHat.

Regarding speed, the first million or so documents is done very quickly
(maybe 3 hrs) but after that, things slows down tremendously.

Thanks for the advice regarding solrj.  I'll definitely look into that.

P.


On Mon, Nov 29, 2010 at 2:39 PM, Erick Erickson <er...@gmail.com>wrote:

> You're right, the OS is asking the server to shut down.  In the default
> example under Jetty, this is a result of issuing a crtl-c. Is it possible
> that something is asking your server to quit? What servlet container
> are you running under? Does the Solr server run for more than this
> period if you're NOT indexing? And are you sure you have enough
> resources, especially disk space?
>
> On another note, I'm surprised that it's taking 2 days to index 5m
> documents.
> That's less than 30 docs/second and Solr should handle a considerably
> greater load than that. For whatever that's worth...
>
> And what version of Solr are you using? You may want to consider
> writing something in SolrJ to do your indexing, it'll provide you more
> flexible control over indexing than DIH..
>
> Best
> Erick
>
> On Mon, Nov 29, 2010 at 1:20 PM, Phong Dais <ph...@gmail.com> wrote:
>
> > Hi,
> >
> > I am in the process of trying to index about 50 mil documents using the
> > data
> > import handler.
> > For some reason, about 2 days into the import, I see this message
> "shutdown
> > hook executing" in the log and the solr web server instance exits
> > "gracefully".
> > I do not see any errors in the entire log.  This has happened twice now,
> > usually 5 mil or so documents into the import process.
> >
> > Does anyone out there knows what this message mean?  It's an INFO log
> > message so I don't think it is caused by any error.
> > Does this problem occur because the os is asking the server to shut down
> > (for whatever reason) or is there something wrong with the server causing
> > it
> > to shutdown?
> >
> > Thanks for any help,
> > Phong
> >
>

Re: DIH causing "shutdown hook executing"?

[Erick Erickson <er...@gmail.com>]

You're right, the OS is asking the server to shut down.  In the default
example under Jetty, this is a result of issuing a crtl-c. Is it possible
that something is asking your server to quit? What servlet container
are you running under? Does the Solr server run for more than this
period if you're NOT indexing? And are you sure you have enough
resources, especially disk space?

On another note, I'm surprised that it's taking 2 days to index 5m
documents.
That's less than 30 docs/second and Solr should handle a considerably
greater load than that. For whatever that's worth...

And what version of Solr are you using? You may want to consider
writing something in SolrJ to do your indexing, it'll provide you more
flexible control over indexing than DIH..

Best
Erick

On Mon, Nov 29, 2010 at 1:20 PM, Phong Dais <ph...@gmail.com> wrote:

> Hi,
>
> I am in the process of trying to index about 50 mil documents using the
> data
> import handler.
> For some reason, about 2 days into the import, I see this message "shutdown
> hook executing" in the log and the solr web server instance exits
> "gracefully".
> I do not see any errors in the entire log.  This has happened twice now,
> usually 5 mil or so documents into the import process.
>
> Does anyone out there knows what this message mean?  It's an INFO log
> message so I don't think it is caused by any error.
> Does this problem occur because the os is asking the server to shut down
> (for whatever reason) or is there something wrong with the server causing
> it
> to shutdown?
>
> Thanks for any help,
> Phong
>