You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Paden <ru...@gmail.com> on 2015/07/20 23:03:13 UTC
Data Import Handler Stays Idle
Hello,
I'm currently trying to index about 54,000 files with the Solr Data Import
Handler and I've got a small problem. It fetches about half (28,289) of the
54,000 files and it process about 14,146 documents before it stops and just
stands idle. Here's the status output
{
"responseHeader": {
"status": 0,
"QTime": 0
},
"initArgs": [
"defaults",
[
"config",
"db-data-config.xml",
"update.chain",
"skip-empty"
]
],
"command": "status",
"status": "idle",
"importResponse": "",
"statusMessages": {
"Time Elapsed": "2:39:53.191",
"Total Requests made to DataSource": "1",
"Total Rows Fetched": "28289",
"Total Documents Processed": "14146",
"Total Documents Skipped": "0",
"Full Dump Started": "2015-07-20 18:19:17"
}
}
it has a green arrow next to the header where it says number or documents
fetched/process but it doesn't say that it's done indexing. It also doesn't
have the commit line that I've seen on my other core that I indexed about
290 documents on. This is the second time that I have tried to index these
files. I swung by the office this last weekend to see how the index was
going and (I didn't write the numbers down but I guess I should have) I seem
to remember it being pretty much at this EXACT spot when the dataimport
handler starting being idle the last time too. Is there some line in the
solr config that I have to change to actually commit some of the documents.
That way so it isn't all at once? Is there some doc limit I have reached
that I don't know exists? Are the PDF's too large and killing tika (and solr
with it). I'm really kind of stuck here.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp4218250.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Paden <ru...@gmail.com>.
Okay. I'm going to run the index again with specifications that you
recommended. This could take a few hours but I will post the entire trace on
that error when it pops up again and I will let you guys know the results of
increasing the heap size.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp4218250p4218382.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Data Import Handler Stays Idle
Posted by "Allison, Timothy B." <ta...@mitre.org>.
> There are some zip files inside the directory and have been addressed
> to in the database. I'm thinking those are the one's it's jumping
> right over.
With SOLR-7189, which should have kicked in for 5.1, Tika shouldn't skip over Zip files, it should process all the contents of those zips and concatenate the extracted text into one string.
-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Tuesday, July 21, 2015 10:41 AM
To: solr-user@lucene.apache.org
Subject: Re: Data Import Handler Stays Idle
On 7/21/2015 8:17 AM, Paden wrote:
> There are some zip files inside the directory and have been addressed
> to in the database. I'm thinking those are the one's it's jumping
> right over. They are not the issue. At least I'm 95% sure. And Shawn
> if you're still watching I'm sorry I'm using solr-5.1.0.
Have you started Solr with a larger heap than the default 512MB in Solr 5.x? Tika can require a lot of memory. I would have expected there to be OutOfMemoryError exceptions in the log if that were the problem, though.
You may need to use the "-m" option on the startup scripts to increase the max heap. Starting with "-m 2g" would be a good idea.
Also, seeing the entire multi-line IOException from the log (which may be dozens of lines) could be important.
Thanks,
Shawn
RE: Data Import Handler Stays Idle
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Only a month late to respond, and the response likely won't help.
I agree with Shawn that Tika can be a memory hog. I try to leave 1GB per thread, but your mileage will vary dramatically depending on your docs. I'd expect that you'd get an OOM, though, somewhere...
There have been rare bugs in various parsers, including the PDFParser, in various versions of Tika that cause permanent hangs. I haven't experimented with DIH and known trigger files, but I suspect you'd get the behavior that you're seeing if this were to happen.
So, short of rolling your own ETL'r in lieu of DIH or hardening DIH to run tika in a different process (tika-server, perhaps -- https://issues.apache.org/jira/browse/SOLR-7632) or going big with Hadoop, morphlines, etc, your only hope is to upgrade Tika and hope that that was one of the bugs that we've already identified and fixed.
If you do go with morphlines...I don't think this has been fixed yet: https://github.com/kite-sdk/kite/issues/397
Did you ever figure out what was going wrong?
Best,
Tim
-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Tuesday, July 21, 2015 10:41 AM
To: solr-user@lucene.apache.org
Subject: Re: Data Import Handler Stays Idle
On 7/21/2015 8:17 AM, Paden wrote:
> There are some zip files inside the directory and have been addressed
> to in the database. I'm thinking those are the one's it's jumping
> right over. They are not the issue. At least I'm 95% sure. And Shawn
> if you're still watching I'm sorry I'm using solr-5.1.0.
Have you started Solr with a larger heap than the default 512MB in Solr 5.x? Tika can require a lot of memory. I would have expected there to be OutOfMemoryError exceptions in the log if that were the problem, though.
You may need to use the "-m" option on the startup scripts to increase the max heap. Starting with "-m 2g" would be a good idea.
Also, seeing the entire multi-line IOException from the log (which may be dozens of lines) could be important.
Thanks,
Shawn
Re: Data Import Handler Stays Idle
Posted by Paden <ru...@gmail.com>.
Hey shawn when I use the -m 2g command in my script I get the error a 'cannot
open [path]/server/logs/solr.log for reading: No such file or directory' I
do not see how this would affect that.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp4218250p4218389.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/21/2015 8:17 AM, Paden wrote:
> There are some zip files inside the directory and have been addressed to in
> the database. I'm thinking those are the one's it's jumping right over. They
> are not the issue. At least I'm 95% sure. And Shawn if you're still watching
> I'm sorry I'm using solr-5.1.0.
Have you started Solr with a larger heap than the default 512MB in Solr
5.x? Tika can require a lot of memory. I would have expected there to
be OutOfMemoryError exceptions in the log if that were the problem, though.
You may need to use the "-m" option on the startup scripts to increase
the max heap. Starting with "-m 2g" would be a good idea.
Also, seeing the entire multi-line IOException from the log (which may
be dozens of lines) could be important.
Thanks,
Shawn
Re: Data Import Handler Stays Idle
Posted by Paden <ru...@gmail.com>.
There are some zip files inside the directory and have been addressed to in
the database. I'm thinking those are the one's it's jumping right over. They
are not the issue. At least I'm 95% sure. And Shawn if you're still watching
I'm sorry I'm using solr-5.1.0.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp4218250p4218371.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Raja Pothuganti <RP...@competitrack.com>.
>Yes the number of unimported matches (with IOExceptions)
What is the IOException about?
On 7/20/15, 5:10 PM, "Paden" <ru...@gmail.com> wrote:
>Yes the number of unimported matches. No I did not specify "false" to
>commit
>on any of my dataimporthandler. Since it defaults to true I really didn't
>take it into account though.
>
>
>
>--
>View this message in context:
>http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp421825
>0p4218262.html
>Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Paden <ru...@gmail.com>.
Yes the number of unimported matches. No I did not specify "false" to commit
on any of my dataimporthandler. Since it defaults to true I really didn't
take it into account though.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp4218250p4218262.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Raja Pothuganti <RP...@competitrack.com>.
Number of Ioexceptions , are they equal to un-imported/un processed
documents?
By any chance commit set to false in import request
example:
http://localhost:8983/solr/db/dataimport?command=full-import&commit=false
Thanks
Raja
On 7/20/15, 4:51 PM, "Paden" <ru...@gmail.com> wrote:
>I was consistently checking the logs to see if there were any errors that
>would give me any idling. There were no errors except for a few skipped
>documents due to some Illegal IOexceptions from Tika but none of those
>occurred around the time that solr began idling. A lot of font warnings.
>But
>again. Nothing but font warnings around time of idling.
>
>
>
>--
>View this message in context:
>http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp421825
>0p4218260.html
>Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Paden <ru...@gmail.com>.
I was consistently checking the logs to see if there were any errors that
would give me any idling. There were no errors except for a few skipped
documents due to some Illegal IOexceptions from Tika but none of those
occurred around the time that solr began idling. A lot of font warnings. But
again. Nothing but font warnings around time of idling.
--
View this message in context: http://lucene.472066.n3.nabble.com/Data-Import-Handler-Stays-Idle-tp4218250p4218260.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Data Import Handler Stays Idle
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/20/2015 3:03 PM, Paden wrote:
> I'm currently trying to index about 54,000 files with the Solr Data Import
> Handler and I've got a small problem. It fetches about half (28,289) of the
> 54,000 files and it process about 14,146 documents before it stops and just
> stands idle. Here's the status output
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 0
> },
> "initArgs": [
> "defaults",
> [
> "config",
> "db-data-config.xml",
> "update.chain",
> "skip-empty"
> ]
> ],
> "command": "status",
> "status": "idle",
> "importResponse": "",
> "statusMessages": {
> "Time Elapsed": "2:39:53.191",
> "Total Requests made to DataSource": "1",
> "Total Rows Fetched": "28289",
> "Total Documents Processed": "14146",
> "Total Documents Skipped": "0",
> "Full Dump Started": "2015-07-20 18:19:17"
> }
> }
>
> it has a green arrow next to the header where it says number or documents
> fetched/process but it doesn't say that it's done indexing. It also doesn't
> have the commit line that I've seen on my other core that I indexed about
> 290 documents on. This is the second time that I have tried to index these
> files. I swung by the office this last weekend to see how the index was
> going and (I didn't write the numbers down but I guess I should have) I seem
> to remember it being pretty much at this EXACT spot when the dataimport
> handler starting being idle the last time too. Is there some line in the
> solr config that I have to change to actually commit some of the documents.
> That way so it isn't all at once? Is there some doc limit I have reached
> that I don't know exists? Are the PDF's too large and killing tika (and solr
> with it). I'm really kind of stuck here.
What Solr version are you using, and if you look for the Solr logfile on
the disk, do you see any errors in it? There may be a few more
questions to ask, but they will depend on the answers to those two.
You may be on to something with the idea of a PDF document that's
killing Tika.
Thanks,
Shawn