You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Phil Scadden <P....@gns.cri.nz> on 2017/03/01 23:41:42 UTC
Excessive Wire logging while indexing.
Using Solr 6.4.1 on windows. Installed and trial POST on my directories worked okay. However, now trying to create an index from code running on tomcat on the same machine as SOLR server with my own schema. Indexing of PDF is very slow. Investigating that find my tomcat output full of wire
12:07:06,758 DEBUG wire:72 - >> "[0xe3][0xe0][0xc7]L[0xf5][0xea][\r]?
12:07:06,763 DEBUG wire:72 - >> "f[0x81][0xb0]b[0xca][0xfa][0xb7][0x1f]n[0xff][0x0][0xa8][0xd0][0x16][0xbb]*[0xfe][0x95][0x98]-[0xbd][0xb7]-
12:07:06,805 DEBUG wire:72 - >> "X[0x3][0xd4][0xcf]OOS:[0x94][0x8b][0xe2][0x89][0x8f][0xc9]rClVQ[0x85][0x1e][0x82]T[0x9e][0xe7]N[0xf4][0xfa]-
This cant be helping. My code is:
try {
ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/update/extract");
File f = new File(filename);
ContentStreamBase.FileStream cs = new ContentStreamBase.FileStream(f);
up.addContentStream(cs);
up.setParam("literal.id",f.getPath());
up.setParam("literal.location", idString);
up.setParam("literal.access",access.toString());
up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
solr.request(up);
All the logging generated by last line. I don’t have any httpclient.wire lines in my log4j.properties (presume these are from httpclient.wire). What do I do to turn this off?
Phil Scadden, Geoscientist
GNS Science I Te Pῡ Ao
764 Cumberland Street, Dunedin 9016, Private Bag 1930, Dunedin 9054, New Zealand
Ph +64 3 4799663 Mob +64 027 3463185 I Fax OFFICE-FAX +64 3 477 5232
http://www.gns.cri.nz/ I Email: p.scadden@gns.cri.nz
Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
Re: Excessive Wire logging while indexing.
Posted by Erick Erickson <er...@gmail.com>.
Glad to hear it's working. The trick (as you've probably discovered)
is to properly
map the meta-data to Solr fields. The extracting request handler does
this, but the
real underlying issue is that there's no real standard. Word docs
might have "last_editor",
PDFs might have just "author". And on and on and on.
Anyway, sounds like you're on your way. The code snippet Shawn
referenced dumps all
the meta-data Tika finds so you can figure out what you need.
Best,
Erick
On Thu, Mar 2, 2017 at 11:56 AM, Phil Scadden <P....@gns.cri.nz> wrote:
> Got it all working with Tika and SolrJ. (Got the correct artifacts). Much faster now too which is good. Thanks very much for your help.
> Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
RE: Excessive Wire logging while indexing.
Posted by Phil Scadden <P....@gns.cri.nz>.
Got it all working with Tika and SolrJ. (Got the correct artifacts). Much faster now too which is good. Thanks very much for your help.
Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
Re: Excessive Wire logging while indexing.
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/1/2017 6:59 PM, Phil Scadden wrote:
> Exceptions never triggered but metadata was essentially empty except
> for contentType, and content was always an empty string. I don\u2019t know
> what parser was doing, but I gave up and with the extractHandler route
> instead which did at least build a full index.
With the extracting request handler, Tika is running inside Solr. The
handler code is customized for Solr's needs, so it usually works. When
Tika has one of its well-known issues though, the entire JVM (which
includes Solr) suffers as well.
I do not know how to write Tika code, but this blog post covers an
example program that uses Tika with SolrJ, so the processing is outside
Solr:
https://lucidworks.com/2012/02/14/indexing-with-solrj/
It also uses a database, but it should be relatively easy to remove that.
Thanks,
Shawn
RE: Excessive Wire logging while indexing.
Posted by Phil Scadden <P....@gns.cri.nz>.
>Another side issue: Using the extracting handler for handling rich documents is discouraged. Tika (which is what is used by the extracting
>handler) is pretty amazing software, but it has a habit of crashing or consuming all the heap memory when it encounters a document that it doesn't >know how to properly handle. It is best to run Tika in your external program and send its output to Solr, so that if there's a problem, it won't affect >your search capability.
As an alternative to earlier code, I had tried this (exactly the same set of files going in)
File f = new File(filename);
ContentHandler textHandler = new BodyContentHandler(10*1024*1024);
Metadata metadata = new Metadata();
Parser parser = new AutoDetectParser();
ParseContext context = new ParseContext();
InputStream input = new FileInputStream(f);
try {
parser.parse(input, textHandler, metadata, context);
} catch (Exception e) {
Logger.getLogger(JsMapAdminService.class.getName()).log(Level.SEVERE, null,String.format("File %s failed", f.getCanonicalPath()));
e.printStackTrace();
}
SolrInputDocument up = new SolrInputDocument();
up.addField("id",f.getCanonicalPath());
up.addField("fileLocation",idString);
up.addField("access",access);
up.addField("title",metadata.get("title"));
up.addField("author",metadata.get("author"));
String content = textHandler.toString();
up.addField("_text_",content);
solr.add(up);
return true;
Exceptions never triggered but metadata was essentially empty except for contentType, and content was always an empty string. I don’t know what parser was doing, but I gave up and with the extractHandler route instead which did at least build a full index.
Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
RE: Excessive Wire logging while indexing.
Posted by Phil Scadden <P....@gns.cri.nz>.
The logging is coming from application which is running in Tomcat. Solr itself is running in the embedded Jetty.
And yes, another look at the log4j and I see that rootlogger is set to DEBUG. I've changed that/
>On the Solr server side, the 6.4.x versions have a bug that causes extremely high CPU usage and very slow operation. This will be fixed in 6.4.2, which >will hopefully be out soon. There is currently no ETA for this release.
Thanks for that. I will watch for the release.
>Another side issue: Using the extracting handler for handling rich documents is discouraged. Tika (which is what is used by the extracting
>handler) is pretty amazing software, but it has a habit of crashing or consuming all the heap memory when it encounters a document that it doesn't >know how to properly handle. It is best to run Tika in your external program and send its output to Solr, so that if there's a problem, it won't affect >your search capability.
That was frankly what I thought I was doing.
The calling code is:
public Long SolrIndex(String path, String target, String URL, List<Role>roles, CurrentState curState) {
HttpSolrClient solr = new HttpSolrClient(URL);
try {
solr.deleteByQuery("*:*");
solr.commit();
} catch (SolrServerException | IOException ex) {
Logger.getLogger(JsMapService.class.getName()).log(Level.SEVERE, null, ex);
}
... lot of stuff to find data to index and set fields
...
if (solrIndexFile(solr,filename,access,mi.getMcontent())) count++;
try {
solr.commit();
} catch (SolrServerException | IOException ex) {
Logger.getLogger(JsMapService.class.getName()).log(Level.SEVERE, null, ex);
}
Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
Re: Excessive Wire logging while indexing.
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/1/2017 4:41 PM, Phil Scadden wrote:
> Using Solr 6.4.1 on windows. Installed and trial POST on my directories worked okay. However, now trying to create an index from code running on tomcat on the same machine as SOLR server with my own schema. Indexing of PDF is very slow. Investigating that find my tomcat output full of wire
> 12:07:06,758 DEBUG wire:72 - >> "[0xe3][0xe0][0xc7]L[0xf5][0xea][\r]?
> 12:07:06,763 DEBUG wire:72 - >> "f[0x81][0xb0]b[0xca][0xfa][0xb7][0x1f]n[0xff][0x0][0xa8][0xd0][0x16][0xbb]*[0xfe][0x95][0x98]-[0xbd][0xb7]-
> 12:07:06,805 DEBUG wire:72 - >> "X[0x3][0xd4][0xcf]OOS:[0x94][0x8b][0xe2][0x89][0x8f][0xc9]rClVQ[0x85][0x1e][0x82]T[0x9e][0xe7]N[0xf4][0xfa]-
> This cant be helping. My code is:
>
> try {
> ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/update/extract");
> File f = new File(filename);
> ContentStreamBase.FileStream cs = new ContentStreamBase.FileStream(f);
> up.addContentStream(cs);
> up.setParam("literal.id",f.getPath());
> up.setParam("literal.location", idString);
> up.setParam("literal.access",access.toString());
> up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> solr.request(up);
>
> All the logging generated by last line. I don\u2019t have any httpclient.wire lines in my log4j.properties (presume these are from httpclient.wire). What do I do to turn this off?
Part of the problem is that you've got your logging set to at least
DEBUG. You shouldn't do that unless you're actually debugging
something, and even then I would strongly recommend that you NOT set the
global level to DEBUG, but only set the specific classes you are
troubleshooting. SolrJ depends on many pieces of software, all of which
have their own logging. DEBUG logging tends to be VERY verbose.
I searched the codebase and did not find the specific logging that you
have mentioned, so it's probably being logged by one of SolrJ's
dependencies, or possibly Tomcat, not SolrJ itself. Be aware that if
Solr is running in Tomcat, it's an unsupported configuration. You
haven't said that this is the case, so I am not going to assume that
it's the case.
https://wiki.apache.org/solr/WhyNoWar
On the Solr server side, the 6.4.x versions have a bug that causes
extremely high CPU usage and very slow operation. This will be fixed in
6.4.2, which will hopefully be out soon. There is currently no ETA for
this release.
https://issues.apache.org/jira/browse/SOLR-10130
Another side issue: Using the extracting handler for handling rich
documents is discouraged. Tika (which is what is used by the extracting
handler) is pretty amazing software, but it has a habit of crashing or
consuming all the heap memory when it encounters a document that it
doesn't know how to properly handle. It is best to run Tika in your
external program and send its output to Solr, so that if there's a
problem, it won't affect your search capability.
Thanks,
Shawn