You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Simon Blandford <si...@bkconnect.net> on 2016/06/01 14:45:33 UTC
Re: Metadata and HTML ending up in searchable text
Thanks Timothy,
Will give the DIH a try. I have submitted a bug report.
Regards,
Simon
On 31/05/16 13:22, Allison, Timothy B. wrote:
>>> From the same page, extractFormat=text only applies when extractOnly
>>> is true, which just shows the output from tika without indexing the document.
> Y, sorry. I just looked through the source code. You're right. If you use DIH (TikaEntityProcessor) instead of Solr Cell (ExtractingDocumentLoader), you should be able to set the handler type by setting the "format" attribute, and "text" is one option there.
>
>>> I just want to make sure I'm not missing something really obvious before submitting a bug report.
> I don't think you are.
>
>>> From the same page, extractFormat=text only applies when extractOnly
>>> is true, which just shows the output from tika without indexing the document.
>>> Running it in "extractOnly" mode resulting in a XML output. The
>>> difference between selecting "text" or "xml" format is that the
>>> escaped document in the <response> tag is either the original HTML
>>> (xml mode) or stripped HTML (text mode). It seems some Javascript
>>> creeps into the text version. (See below)
>>>
>>> Regards,
>>> Simon
>>>
>>> HTML mode sample:
>>> <?xml version="1.0" encoding="UTF-8"?> <response> <lst
>>> name="responseHeader"><int name="status">0</int><int
>>> name="QTime">51</int></lst><str
>>> name="UsingMailingLists.html"><?xml
>>> version="1.0" encoding="UTF-8"?>
>>> <html xmlns="http://www.w3.org/1999/xhtml">
>>> <head>
>>> <link
>>> rel="stylesheet" type="text/css" charset="utf-8" media="all"
>>> href="/wiki/modernized/css/common.css"/>
>>> <link rel="stylesheet" type="text/css" charset="utf-8"
>>> media="screen" href="/wiki/modernized/css/screen.css"/>
>>> <link rel="stylesheet" type="text/css" charset="utf-8"
>>> media="print" href="/wiki/modernized/css/print.css"/>.......
>>>
>>> TEXT mode (Blank lines stripped):
>>> <response>
>>> <lst name="responseHeader"><int name="status">0</int><int
>>> name="QTime">47</int></lst><str name="UsingMailingLists.html">
>>> UsingMailingLists - Solr Wiki
>>> Search:
>>> <!--// Initialize search form
>>> var f = document.getElementById('searchform');
>>> f.getElementsByTagName('label')[0].style.display = 'none'; var e =
>>> document.getElementById('searchinput');
>>> searchChange(e);
>>> searchBlur(e);
>>> //-->
>>> Solr Wiki
>>> Login
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 27/05/16 13:31, Allison, Timothy B. wrote:
>>>> I'm only minimally familiar with Solr Cell, but...
>>>>
>>>> 1) It looks like you aren't setting extractFormat=text. According
>>>> to [0]...the default is xhtml which will include a bunch of the metadata.
>>>> 2) is there an attr_* dynamic field in your index with type="ignored"?
>>>> This would strip out the attr_ fields so they wouldn't even be
>>>> indexed...if you don't want them.
>>>>
>>>> As for the HTML file, it looks like Tika is failing to strip out the
>>>> style section. Try running the file alone with tika-app: java -jar
>>>> tika-app.jar -t inputfile.html. If you are finding the noise there.
>>>> Please open an issue on our JIRA:
>>>> https://issues.apache.org/jira/browse/tika
>>>>
>>>>
>>>> [0]
>>>> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with
>>>> +Solr+Cell+using+Apache+Tika
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Simon Blandford [mailto:simon.blandford@bkconnect.net]
>>>> Sent: Thursday, May 26, 2016 9:49 AM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: Metadata and HTML ending up in searchable text
>>>>
>>>> Hi,
>>>>
>>>> I am using Solr 6.0 on Ubuntu 14.04.
>>>>
>>>> I am ending up with loads of junk in the text body. It starts like,
>>>>
>>>> The JSON entry output of a search result shows the indexed text
>>>> starting with...
>>>> body_txt_en: " stream_size 36499 X-Parsed-By
>>>> org.apache.tika.parser.DefaultParser X-Parsed-By...."
>>>>
>>>> And then once it gets to the actual text I get CSS class names
>>>> appearing that were in <p> or <div> tags etc.
>>>> e.g. "....the power of calibre3 silence calibre2 and....", where
>>>> "calibre3" etc are the CSS class names.
>>>>
>>>> All this junk is searchable and is polluting the index.
>>>>
>>>> I would like to index _only_ the actual content I am interested in
>>>> searching for.
>>>>
>>>> Steps to reproduce:
>>>>
>>>> 1) Solr installed by untaring solr tgz in /opt.
>>>>
>>>> 2) Core created by typing "bin/solr create -c mycore"
>>>>
>>>> 3) Solr started with bin/solr start
>>>>
>>>> 4) TXT document index using the following command curl
>>>> "http://localhost:8983/solr/mycore/update/extract?literal.id=doc1&uprefix=attr_&fmap.content=body_txt_en&commit=true"
>>>> -F
>>>>
>>>> "content/UsingMailingLists.txt=@/home/user/Documents/library/UsingMailingLists.txt"
>>>>
>>>> 5) HTML document index using following command curl
>>>> "http://localhost:8983/solr/mycore/update/extract?literal.id=doc2&uprefix=attr_&fmap.content=body_txt_en&commit=true"
>>>> -F
>>>>
>>>> "content/UsingMailingLists.html=@/home/user/Documents/library/UsingMailingLists.html"
>>>>
>>>> 6) Query using URL:
>>>> http://localhost:8983/solr/mycore/select?q=especially&wt=json
>>>>
>>>> Result:
>>>>
>>>> For the txt file, I get the following JSON for the document...
>>>>
>>>> {
>>>> id: "doc1",
>>>> attr_stream_size: [
>>>> "8107"
>>>> ],
>>>> attr_x_parsed_by: [
>>>> "org.apache.tika.parser.DefaultParser",
>>>> "org.apache.tika.parser.txt.TXTParser"
>>>> ],
>>>> attr_stream_content_type: [
>>>> "text/plain"
>>>> ],
>>>> attr_stream_name: [
>>>> "UsingMailingLists.txt"
>>>> ],
>>>> attr_stream_source_info: [
>>>> "content/UsingMailingLists.txt"
>>>> ],
>>>> attr_content_encoding: [
>>>> "ISO-8859-1"
>>>> ],
>>>> attr_content_type: [
>>>> "text/plain; charset=ISO-8859-1"
>>>> ],
>>>> body_txt_en: " stream_size 8107 X-Parsed-By
>>>> org.apache.tika.parser.DefaultParser X-Parsed-By
>>>> org.apache.tika.parser.txt.TXTParser stream_content_type text/plain
>>>> stream_name UsingMailingLists.txt stream_source_info
>>>> content/UsingMailingLists.txt Content-Encoding ISO-8859-1
>>>> Content-Type text/plain; charset=ISO-8859-1 Search: [value ]
>>>> [Titles] [Text] Solr_Wiki Login ****** UsingMailingLists ****** *
>>>> FrontPage * RecentChanges...etc",
>>>> _version_: 1535398235801124900
>>>> }
>>>>
>>>> For the HTML file, I get the following JSON for the document...
>>>>
>>>> {
>>>> id: "doc2",
>>>> attr_stream_size: [
>>>> "20440"
>>>> ],
>>>> attr_x_parsed_by: [
>>>> "org.apache.tika.parser.DefaultParser",
>>>> "org.apache.tika.parser.html.HtmlParser"
>>>> ],
>>>> attr_stream_content_type: [
>>>> "text/html"
>>>> ],
>>>> attr_stream_name: [
>>>> "UsingMailingLists.html"
>>>> ],
>>>> attr_stream_source_info: [
>>>> "content/UsingMailingLists.html"
>>>> ],
>>>> attr_dc_title: [
>>>> "UsingMailingLists - Solr Wiki"
>>>> ],
>>>> attr_content_encoding: [
>>>> "UTF-8"
>>>> ],
>>>> attr_robots: [
>>>> "index,nofollow"
>>>> ],
>>>> attr_title: [
>>>> "UsingMailingLists - Solr Wiki"
>>>> ],
>>>> attr_content_type: [
>>>> "text/html; charset=utf-8"
>>>> ],
>>>> body_txt_en: " stylesheet text/css utf-8 all
>>>> /wiki/modernized/css/common.css stylesheet text/css utf-8 screen
>>>> /wiki/modernized/css/screen.css stylesheet text/css utf-8 print
>>>> /wiki/modernized/css/print.css stylesheet text/css utf-8 projection
>>>> /wiki/modernized/css/projection.css alternate Solr Wiki:
>>>> UsingMailingLists
>>>>
>>>> /solr/UsingMailingLists?diffs=1&show_att=1&action=rss_rc&unique=0&pa
>>>> ge=UsingMailingLists&ddiffs=1 application/rss+xml Start
>>>> /solr/FrontPage Alternate Wiki Markup
>>>> /solr/UsingMailingLists?action=raw Alternate print Print View
>>>> /solr/UsingMailingLists?action=print Search /solr/FindPage Index
>>>> /solr/TitleIndex Glossary /solr/WordIndex Help
>>>> /solr/HelpOnFormatting stream_size 20440 X-Parsed-By
>>>> org.apache.tika.parser.DefaultParser
>>>> X-Parsed-By org.apache.tika.parser.html.HtmlParser
>>>> stream_content_type text/html stream_name UsingMailingLists.html stream_source_info...etc",
>>>> _version_: 1535398408383103000 }
>>>>
>>>>
>>>>