You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ahmet Arslan <io...@yahoo.com> on 2012/08/15 01:30:09 UTC
scanned pdf with solr cell
Hi All,
I have set of rich documents. Some of them are scanned pdf files. When I send a scanned pdf to extraction request handler, below icon appears in my Dock.
http://tinypic.com/r/2mpmo7o/6
http://tinypic.com/r/28ukxhj/6
Does anyone know what this is?
curl "http://localhost:8983/solr/documents/update/extract?literal.ID=ticaret_sicil_gazetesi&literal.URL=ticaret_sicil_gazetesi&commit=true" -F "myfile=@ticaret_sicil_gazetesi.pdf"
No exception is seen on solr logs. Doc is indexed, content field is:
xmpTPg:NPages 4 Creation-Date 2011-08-24T13:03:16Z stream_source_info myfile created Wed Aug 24 16:03:16 EEST 2011 stream_content_type application/octet-stream stream_size 2302337 producer Image Recognition Integrated Systems, Autoformat5,0,0,229 stream_name ticaret_sicil_gazetesi.pdf Content-Type application/pdf creator I.R.I.S. page page page page
Environment: solr-trunk, Mac OS X Version 10.7.4, Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode), jetty.
Same thing happens with Solr 4.0-beta and Tomcat too.
Thanks,
Re: scanned pdf with solr cell
Posted by Michael Della Bitta <mi...@appinions.com>.
It's pretty easy to accidentally run into the AWT stuff if you're
doing anything that involves image processing, which I would expect a
generic RTF parser might do.
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Sun, Aug 19, 2012 at 10:29 PM, Lance Norskog <go...@gmail.com> wrote:
> The backstory here is that Tika uses a library that for some crazy
> reason is inside the Java AWG graphics toolkit. (I think the RTF
> parser?)
>
> On Wed, Aug 15, 2012 at 5:57 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>>> You can try passing
>>> -Djava.awt.headless=true as one of the arguments
>>> when you start Jetty to see if you can get this to go away
>>> with no ill
>>> effects.
>>
>> I started jetty using : 'java -Djava.awt.headless=true -jar start.jar' and successfully indexed two pdf files. That icon didn't appeared :) Thanks!
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: scanned pdf with solr cell
Posted by Lance Norskog <go...@gmail.com>.
The backstory here is that Tika uses a library that for some crazy
reason is inside the Java AWG graphics toolkit. (I think the RTF
parser?)
On Wed, Aug 15, 2012 at 5:57 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>> You can try passing
>> -Djava.awt.headless=true as one of the arguments
>> when you start Jetty to see if you can get this to go away
>> with no ill
>> effects.
>
> I started jetty using : 'java -Djava.awt.headless=true -jar start.jar' and successfully indexed two pdf files. That icon didn't appeared :) Thanks!
--
Lance Norskog
goksron@gmail.com
Re: scanned pdf with solr cell
Posted by Ahmet Arslan <io...@yahoo.com>.
> You can try passing
> -Djava.awt.headless=true as one of the arguments
> when you start Jetty to see if you can get this to go away
> with no ill
> effects.
I started jetty using : 'java -Djava.awt.headless=true -jar start.jar' and successfully indexed two pdf files. That icon didn't appeared :) Thanks!
Re: scanned pdf with solr cell
Posted by Michael Della Bitta <mi...@appinions.com>.
You can try passing -Djava.awt.headless=true as one of the arguments
when you start Jetty to see if you can get this to go away with no ill
effects.
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Wed, Aug 15, 2012 at 7:07 AM, Paul Libbrecht <pa...@hoplahup.net> wrote:
>
>
> Le 15 août 2012 à 13:03, Ahmet Arslan a écrit :
>
>> Hi Paul, thanks for the explanation. So is it nothing to worry about?
>
> it is nothing to worry about except to remember that you can't run this step in a daemon-like process.
> (on Linux, I had to set-up a VNC-server for similar tasks)
>
> paul
Re: scanned pdf with solr cell
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Le 15 août 2012 à 13:03, Ahmet Arslan a écrit :
> Hi Paul, thanks for the explanation. So is it nothing to worry about?
it is nothing to worry about except to remember that you can't run this step in a daemon-like process.
(on Linux, I had to set-up a VNC-server for similar tasks)
paul
Re: scanned pdf with solr cell
Posted by Ahmet Arslan <io...@yahoo.com>.
> the dock icon appears when AWT starts, e.g. when a font is
> loaded.
> You can prevent it using the headless mode but this is
> likely to trigger an exception.
> Same if your user is not UI-logged-in.
Hi Paul, thanks for the explanation. So is it nothing to worry about?
Re: scanned pdf with solr cell
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Ahmet,
the dock icon appears when AWT starts, e.g. when a font is loaded.
You can prevent it using the headless mode but this is likely to trigger an exception.
Same if your user is not UI-logged-in.
hope it helps.
Paul
Le 15 août 2012 à 01:30, Ahmet Arslan a écrit :
> Hi All,
>
> I have set of rich documents. Some of them are scanned pdf files. When I send a scanned pdf to extraction request handler, below icon appears in my Dock.
>
> http://tinypic.com/r/2mpmo7o/6
> http://tinypic.com/r/28ukxhj/6
>
> Does anyone know what this is?
>
> curl "http://localhost:8983/solr/documents/update/extract?literal.ID=ticaret_sicil_gazetesi&literal.URL=ticaret_sicil_gazetesi&commit=true" -F "myfile=@ticaret_sicil_gazetesi.pdf"
>
> No exception is seen on solr logs. Doc is indexed, content field is:
>
> xmpTPg:NPages 4 Creation-Date 2011-08-24T13:03:16Z stream_source_info myfile created Wed Aug 24 16:03:16 EEST 2011 stream_content_type application/octet-stream stream_size 2302337 producer Image Recognition Integrated Systems, Autoformat5,0,0,229 stream_name ticaret_sicil_gazetesi.pdf Content-Type application/pdf creator I.R.I.S. page page page page
>
> Environment: solr-trunk, Mac OS X Version 10.7.4, Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode), jetty.
>
> Same thing happens with Solr 4.0-beta and Tomcat too.
>
> Thanks,
Re: scanned pdf with solr cell
Posted by Jack Krupansky <ja...@basetechnology.com>.
Some PDFs are purely scanned documents and have only a bitmap image for each
page with no text, or sometimes they do a mediocre OCR on the page images
which produces a fair amount of garbage in the text. My recollection is that
PDFBox is parsing the PostScript for the text layer, which may be either
empty or maybe has some special/hidden text that relates to the processing
of the content but may not be the actual content (page image.)
You can usually tell if a PDF is "scanned" by zooming in and looking for
edge artifacts on curved letters, which won't appear for PostScript fonts.
Try a "normal" PDF for comparison.
-- Jack Krupansky
-----Original Message-----
From: Ahmet Arslan
Sent: Tuesday, August 14, 2012 7:30 PM
To: solr-user@lucene.apache.org
Subject: scanned pdf with solr cell
Hi All,
I have set of rich documents. Some of them are scanned pdf files. When I
send a scanned pdf to extraction request handler, below icon appears in my
Dock.
http://tinypic.com/r/2mpmo7o/6
http://tinypic.com/r/28ukxhj/6
Does anyone know what this is?
curl
"http://localhost:8983/solr/documents/update/extract?literal.ID=ticaret_sicil_gazetesi&literal.URL=ticaret_sicil_gazetesi&commit=true"
-F "myfile=@ticaret_sicil_gazetesi.pdf"
No exception is seen on solr logs. Doc is indexed, content field is:
xmpTPg:NPages 4 Creation-Date 2011-08-24T13:03:16Z stream_source_info
myfile created Wed Aug 24 16:03:16 EEST 2011 stream_content_type
application/octet-stream stream_size 2302337 producer Image Recognition
Integrated Systems, Autoformat5,0,0,229 stream_name
ticaret_sicil_gazetesi.pdf Content-Type application/pdf creator I.R.I.S.
page page page page
Environment: solr-trunk, Mac OS X Version 10.7.4, Java HotSpot(TM) 64-Bit
Server VM (build 20.8-b03-424, mixed mode), jetty.
Same thing happens with Solr 4.0-beta and Tomcat too.
Thanks,
Re: scanned pdf with solr cell
Posted by Ahmet Arslan <io...@yahoo.com>.
> When I send a scanned pdf to extraction request
> handler, below icon appears in my Dock.
>
> http://tinypic.com/r/2mpmo7o/6
> http://tinypic.com/r/28ukxhj/6
I found that text-extractable pdf files triggers above weird icon too.
curl "http://localhost:8983/solr/update/extract?literal.id=solr-word&commit=true" -F "myfile=@solr-word.pdf"
I wrote a standalone java program using tika. When text-extracting from all kinds of pdf files, that weird icon pops-up :)
I will ask tika-ML about this.
AutoDetectParser _autoParser = new AutoDetectParser();
File file = new File("solr-word.pdf");
BodyContentHandler textHandler = new BodyContentHandler();
Metadata metadata = new Metadata();
ParseContext context = new ParseContext();
InputStream input = new FileInputStream(file);
_autoParser.parse(input, textHandler, metadata, context);
System.out.println("text : " + textHandler.toString());
input.close();
while (true) { }