You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stefan Grroschupf (JIRA)" <ji...@apache.org> on 2005/03/26 15:44:20 UTC

[jira] Created: (NUTCH-21) parser plugin for MS PowerPoint slides

parser plugin for MS PowerPoint slides
--------------------------------------

         Key: NUTCH-21
         URL: http://issues.apache.org/jira/browse/NUTCH-21
     Project: Nutch
        Type: Improvement
  Components: fetcher  
    Reporter: Stefan Grroschupf
    Priority: Trivial


transfered from:
http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
submitted by:
Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "David Spencer (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_64531 ]
     
David Spencer commented on NUTCH-21:
------------------------------------

I figured out why I was getting docs with zero body.
Here's a stack trace, but note that I changed the package to com.tropo.ppt for my uses..

java.lang.StringIndexOutOfBoundsException: String index out of range: 1756156169
	at java.lang.String.checkBounds(String.java:287)
	at java.lang.String.<init>(String.java:370)
	at com.tropo.ppt.ContentReaderListener.extractSlides(ContentReaderListener.java:353)
	at com.tropo.ppt.ContentReaderListener.processPOIFSReaderEvent(ContentReaderListener.java:121)
	at org.apache.poi.poifs.eventfilesystem.POIFSReader.processProperties(POIFSReader.java:259)
	at org.apache.poi.poifs.eventfilesystem.POIFSReader.read(POIFSReader.java:95)
	at com.tropo.ppt.PPT2Text.init(PPT2Text.java:92)
	at com.tropo.ppt.PPT2Text.getProperties(PPT2Text.java:132)
	at com.tropo.ppt.PPT2Text.main(PPT2Text.java:144)

---
What's happening is in ContentReaderListener.java.

In processPOIFSReaderEvent() there's an empty catch(Throwable) block that hides the error.

In extractSlides() it happily goes thru some data but then, for some reason, 'size' is larger than pptdata.length.

One hack to "fix" this is to replace this:

final String strTempContent = new String(pptdata, (int) i + 6,
            (int) (size) + 2);

String strTempContent;

try {
    strTempContent = new String(pptdata, (int) i + 6, (int) (size) + 2);
}
catch( StringIndexOutOfBoundsException ouch) {
    strTempContent = "";
}

When I do this I get data out of a ppt file that previously seemed to to have a zero length body...





> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Renat Lumpau (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_12320793 ] 

Renat Lumpau commented on NUTCH-21:
-----------------------------------

I had to hack MSPowerPointParser.java to get this working with nutch-0.7. I've attached the modified file. 

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>  Attachments: MSPowerPointParser.java, build.xml.patch.txt, parse-mspowerpoint.zip, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Aktualisiert: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=all ]

Stephan Strittmatter updated NUTCH-21:
--------------------------------------

    Attachment: parse-mspowerpoint.zip

Updated plugin sources in respect of changed Nutch interface

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Kommentiert: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_63039 ]
     
Stephan Strittmatter commented on NUTCH-21:
-------------------------------------------

Was someone able to use this plugin successfully? 
Shall I provide a compiled version also?

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Kommentiert: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_12320763 ] 

Stephan Strittmatter commented on NUTCH-21:
-------------------------------------------

I will verify the Unit-Tests until next week!

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Kommentiert: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_12313578 ] 

Stephan Strittmatter commented on NUTCH-21:
-------------------------------------------

Could someone send me a ppt-file which produces such errors for debugging?
I tested several files but I was not abel to reproduce them.

BTW: The exception is not completly hidden it is logged.

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_12320717 ] 

Jerome Charron commented on NUTCH-21:
-------------------------------------

Want to commit it, but unit tests failed.

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Updated: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=history ]

Stephan Strittmatter updated NUTCH-21:
--------------------------------------

    Attachment: parse-mspowerpoint.zip
                build.xml.patch.txt

Atached you can find the complete PowerPoint parser.
Also included in the zip are:
* JUnit test with one sample (the protocol-file plugin is required to run this)
  It is a very detailed test, which could check on char basis the result!
  I think It would be a good idea to extract this as a small test 
  environment for all parsers to get the most useful parsing results.
* The required POI jars are also included.

The build.xml file of the plugin directory has to be updated for this additional plugin. For tis I attached the build.xml.patch.txt

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_61584 ]
     
Stephan Strittmatter commented on NUTCH-21:
-------------------------------------------

I am refactoring it at the moment to provide a org.apache.nutch conform patch.

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial

>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
   http://www.atlassian.com/software/jira


[jira] Closed: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=all ]
     
Jerome Charron closed NUTCH-21:
-------------------------------

    Fix Version: 0.8-dev
     Resolution: Fixed

Commited to trunk (http://svn.apache.org/viewcvs.cgi?rev=267226&view=rev)
Thanks to Stephan Strittmatter.

Note: Take care of the patches attached to this issue since the unit tests are platform dependent (only successed on windows). The committed code is platform independent (I hope). I tested it on Linux, so if someone can test it on other platforms it would be a good idea.


> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>      Fix For: 0.8-dev
>  Attachments: MSPowerPointParser.java, build.xml.patch.txt, parse-mspowerpoint.zip, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Commented: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "David Spencer (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=comments#action_64520 ]
     
David Spencer commented on NUTCH-21:
------------------------------------

This may be of some use:

I needed a PPT parser in the context of Lucene, so I copied the code from here, commented out a few nutch-specific things (e.g. the logging calls), and tested it on some local PPT files. I'm using POI-2.5.1-final.

The code is not perfect, nor is the PPT I have :) but it's pretty good.
When it works it works well.
Went it fails it sometimes says there is no content, but in the doc I spot checked there seemed to be textual content. I have only spot checked a few docs but I did run it thru my disk:

In a test run:
[a] I had 195 PPT files
[b] In 36 files it said there was no body
[c] With one file it thru an exception
[d] With 158 files it found content

Wrt [b] this is not necessarily wrong e.g. if there are only images, however in the 1 file I spot checked there was apparently textual content.

Wrt [d], I didn't spot check many files but the ones I did seemed fine.

Personally I would advocate using this esp if someone verifies this within nutch - but I'm confident it will work as I didn't change much to use it in Lucene.



This was the "bug" that happened in 1 file
Caused by: java.io.IOException: Cannot remove block[ 18805 ]; out of range
	at org.apache.poi.poifs.storage.BlockListImpl.remove(BlockListImpl.java:103)
	at org.apache.poi.poifs.storage.BlockAllocationTableReader.<init>(BlockAllocationTableReader.java:92)
	at org.apache.poi.poifs.eventfilesystem.POIFSReader.read(POIFSReader.java:83)
	at com.tropo.ppt.PPT2Text.init(PPT2Text.java:92)



-- Dave







> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Grroschupf
>     Priority: Trivial
>  Attachments: build.xml.patch.txt, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


[jira] Updated: (NUTCH-21) parser plugin for MS PowerPoint slides

Posted by "Renat Lumpau (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/NUTCH-21?page=all ]

Renat Lumpau updated NUTCH-21:
------------------------------

    Attachment: MSPowerPointParser.java

> parser plugin for MS PowerPoint slides
> --------------------------------------
>
>          Key: NUTCH-21
>          URL: http://issues.apache.org/jira/browse/NUTCH-21
>      Project: Nutch
>         Type: Improvement
>   Components: fetcher
>     Reporter: Stefan Groschupf
>     Priority: Trivial
>  Attachments: MSPowerPointParser.java, build.xml.patch.txt, parse-mspowerpoint.zip, parse-mspowerpoint.zip
>
> transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1109321&group_id=59548&atid=491356
> submitted by:
> Stephan Strittmatter

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira