You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by kiran chitturi <ch...@gmail.com> on 2012/11/01 14:48:32 UTC

Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails

Thank you alxsss for the suggestion. It displays the actualSize and
inHeaderSize for every file and two more lines in logs but it did not much
information even when i set parserJob to Debug.

I had the same problem when i re-compiled everything today. I have to run
the parse command multiple times to get all the files parsed.

I am using SQL with GORA. Its mysql database.

For now, atleast the files are getting parsed, do  i need to open a issue
for this ?

Thank you,

Regards,
Kiran.


On Wed, Oct 31, 2012 at 4:36 PM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:

> Hi Kiran
>
> Interesting. Which backend are you using with GORA? The SQL one? Could be a
> problem at that level
>
> Julien
>
> On 31 October 2012 17:01, kiran chitturi <ch...@gmail.com>
> wrote:
>
> > Hi Julien,
> >
> > I have just noticed something when running the parse.
> >
> > First when i ran the parse command 'sh bin/nutch parse
> > 1351188762-1772522488', the parsing of all the PDF files has failed.
> >
> > When i ran the command again one pdf file got parsed. Next time, another
> > pdf file got parsed.
> >
> > When i ran the parse command the number of times the total number of pdf
> > files, all the pdf files got parsed.
> >
> > In my case,  i ran it 17 times and all the pdf files are parsed. Before
> > that, not everything is parsed.
> >
> > This sounds strange, do you think it is some configuration problem ?
> >
> > I have tried this 2 times and same thing happened two times for me .
> >
> > I am not sure why this is happening.
> >
> > Thanks for your help.
> >
> > Regards,
> > Kiran.
> >
> >
> > On Wed, Oct 31, 2012 at 10:28 AM, Julien Nioche <
> > lists.digitalpebble@gmail.com> wrote:
> >
> > > Hi
> > >
> > >
> > > > Sorry about that. I did not notice the parsecodes are actually nutch
> > and
> > > > not tika.
> > > >
> > > > no problems!
> > >
> > >
> > > > The setup is local on Mac desktop and i am using through command line
> > and
> > > > remote debugging through eclipse (
> > > >
> > >
> >
> http://wiki.apache.org/nutch/RunNutchInEclipse#Remote_Debugging_in_Eclipse
> > > > ).
> > > >
> > >
> > > OK
> > >
> > > >
> > > > I have set both http.content.limit and file.content.limit to -1. The
> > logs
> > > > just say 'WARN  parse.ParseUtil - Unable to successfully parse
> content
> > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/pdf/watson.pdf of
> type
> > > > application/pdf'.
> > > >
> > >
> > > you set it in $NUTCH_HOME/runtime/local/conf/nutch-site.xml right? (not
> > > in $NUTCH_HOME/conf/nutch-site.xml unless you call 'ant clean runtime')
> > >
> > >
> > > >
> > > > All the html's are getting parsed and when i crawl this page (
> > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/), all the html's and
> > > some
> > > > of the pdf files get parsed. Like, half of the pdf files get parsed
> and
> > > the
> > > > other half don't get parsed.
> > > >
> > >
> > > do the ones that are not parsed have something in common? length?
> > >
> > >
> > > > I am not sure about what causing the problem as you said parsechecker
> > is
> > > > actually work. I want the parser to crawl the full-text of the pdf
> and
> > > the
> > > > metadata, title.
> > > >
> > >
> > > OK
> > >
> > >
> > > >
> > > > The metatags are also getting crawled for failed pdf parsing.
> > > >
> > >
> > > They would be discarded because of the failure even if they
> > > were successfully extracted indeed. The current mechanism does not
> cater
> > > for semi-failures
> > >
> > > J.
> > >
> > > --
> > > *
> > > *Open Source Solutions for Text Engineering
> > >
> > > http://digitalpebble.blogspot.com/
> > > http://www.digitalpebble.com
> > > http://twitter.com/digitalpebble
> > >
> >
> >
> >
> > --
> > Kiran Chitturi
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>



-- 
Kiran Chitturi

Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails

Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Kiran,

Thanks for the persistence on this one, it is greatly appreciated.

Please feel free to open an issue... may be best to even open over on
the GORA jira?

Best

Lewis

On Mon, Nov 5, 2012 at 2:50 PM, kiran chitturi
<ch...@gmail.com> wrote:
> I have just tested with Hbase as gora-backend and all the pdf files are
> parsed in the first attempt. This is an sql backend problem as Julien has
> noted before.
>
> Thanks,
> Kiran.
>
>
> On Thu, Nov 1, 2012 at 1:49 PM, <al...@aim.com> wrote:
>
>> Hi,
>>
>> I think in order to be sure that this is gora-sql problem, you need to do
>> the same crawling with nutch/hbase. It must not take much time if you run
>> it in local mode. Simply install hbase and follow quick start tutorial.
>>
>> Alex.
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: kiran chitturi <ch...@gmail.com>
>> To: user <us...@nutch.apache.org>
>> Sent: Thu, Nov 1, 2012 9:29 am
>> Subject: Re: bin/nutch parsechecker -dumpText works but bin/nutch parse
>> fails
>>
>>
>> Hi,
>>
>> I have created an issue (https://issues.apache.org/jira/browse/NUTCH-1487
>> ).
>>
>> Do you think this is because of the SQL backend ? Its failing for PDF files
>> but working for HTML files.
>>
>> Can the problem be due to some bug in the tika.parser code (since tika
>> plugin handles the PDF parsing) ?
>>
>> I am interesting in fixing this problem, if i can find out where the issue
>> starts.
>>
>> Does anyone have inputs for this ?
>>
>> Thanks,
>> Kiran.
>>
>>
>>
>> On Thu, Nov 1, 2012 at 10:15 AM, Julien Nioche <
>> lists.digitalpebble@gmail.com> wrote:
>>
>> > Hi
>> >
>> > Yes please do open an issue. The docs should be parsed in one go and I
>> > suspect (yet another) issue with the SQL backend
>> >
>> > Thanks
>> >
>> > J
>> >
>> > On 1 November 2012 13:48, kiran chitturi <ch...@gmail.com>
>> > wrote:
>> >
>> > > Thank you alxsss for the suggestion. It displays the actualSize and
>> > > inHeaderSize for every file and two more lines in logs but it did not
>> > much
>> > > information even when i set parserJob to Debug.
>> > >
>> > > I had the same problem when i re-compiled everything today. I have to
>> run
>> > > the parse command multiple times to get all the files parsed.
>> > >
>> > > I am using SQL with GORA. Its mysql database.
>> > >
>> > > For now, atleast the files are getting parsed, do  i need to open a
>> issue
>> > > for this ?
>> > >
>> > > Thank you,
>> > >
>> > > Regards,
>> > > Kiran.
>> > >
>> > >
>> > > On Wed, Oct 31, 2012 at 4:36 PM, Julien Nioche <
>> > > lists.digitalpebble@gmail.com> wrote:
>> > >
>> > > > Hi Kiran
>> > > >
>> > > > Interesting. Which backend are you using with GORA? The SQL one?
>> Could
>> > > be a
>> > > > problem at that level
>> > > >
>> > > > Julien
>> > > >
>> > > > On 31 October 2012 17:01, kiran chitturi <ch...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Julien,
>> > > > >
>> > > > > I have just noticed something when running the parse.
>> > > > >
>> > > > > First when i ran the parse command 'sh bin/nutch parse
>> > > > > 1351188762-1772522488', the parsing of all the PDF files has
>> failed.
>> > > > >
>> > > > > When i ran the command again one pdf file got parsed. Next time,
>> > > another
>> > > > > pdf file got parsed.
>> > > > >
>> > > > > When i ran the parse command the number of times the total number
>> of
>> > > pdf
>> > > > > files, all the pdf files got parsed.
>> > > > >
>> > > > > In my case,  i ran it 17 times and all the pdf files are parsed.
>> > Before
>> > > > > that, not everything is parsed.
>> > > > >
>> > > > > This sounds strange, do you think it is some configuration problem
>> ?
>> > > > >
>> > > > > I have tried this 2 times and same thing happened two times for me
>> .
>> > > > >
>> > > > > I am not sure why this is happening.
>> > > > >
>> > > > > Thanks for your help.
>> > > > >
>> > > > > Regards,
>> > > > > Kiran.
>> > > > >
>> > > > >
>> > > > > On Wed, Oct 31, 2012 at 10:28 AM, Julien Nioche <
>> > > > > lists.digitalpebble@gmail.com> wrote:
>> > > > >
>> > > > > > Hi
>> > > > > >
>> > > > > >
>> > > > > > > Sorry about that. I did not notice the parsecodes are actually
>> > > nutch
>> > > > > and
>> > > > > > > not tika.
>> > > > > > >
>> > > > > > > no problems!
>> > > > > >
>> > > > > >
>> > > > > > > The setup is local on Mac desktop and i am using through
>> command
>> > > line
>> > > > > and
>> > > > > > > remote debugging through eclipse (
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://wiki.apache.org/nutch/RunNutchInEclipse#Remote_Debugging_in_Eclipse
>> > > > > > > ).
>> > > > > > >
>> > > > > >
>> > > > > > OK
>> > > > > >
>> > > > > > >
>> > > > > > > I have set both http.content.limit and file.content.limit to
>> -1.
>> > > The
>> > > > > logs
>> > > > > > > just say 'WARN  parse.ParseUtil - Unable to successfully parse
>> > > > content
>> > > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/pdf/watson.pdfof
>> > > > type
>> > > > > > > application/pdf'.
>> > > > > > >
>> > > > > >
>> > > > > > you set it in $NUTCH_HOME/runtime/local/conf/nutch-site.xml
>> right?
>> > > (not
>> > > > > > in $NUTCH_HOME/conf/nutch-site.xml unless you call 'ant clean
>> > > runtime')
>> > > > > >
>> > > > > >
>> > > > > > >
>> > > > > > > All the html's are getting parsed and when i crawl this page (
>> > > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/), all the
>> html's
>> > > and
>> > > > > > some
>> > > > > > > of the pdf files get parsed. Like, half of the pdf files get
>> > parsed
>> > > > and
>> > > > > > the
>> > > > > > > other half don't get parsed.
>> > > > > > >
>> > > > > >
>> > > > > > do the ones that are not parsed have something in common? length?
>> > > > > >
>> > > > > >
>> > > > > > > I am not sure about what causing the problem as you said
>> > > parsechecker
>> > > > > is
>> > > > > > > actually work. I want the parser to crawl the full-text of the
>> > pdf
>> > > > and
>> > > > > > the
>> > > > > > > metadata, title.
>> > > > > > >
>> > > > > >
>> > > > > > OK
>> > > > > >
>> > > > > >
>> > > > > > >
>> > > > > > > The metatags are also getting crawled for failed pdf parsing.
>> > > > > > >
>> > > > > >
>> > > > > > They would be discarded because of the failure even if they
>> > > > > > were successfully extracted indeed. The current mechanism does
>> not
>> > > > cater
>> > > > > > for semi-failures
>> > > > > >
>> > > > > > J.
>> > > > > >
>> > > > > > --
>> > > > > > *
>> > > > > > *Open Source Solutions for Text Engineering
>> > > > > >
>> > > > > > http://digitalpebble.blogspot.com/
>> > > > > > http://www.digitalpebble.com
>> > > > > > http://twitter.com/digitalpebble
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Kiran Chitturi
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > *
>> > > > *Open Source Solutions for Text Engineering
>> > > >
>> > > > http://digitalpebble.blogspot.com/
>> > > > http://www.digitalpebble.com
>> > > > http://twitter.com/digitalpebble
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kiran Chitturi
>> > >
>> >
>> >
>> >
>> > --
>> > *
>> > *Open Source Solutions for Text Engineering
>> >
>> > http://digitalpebble.blogspot.com/
>> > http://www.digitalpebble.com
>> > http://twitter.com/digitalpebble
>> >
>>
>>
>>
>> --
>> Kiran Chitturi
>>
>>
>>
>
>
> --
> Kiran Chitturi



-- 
Lewis

Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails

Posted by kiran chitturi <ch...@gmail.com>.
I have just tested with Hbase as gora-backend and all the pdf files are
parsed in the first attempt. This is an sql backend problem as Julien has
noted before.

Thanks,
Kiran.


On Thu, Nov 1, 2012 at 1:49 PM, <al...@aim.com> wrote:

> Hi,
>
> I think in order to be sure that this is gora-sql problem, you need to do
> the same crawling with nutch/hbase. It must not take much time if you run
> it in local mode. Simply install hbase and follow quick start tutorial.
>
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: kiran chitturi <ch...@gmail.com>
> To: user <us...@nutch.apache.org>
> Sent: Thu, Nov 1, 2012 9:29 am
> Subject: Re: bin/nutch parsechecker -dumpText works but bin/nutch parse
> fails
>
>
> Hi,
>
> I have created an issue (https://issues.apache.org/jira/browse/NUTCH-1487
> ).
>
> Do you think this is because of the SQL backend ? Its failing for PDF files
> but working for HTML files.
>
> Can the problem be due to some bug in the tika.parser code (since tika
> plugin handles the PDF parsing) ?
>
> I am interesting in fixing this problem, if i can find out where the issue
> starts.
>
> Does anyone have inputs for this ?
>
> Thanks,
> Kiran.
>
>
>
> On Thu, Nov 1, 2012 at 10:15 AM, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
> > Hi
> >
> > Yes please do open an issue. The docs should be parsed in one go and I
> > suspect (yet another) issue with the SQL backend
> >
> > Thanks
> >
> > J
> >
> > On 1 November 2012 13:48, kiran chitturi <ch...@gmail.com>
> > wrote:
> >
> > > Thank you alxsss for the suggestion. It displays the actualSize and
> > > inHeaderSize for every file and two more lines in logs but it did not
> > much
> > > information even when i set parserJob to Debug.
> > >
> > > I had the same problem when i re-compiled everything today. I have to
> run
> > > the parse command multiple times to get all the files parsed.
> > >
> > > I am using SQL with GORA. Its mysql database.
> > >
> > > For now, atleast the files are getting parsed, do  i need to open a
> issue
> > > for this ?
> > >
> > > Thank you,
> > >
> > > Regards,
> > > Kiran.
> > >
> > >
> > > On Wed, Oct 31, 2012 at 4:36 PM, Julien Nioche <
> > > lists.digitalpebble@gmail.com> wrote:
> > >
> > > > Hi Kiran
> > > >
> > > > Interesting. Which backend are you using with GORA? The SQL one?
> Could
> > > be a
> > > > problem at that level
> > > >
> > > > Julien
> > > >
> > > > On 31 October 2012 17:01, kiran chitturi <ch...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Julien,
> > > > >
> > > > > I have just noticed something when running the parse.
> > > > >
> > > > > First when i ran the parse command 'sh bin/nutch parse
> > > > > 1351188762-1772522488', the parsing of all the PDF files has
> failed.
> > > > >
> > > > > When i ran the command again one pdf file got parsed. Next time,
> > > another
> > > > > pdf file got parsed.
> > > > >
> > > > > When i ran the parse command the number of times the total number
> of
> > > pdf
> > > > > files, all the pdf files got parsed.
> > > > >
> > > > > In my case,  i ran it 17 times and all the pdf files are parsed.
> > Before
> > > > > that, not everything is parsed.
> > > > >
> > > > > This sounds strange, do you think it is some configuration problem
> ?
> > > > >
> > > > > I have tried this 2 times and same thing happened two times for me
> .
> > > > >
> > > > > I am not sure why this is happening.
> > > > >
> > > > > Thanks for your help.
> > > > >
> > > > > Regards,
> > > > > Kiran.
> > > > >
> > > > >
> > > > > On Wed, Oct 31, 2012 at 10:28 AM, Julien Nioche <
> > > > > lists.digitalpebble@gmail.com> wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > >
> > > > > > > Sorry about that. I did not notice the parsecodes are actually
> > > nutch
> > > > > and
> > > > > > > not tika.
> > > > > > >
> > > > > > > no problems!
> > > > > >
> > > > > >
> > > > > > > The setup is local on Mac desktop and i am using through
> command
> > > line
> > > > > and
> > > > > > > remote debugging through eclipse (
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://wiki.apache.org/nutch/RunNutchInEclipse#Remote_Debugging_in_Eclipse
> > > > > > > ).
> > > > > > >
> > > > > >
> > > > > > OK
> > > > > >
> > > > > > >
> > > > > > > I have set both http.content.limit and file.content.limit to
> -1.
> > > The
> > > > > logs
> > > > > > > just say 'WARN  parse.ParseUtil - Unable to successfully parse
> > > > content
> > > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/pdf/watson.pdfof
> > > > type
> > > > > > > application/pdf'.
> > > > > > >
> > > > > >
> > > > > > you set it in $NUTCH_HOME/runtime/local/conf/nutch-site.xml
> right?
> > > (not
> > > > > > in $NUTCH_HOME/conf/nutch-site.xml unless you call 'ant clean
> > > runtime')
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > All the html's are getting parsed and when i crawl this page (
> > > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/), all the
> html's
> > > and
> > > > > > some
> > > > > > > of the pdf files get parsed. Like, half of the pdf files get
> > parsed
> > > > and
> > > > > > the
> > > > > > > other half don't get parsed.
> > > > > > >
> > > > > >
> > > > > > do the ones that are not parsed have something in common? length?
> > > > > >
> > > > > >
> > > > > > > I am not sure about what causing the problem as you said
> > > parsechecker
> > > > > is
> > > > > > > actually work. I want the parser to crawl the full-text of the
> > pdf
> > > > and
> > > > > > the
> > > > > > > metadata, title.
> > > > > > >
> > > > > >
> > > > > > OK
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > The metatags are also getting crawled for failed pdf parsing.
> > > > > > >
> > > > > >
> > > > > > They would be discarded because of the failure even if they
> > > > > > were successfully extracted indeed. The current mechanism does
> not
> > > > cater
> > > > > > for semi-failures
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > --
> > > > > > *
> > > > > > *Open Source Solutions for Text Engineering
> > > > > >
> > > > > > http://digitalpebble.blogspot.com/
> > > > > > http://www.digitalpebble.com
> > > > > > http://twitter.com/digitalpebble
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kiran Chitturi
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *
> > > > *Open Source Solutions for Text Engineering
> > > >
> > > > http://digitalpebble.blogspot.com/
> > > > http://www.digitalpebble.com
> > > > http://twitter.com/digitalpebble
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Chitturi
> > >
> >
> >
> >
> > --
> > *
> > *Open Source Solutions for Text Engineering
> >
> > http://digitalpebble.blogspot.com/
> > http://www.digitalpebble.com
> > http://twitter.com/digitalpebble
> >
>
>
>
> --
> Kiran Chitturi
>
>
>


-- 
Kiran Chitturi

Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails

Posted by al...@aim.com.
Hi,

I think in order to be sure that this is gora-sql problem, you need to do the same crawling with nutch/hbase. It must not take much time if you run it in local mode. Simply install hbase and follow quick start tutorial.

Alex.

 

 

 

-----Original Message-----
From: kiran chitturi <ch...@gmail.com>
To: user <us...@nutch.apache.org>
Sent: Thu, Nov 1, 2012 9:29 am
Subject: Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails


Hi,

I have created an issue (https://issues.apache.org/jira/browse/NUTCH-1487).

Do you think this is because of the SQL backend ? Its failing for PDF files
but working for HTML files.

Can the problem be due to some bug in the tika.parser code (since tika
plugin handles the PDF parsing) ?

I am interesting in fixing this problem, if i can find out where the issue
starts.

Does anyone have inputs for this ?

Thanks,
Kiran.



On Thu, Nov 1, 2012 at 10:15 AM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:

> Hi
>
> Yes please do open an issue. The docs should be parsed in one go and I
> suspect (yet another) issue with the SQL backend
>
> Thanks
>
> J
>
> On 1 November 2012 13:48, kiran chitturi <ch...@gmail.com>
> wrote:
>
> > Thank you alxsss for the suggestion. It displays the actualSize and
> > inHeaderSize for every file and two more lines in logs but it did not
> much
> > information even when i set parserJob to Debug.
> >
> > I had the same problem when i re-compiled everything today. I have to run
> > the parse command multiple times to get all the files parsed.
> >
> > I am using SQL with GORA. Its mysql database.
> >
> > For now, atleast the files are getting parsed, do  i need to open a issue
> > for this ?
> >
> > Thank you,
> >
> > Regards,
> > Kiran.
> >
> >
> > On Wed, Oct 31, 2012 at 4:36 PM, Julien Nioche <
> > lists.digitalpebble@gmail.com> wrote:
> >
> > > Hi Kiran
> > >
> > > Interesting. Which backend are you using with GORA? The SQL one? Could
> > be a
> > > problem at that level
> > >
> > > Julien
> > >
> > > On 31 October 2012 17:01, kiran chitturi <ch...@gmail.com>
> > > wrote:
> > >
> > > > Hi Julien,
> > > >
> > > > I have just noticed something when running the parse.
> > > >
> > > > First when i ran the parse command 'sh bin/nutch parse
> > > > 1351188762-1772522488', the parsing of all the PDF files has failed.
> > > >
> > > > When i ran the command again one pdf file got parsed. Next time,
> > another
> > > > pdf file got parsed.
> > > >
> > > > When i ran the parse command the number of times the total number of
> > pdf
> > > > files, all the pdf files got parsed.
> > > >
> > > > In my case,  i ran it 17 times and all the pdf files are parsed.
> Before
> > > > that, not everything is parsed.
> > > >
> > > > This sounds strange, do you think it is some configuration problem ?
> > > >
> > > > I have tried this 2 times and same thing happened two times for me .
> > > >
> > > > I am not sure why this is happening.
> > > >
> > > > Thanks for your help.
> > > >
> > > > Regards,
> > > > Kiran.
> > > >
> > > >
> > > > On Wed, Oct 31, 2012 at 10:28 AM, Julien Nioche <
> > > > lists.digitalpebble@gmail.com> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > >
> > > > > > Sorry about that. I did not notice the parsecodes are actually
> > nutch
> > > > and
> > > > > > not tika.
> > > > > >
> > > > > > no problems!
> > > > >
> > > > >
> > > > > > The setup is local on Mac desktop and i am using through command
> > line
> > > > and
> > > > > > remote debugging through eclipse (
> > > > > >
> > > > >
> > > >
> > >
> >
> http://wiki.apache.org/nutch/RunNutchInEclipse#Remote_Debugging_in_Eclipse
> > > > > > ).
> > > > > >
> > > > >
> > > > > OK
> > > > >
> > > > > >
> > > > > > I have set both http.content.limit and file.content.limit to -1.
> > The
> > > > logs
> > > > > > just say 'WARN  parse.ParseUtil - Unable to successfully parse
> > > content
> > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/pdf/watson.pdf of
> > > type
> > > > > > application/pdf'.
> > > > > >
> > > > >
> > > > > you set it in $NUTCH_HOME/runtime/local/conf/nutch-site.xml right?
> > (not
> > > > > in $NUTCH_HOME/conf/nutch-site.xml unless you call 'ant clean
> > runtime')
> > > > >
> > > > >
> > > > > >
> > > > > > All the html's are getting parsed and when i crawl this page (
> > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/), all the html's
> > and
> > > > > some
> > > > > > of the pdf files get parsed. Like, half of the pdf files get
> parsed
> > > and
> > > > > the
> > > > > > other half don't get parsed.
> > > > > >
> > > > >
> > > > > do the ones that are not parsed have something in common? length?
> > > > >
> > > > >
> > > > > > I am not sure about what causing the problem as you said
> > parsechecker
> > > > is
> > > > > > actually work. I want the parser to crawl the full-text of the
> pdf
> > > and
> > > > > the
> > > > > > metadata, title.
> > > > > >
> > > > >
> > > > > OK
> > > > >
> > > > >
> > > > > >
> > > > > > The metatags are also getting crawled for failed pdf parsing.
> > > > > >
> > > > >
> > > > > They would be discarded because of the failure even if they
> > > > > were successfully extracted indeed. The current mechanism does not
> > > cater
> > > > > for semi-failures
> > > > >
> > > > > J.
> > > > >
> > > > > --
> > > > > *
> > > > > *Open Source Solutions for Text Engineering
> > > > >
> > > > > http://digitalpebble.blogspot.com/
> > > > > http://www.digitalpebble.com
> > > > > http://twitter.com/digitalpebble
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kiran Chitturi
> > > >
> > >
> > >
> > >
> > > --
> > > *
> > > *Open Source Solutions for Text Engineering
> > >
> > > http://digitalpebble.blogspot.com/
> > > http://www.digitalpebble.com
> > > http://twitter.com/digitalpebble
> > >
> >
> >
> >
> > --
> > Kiran Chitturi
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>



-- 
Kiran Chitturi

 

Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails

Posted by kiran chitturi <ch...@gmail.com>.
Hi,

I have created an issue (https://issues.apache.org/jira/browse/NUTCH-1487).

Do you think this is because of the SQL backend ? Its failing for PDF files
but working for HTML files.

Can the problem be due to some bug in the tika.parser code (since tika
plugin handles the PDF parsing) ?

I am interesting in fixing this problem, if i can find out where the issue
starts.

Does anyone have inputs for this ?

Thanks,
Kiran.



On Thu, Nov 1, 2012 at 10:15 AM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:

> Hi
>
> Yes please do open an issue. The docs should be parsed in one go and I
> suspect (yet another) issue with the SQL backend
>
> Thanks
>
> J
>
> On 1 November 2012 13:48, kiran chitturi <ch...@gmail.com>
> wrote:
>
> > Thank you alxsss for the suggestion. It displays the actualSize and
> > inHeaderSize for every file and two more lines in logs but it did not
> much
> > information even when i set parserJob to Debug.
> >
> > I had the same problem when i re-compiled everything today. I have to run
> > the parse command multiple times to get all the files parsed.
> >
> > I am using SQL with GORA. Its mysql database.
> >
> > For now, atleast the files are getting parsed, do  i need to open a issue
> > for this ?
> >
> > Thank you,
> >
> > Regards,
> > Kiran.
> >
> >
> > On Wed, Oct 31, 2012 at 4:36 PM, Julien Nioche <
> > lists.digitalpebble@gmail.com> wrote:
> >
> > > Hi Kiran
> > >
> > > Interesting. Which backend are you using with GORA? The SQL one? Could
> > be a
> > > problem at that level
> > >
> > > Julien
> > >
> > > On 31 October 2012 17:01, kiran chitturi <ch...@gmail.com>
> > > wrote:
> > >
> > > > Hi Julien,
> > > >
> > > > I have just noticed something when running the parse.
> > > >
> > > > First when i ran the parse command 'sh bin/nutch parse
> > > > 1351188762-1772522488', the parsing of all the PDF files has failed.
> > > >
> > > > When i ran the command again one pdf file got parsed. Next time,
> > another
> > > > pdf file got parsed.
> > > >
> > > > When i ran the parse command the number of times the total number of
> > pdf
> > > > files, all the pdf files got parsed.
> > > >
> > > > In my case,  i ran it 17 times and all the pdf files are parsed.
> Before
> > > > that, not everything is parsed.
> > > >
> > > > This sounds strange, do you think it is some configuration problem ?
> > > >
> > > > I have tried this 2 times and same thing happened two times for me .
> > > >
> > > > I am not sure why this is happening.
> > > >
> > > > Thanks for your help.
> > > >
> > > > Regards,
> > > > Kiran.
> > > >
> > > >
> > > > On Wed, Oct 31, 2012 at 10:28 AM, Julien Nioche <
> > > > lists.digitalpebble@gmail.com> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > >
> > > > > > Sorry about that. I did not notice the parsecodes are actually
> > nutch
> > > > and
> > > > > > not tika.
> > > > > >
> > > > > > no problems!
> > > > >
> > > > >
> > > > > > The setup is local on Mac desktop and i am using through command
> > line
> > > > and
> > > > > > remote debugging through eclipse (
> > > > > >
> > > > >
> > > >
> > >
> >
> http://wiki.apache.org/nutch/RunNutchInEclipse#Remote_Debugging_in_Eclipse
> > > > > > ).
> > > > > >
> > > > >
> > > > > OK
> > > > >
> > > > > >
> > > > > > I have set both http.content.limit and file.content.limit to -1.
> > The
> > > > logs
> > > > > > just say 'WARN  parse.ParseUtil - Unable to successfully parse
> > > content
> > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/pdf/watson.pdf of
> > > type
> > > > > > application/pdf'.
> > > > > >
> > > > >
> > > > > you set it in $NUTCH_HOME/runtime/local/conf/nutch-site.xml right?
> > (not
> > > > > in $NUTCH_HOME/conf/nutch-site.xml unless you call 'ant clean
> > runtime')
> > > > >
> > > > >
> > > > > >
> > > > > > All the html's are getting parsed and when i crawl this page (
> > > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/), all the html's
> > and
> > > > > some
> > > > > > of the pdf files get parsed. Like, half of the pdf files get
> parsed
> > > and
> > > > > the
> > > > > > other half don't get parsed.
> > > > > >
> > > > >
> > > > > do the ones that are not parsed have something in common? length?
> > > > >
> > > > >
> > > > > > I am not sure about what causing the problem as you said
> > parsechecker
> > > > is
> > > > > > actually work. I want the parser to crawl the full-text of the
> pdf
> > > and
> > > > > the
> > > > > > metadata, title.
> > > > > >
> > > > >
> > > > > OK
> > > > >
> > > > >
> > > > > >
> > > > > > The metatags are also getting crawled for failed pdf parsing.
> > > > > >
> > > > >
> > > > > They would be discarded because of the failure even if they
> > > > > were successfully extracted indeed. The current mechanism does not
> > > cater
> > > > > for semi-failures
> > > > >
> > > > > J.
> > > > >
> > > > > --
> > > > > *
> > > > > *Open Source Solutions for Text Engineering
> > > > >
> > > > > http://digitalpebble.blogspot.com/
> > > > > http://www.digitalpebble.com
> > > > > http://twitter.com/digitalpebble
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Kiran Chitturi
> > > >
> > >
> > >
> > >
> > > --
> > > *
> > > *Open Source Solutions for Text Engineering
> > >
> > > http://digitalpebble.blogspot.com/
> > > http://www.digitalpebble.com
> > > http://twitter.com/digitalpebble
> > >
> >
> >
> >
> > --
> > Kiran Chitturi
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>



-- 
Kiran Chitturi

Re: bin/nutch parsechecker -dumpText works but bin/nutch parse fails

Posted by Julien Nioche <li...@gmail.com>.
Hi

Yes please do open an issue. The docs should be parsed in one go and I
suspect (yet another) issue with the SQL backend

Thanks

J

On 1 November 2012 13:48, kiran chitturi <ch...@gmail.com> wrote:

> Thank you alxsss for the suggestion. It displays the actualSize and
> inHeaderSize for every file and two more lines in logs but it did not much
> information even when i set parserJob to Debug.
>
> I had the same problem when i re-compiled everything today. I have to run
> the parse command multiple times to get all the files parsed.
>
> I am using SQL with GORA. Its mysql database.
>
> For now, atleast the files are getting parsed, do  i need to open a issue
> for this ?
>
> Thank you,
>
> Regards,
> Kiran.
>
>
> On Wed, Oct 31, 2012 at 4:36 PM, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
> > Hi Kiran
> >
> > Interesting. Which backend are you using with GORA? The SQL one? Could
> be a
> > problem at that level
> >
> > Julien
> >
> > On 31 October 2012 17:01, kiran chitturi <ch...@gmail.com>
> > wrote:
> >
> > > Hi Julien,
> > >
> > > I have just noticed something when running the parse.
> > >
> > > First when i ran the parse command 'sh bin/nutch parse
> > > 1351188762-1772522488', the parsing of all the PDF files has failed.
> > >
> > > When i ran the command again one pdf file got parsed. Next time,
> another
> > > pdf file got parsed.
> > >
> > > When i ran the parse command the number of times the total number of
> pdf
> > > files, all the pdf files got parsed.
> > >
> > > In my case,  i ran it 17 times and all the pdf files are parsed. Before
> > > that, not everything is parsed.
> > >
> > > This sounds strange, do you think it is some configuration problem ?
> > >
> > > I have tried this 2 times and same thing happened two times for me .
> > >
> > > I am not sure why this is happening.
> > >
> > > Thanks for your help.
> > >
> > > Regards,
> > > Kiran.
> > >
> > >
> > > On Wed, Oct 31, 2012 at 10:28 AM, Julien Nioche <
> > > lists.digitalpebble@gmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > >
> > > > > Sorry about that. I did not notice the parsecodes are actually
> nutch
> > > and
> > > > > not tika.
> > > > >
> > > > > no problems!
> > > >
> > > >
> > > > > The setup is local on Mac desktop and i am using through command
> line
> > > and
> > > > > remote debugging through eclipse (
> > > > >
> > > >
> > >
> >
> http://wiki.apache.org/nutch/RunNutchInEclipse#Remote_Debugging_in_Eclipse
> > > > > ).
> > > > >
> > > >
> > > > OK
> > > >
> > > > >
> > > > > I have set both http.content.limit and file.content.limit to -1.
> The
> > > logs
> > > > > just say 'WARN  parse.ParseUtil - Unable to successfully parse
> > content
> > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/pdf/watson.pdf of
> > type
> > > > > application/pdf'.
> > > > >
> > > >
> > > > you set it in $NUTCH_HOME/runtime/local/conf/nutch-site.xml right?
> (not
> > > > in $NUTCH_HOME/conf/nutch-site.xml unless you call 'ant clean
> runtime')
> > > >
> > > >
> > > > >
> > > > > All the html's are getting parsed and when i crawl this page (
> > > > > http://scholar.lib.vt.edu/ejournals/ALAN/v29n3/), all the html's
> and
> > > > some
> > > > > of the pdf files get parsed. Like, half of the pdf files get parsed
> > and
> > > > the
> > > > > other half don't get parsed.
> > > > >
> > > >
> > > > do the ones that are not parsed have something in common? length?
> > > >
> > > >
> > > > > I am not sure about what causing the problem as you said
> parsechecker
> > > is
> > > > > actually work. I want the parser to crawl the full-text of the pdf
> > and
> > > > the
> > > > > metadata, title.
> > > > >
> > > >
> > > > OK
> > > >
> > > >
> > > > >
> > > > > The metatags are also getting crawled for failed pdf parsing.
> > > > >
> > > >
> > > > They would be discarded because of the failure even if they
> > > > were successfully extracted indeed. The current mechanism does not
> > cater
> > > > for semi-failures
> > > >
> > > > J.
> > > >
> > > > --
> > > > *
> > > > *Open Source Solutions for Text Engineering
> > > >
> > > > http://digitalpebble.blogspot.com/
> > > > http://www.digitalpebble.com
> > > > http://twitter.com/digitalpebble
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Chitturi
> > >
> >
> >
> >
> > --
> > *
> > *Open Source Solutions for Text Engineering
> >
> > http://digitalpebble.blogspot.com/
> > http://www.digitalpebble.com
> > http://twitter.com/digitalpebble
> >
>
>
>
> --
> Kiran Chitturi
>



-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble