You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Tim Allison <ta...@apache.org> on 2023/10/18 14:28:45 UTC
truncation, parsing and indexing?
I'm trying to configure Nutch to index pages/files that are truncated (in
addition to the successful non-truncated files).
I'm using the okhttp protocol, because I don't think the http protocol
stores truncation information.
I'm using parse-tika, and the "parser.skip.truncated" is set to
default=true.
The particular PDF that I'm experimenting with is returned chunked with gz
compression. There is no length header in the response.
For this PDF, okhttp correctly marks it as truncated, but then the file is
sent to parsetika, which throws a parse exception. The file is then not
sent to the index.
If I understand correctly, ParseSegment is checking for truncation, but it
requires a Content-Length header to work. In my case, there is no
Content-Length header, so it assumes the file is not truncated.
Should I open a ticket to have ParseSegment also check for okhttp's header (
http.content.truncated=true)?
Is there a way to index files even if they are truncated or if there is a
parse exception?
If indexing is a bridge too far, what's the most efficient way to dump a
list of urls that are truncated and/or had a parse exception?
Thank you!
Best,
Tim
Re: truncation, parsing and indexing?
Posted by Tim Allison <ta...@apache.org>.
I opened: https://issues.apache.org/jira/projects/NUTCH/issues/NUTCH-3026 for further discussion.
On 2023/11/03 13:31:26 Tim Allison wrote:
> After thinking about this some more... my overall goal is a status index that contains lots of information about the crawl except the parsed metadata + content. I'd like to write this to an OpenSearch index so that we can perform analysis on the crawl.
>
> If that's the case, maybe the cleanest path forward would be to add a "statusOnly" flag to the IndexingJob and just index all the status bits. I'm still coming up to speed on what's available from the crawldb vs the segments etc so some of this may be offbase, but something like:
>
> target url
> fetched url (in case of redirects?)
> target host
> fetched host (in case of redirects?)
> fetched timestamp(?)
> response code
> fetch status
> content-length
> digest
> truncated
> truncated reason
> content type
> parse status (bonus points for parse exception's stacktrace or parse timeout/crash info)
> inlinks (optional ?)
> outlinks (would require successful parse)
> user injected crawl metadata ... as we now allow for regular indexing via configuration
> user selected parse metadata ... as we now allow for regular indexing via configuration
> whether the file was successfully indexed or not (how do we get this info?)
>
> Does this seem like a reasonable path forward?
> Are there other status bits we should be pulling?
>
> Would others find this generally useful, or should I just copy/paste the IndexinJob into a personal addon repo and modify that?
>
Re: truncation, parsing and indexing?
Posted by Tim Allison <ta...@apache.org>.
After thinking about this some more... my overall goal is a status index that contains lots of information about the crawl except the parsed metadata + content. I'd like to write this to an OpenSearch index so that we can perform analysis on the crawl.
If that's the case, maybe the cleanest path forward would be to add a "statusOnly" flag to the IndexingJob and just index all the status bits. I'm still coming up to speed on what's available from the crawldb vs the segments etc so some of this may be offbase, but something like:
target url
fetched url (in case of redirects?)
target host
fetched host (in case of redirects?)
fetched timestamp(?)
response code
fetch status
content-length
digest
truncated
truncated reason
content type
parse status (bonus points for parse exception's stacktrace or parse timeout/crash info)
inlinks (optional ?)
outlinks (would require successful parse)
user injected crawl metadata ... as we now allow for regular indexing via configuration
user selected parse metadata ... as we now allow for regular indexing via configuration
whether the file was successfully indexed or not (how do we get this info?)
Does this seem like a reasonable path forward?
Are there other status bits we should be pulling?
Would others find this generally useful, or should I just copy/paste the IndexinJob into a personal addon repo and modify that?
Re: truncation, parsing and indexing?
Posted by Sebastian Nagel <sn...@apache.org>.
Hi Tim,
>> I'm using the okhttp protocol, because I don't think the http protocol
>> stores truncation information.
protocol-http could mark truncations as well, however. Please, also open an
issue for this and other protocol plugins.
>> Should I open a ticket to have ParseSegment also check for okhttp's header (
>> http.content.truncated=true)?
Yes, please.
> One work around to ignore parse exceptions (at least in the Tika parser):
> https://github.com/tballison/nutch/tree/ignore-parse-exception
One potential improvement: could still parse MIME types which are parseable
when truncated, most important HTML pages.
>> If I understand correctly, ParseSegment is checking for truncation, but it
>> requires a Content-Length header to work. In my case, there is no
>> Content-Length header, so it assumes the file is not truncated.
With chunked Content-Encoding, there is usually no Content-Length header.
Even if there is a Content-Length header: it indicates the compressed length
with HTTP Content-Encoding "gzip", "deflate" or "brotli".
>> Is there a way to index files even if they are truncated or if there is a
>> parse exception?
>>
>> If indexing is a bridge too far, what's the most efficient way to dump a
>> list of urls that are truncated and/or had a parse exception?
Let me think about it...
Best,
Sebastian
On 10/18/23 18:16, Tim Allison wrote:
> One work around to ignore parse exceptions (at least in the Tika parser): https://github.com/tballison/nutch/tree/ignore-parse-exception
>
> Proposed fix for truncation checking:
> https://github.com/tballison/nutch/tree/okhttp-truncated
>
> On 2023/10/18 14:28:45 Tim Allison wrote:
>> I'm trying to configure Nutch to index pages/files that are truncated (in
>> addition to the successful non-truncated files).
>>
>> I'm using the okhttp protocol, because I don't think the http protocol
>> stores truncation information.
>>
>> I'm using parse-tika, and the "parser.skip.truncated" is set to
>> default=true.
>>
>> The particular PDF that I'm experimenting with is returned chunked with gz
>> compression. There is no length header in the response.
>>
>> For this PDF, okhttp correctly marks it as truncated, but then the file is
>> sent to parsetika, which throws a parse exception. The file is then not
>> sent to the index.
>>
>> If I understand correctly, ParseSegment is checking for truncation, but it
>> requires a Content-Length header to work. In my case, there is no
>> Content-Length header, so it assumes the file is not truncated.
>>
>> Should I open a ticket to have ParseSegment also check for okhttp's header (
>> http.content.truncated=true)?
>>
>> Is there a way to index files even if they are truncated or if there is a
>> parse exception?
>>
>> If indexing is a bridge too far, what's the most efficient way to dump a
>> list of urls that are truncated and/or had a parse exception?
>>
>> Thank you!
>>
>> Best,
>>
>> Tim
>>
Re: truncation, parsing and indexing?
Posted by Tim Allison <ta...@apache.org>.
One work around to ignore parse exceptions (at least in the Tika parser): https://github.com/tballison/nutch/tree/ignore-parse-exception
Proposed fix for truncation checking:
https://github.com/tballison/nutch/tree/okhttp-truncated
On 2023/10/18 14:28:45 Tim Allison wrote:
> I'm trying to configure Nutch to index pages/files that are truncated (in
> addition to the successful non-truncated files).
>
> I'm using the okhttp protocol, because I don't think the http protocol
> stores truncation information.
>
> I'm using parse-tika, and the "parser.skip.truncated" is set to
> default=true.
>
> The particular PDF that I'm experimenting with is returned chunked with gz
> compression. There is no length header in the response.
>
> For this PDF, okhttp correctly marks it as truncated, but then the file is
> sent to parsetika, which throws a parse exception. The file is then not
> sent to the index.
>
> If I understand correctly, ParseSegment is checking for truncation, but it
> requires a Content-Length header to work. In my case, there is no
> Content-Length header, so it assumes the file is not truncated.
>
> Should I open a ticket to have ParseSegment also check for okhttp's header (
> http.content.truncated=true)?
>
> Is there a way to index files even if they are truncated or if there is a
> parse exception?
>
> If indexing is a bridge too far, what's the most efficient way to dump a
> list of urls that are truncated and/or had a parse exception?
>
> Thank you!
>
> Best,
>
> Tim
>