You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by jjmendes <jj...@student.dei.uc.pt> on 2017/03/16 15:59:40 UTC

Content truncated while using commoncrawldump

I am currently attempting to dump the contents of a crawl into multiple
WARC files using 

./bin/nutch commoncrawldump -outputDir nameOfOutputDir -segment
crawl/segments/segmentDir -warc

However, I get multiple occurrences of 

URL skipped. Content of size X was truncated to Y. 

I have set both http.content.limit and file.content.limit to -1 in order
to remove any limits, but I'm guessing neither applies to this
situation. Any way of removing said cap? 

Thanks, 

JJAM