convert segment dump into text for data mining.

[Felix Zimmermann <ma...@felix-zimmermann.eu>]

Hi,

I would like to convert the segment dumpfile into a textfile in order to
analyse it later with a data mining programm. I am not very familar with
file formats and so did not succeed using commands like od/uuencode. I
use Ubuntu 9.10.

Thanks for help,
Felix.

Re: convert segment dump into text for data mining.

[Hannes Carl Meyer <ha...@googlemail.com>]

Hi Felix,

usually there is a lot of data stored in the segments.
Which data do you need? The webpage content or term freqs only?

You should also consider to perform this command to fetch the whole contents
./bin/nutch readseg

For example: ./bin/nutch readseg -dump crawl/segments/XXX/ dump_folder
-nofetch -nogenerate -noparse -noparsedata -noparsetex

Regards

Hannes

https://www.xing.com/profile/HannesCarl_Meyer
http://de.linkedin.com/in/hannescarlmeyer

On Thu, Feb 18, 2010 at 9:45 AM, Felix Zimmermann
<ma...@felix-zimmermann.eu>wrote:

> Hi,
>
> I would like to convert the segment dumpfile into a textfile in order to
> analyse it later with a data mining programm. I am not very familar with
> file formats and so did not succeed using commands like od/uuencode. I
> use Ubuntu 9.10.
>
> Thanks for help,
> Felix.
>
>