You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by jcze <je...@gmail.com> on 2008/10/29 19:56:03 UTC
Lost regrading Stemming in nutch
Hi, i'm using nutch 0.8.1, I'm lost about the stemming of nutch, tried the
wiki on MultiLingual Support. coz it said that it could stem the words..
hmm.. but I'm lost because it said that I need to modify the IndexSegment
class which i couldnt find.. =(
Anywayz, i tried the stemming for nutch 8.. but i'm lost again.. so it didnt
work again.. =(
I need some guidance and help.. really really lost =((
--
View this message in context: http://www.nabble.com/Lost-regrading-Stemming-in-nutch-tp20233602p20233602.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Lost regrading Stemming in nutch
Posted by David Jashi <da...@jashi.ge>.
I managed to connect Nutch 0.9 to my stemming machine. Don't know if
my approach would work on 0.8.1
On Wed, Oct 29, 2008 at 10:56 PM, jcze <je...@gmail.com> wrote:
>
> Hi, i'm using nutch 0.8.1, I'm lost about the stemming of nutch, tried the
> wiki on MultiLingual Support. coz it said that it could stem the words..
> hmm.. but I'm lost because it said that I need to modify the IndexSegment
> class which i couldnt find.. =(
>
> Anywayz, i tried the stemming for nutch 8.. but i'm lost again.. so it didnt
> work again.. =(
>
> I need some guidance and help.. really really lost =((
> --
> View this message in context: http://www.nabble.com/Lost-regrading-Stemming-in-nutch-tp20233602p20233602.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
--
with best regards,
David Jashi
Web development EO,
Caucasus Online
+995(32)970368
David@Jashi.ge
პატივისცემით,
დავით ჯაში
ვებ–განვითარების დირექტორი
"კავკასუს ონლაინი"
+995(32)970368
David@Jashi.ge
Re: Xmx settings
Posted by Alex Basa <al...@yahoo.com>.
I'm running a blade server with 16GB of memory and run 5 simultaneous crawls at a time when I'm crawling big sites otherwise, I use the Xmx of 1GB and run 15 crawls at a time. Most of my big crawls seem to run out of memory even with a setting like 3GB. What settings do you use for sites like wikipedia.com?
--- On Thu, 10/30/08, Alexander Aristov <al...@gmail.com> wrote:
> From: Alexander Aristov <al...@gmail.com>
> Subject: Re: Xmx settings
> To: nutch-user@lucene.apache.org, alex_basa@yahoo.com
> Date: Thursday, October 30, 2008, 12:58 AM
> Try to inclrease swap size as the message states. Even if
> you specify that
> java should use 3G of RAM you might run out of memory if
> you OS doesn't
> support such sizes.
>
> How many memory do you have on the machine?
>
> Alex
>
> 2008/10/29 Alex Basa <al...@yahoo.com>
>
> > I'm getting several OutOfMemoryError messages from
> large sites like this
> > one and gutenberg.com
> >
> > JAVA_HEAP_MAX=-Xmx3528m
> > JAVA_PERM_HEAP=-XX:MaxPermSize=128m
> >
> > Anyone care to share their settings for doing a crawl
> level 1 on a huge
> > site?
> >
> > fetching
> http://www.azlyrics.com/lyrics/3oh3/coloradosunrise.html
> > fetching
> http://www.azlyrics.com/lyrics/unloco/facedown.html
> > fetching
> http://www.azlyrics.com/lyrics/unfinishedthought/starfighter.html
> > #
> > # An unexpected error has been detected by Java
> Runtime Environment:
> > #
> > # java.lang.OutOfMemoryError: requested 145320 bytes
> for Chunk::new. Out of
> > swap
> > space?
> > #
> > # Internal Error (allocation.cpp:218), pid=29949,
> tid=12
> > # Error: Chunk::new
> >
> >
> >
> >
> >
>
>
> --
> Best Regards
> Alexander Aristov
Re: Xmx settings
Posted by Alexander Aristov <al...@gmail.com>.
Try to inclrease swap size as the message states. Even if you specify that
java should use 3G of RAM you might run out of memory if you OS doesn't
support such sizes.
How many memory do you have on the machine?
Alex
2008/10/29 Alex Basa <al...@yahoo.com>
> I'm getting several OutOfMemoryError messages from large sites like this
> one and gutenberg.com
>
> JAVA_HEAP_MAX=-Xmx3528m
> JAVA_PERM_HEAP=-XX:MaxPermSize=128m
>
> Anyone care to share their settings for doing a crawl level 1 on a huge
> site?
>
> fetching http://www.azlyrics.com/lyrics/3oh3/coloradosunrise.html
> fetching http://www.azlyrics.com/lyrics/unloco/facedown.html
> fetching http://www.azlyrics.com/lyrics/unfinishedthought/starfighter.html
> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> # java.lang.OutOfMemoryError: requested 145320 bytes for Chunk::new. Out of
> swap
> space?
> #
> # Internal Error (allocation.cpp:218), pid=29949, tid=12
> # Error: Chunk::new
>
>
>
>
>
--
Best Regards
Alexander Aristov
Xmx settings
Posted by Alex Basa <al...@yahoo.com>.
I'm getting several OutOfMemoryError messages from large sites like this one and gutenberg.com
JAVA_HEAP_MAX=-Xmx3528m
JAVA_PERM_HEAP=-XX:MaxPermSize=128m
Anyone care to share their settings for doing a crawl level 1 on a huge site?
fetching http://www.azlyrics.com/lyrics/3oh3/coloradosunrise.html
fetching http://www.azlyrics.com/lyrics/unloco/facedown.html
fetching http://www.azlyrics.com/lyrics/unfinishedthought/starfighter.html
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 145320 bytes for Chunk::new. Out of swap
space?
#
# Internal Error (allocation.cpp:218), pid=29949, tid=12
# Error: Chunk::new