You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Michael Mitiaguin <mi...@gmail.com> on 2006/05/05 15:19:35 UTC

1.9 indexing

I noticed in development list a comment about using Lucene.Net 1.9 with .Net
2 without  any problem. Quite clearly there is  a memory problem when
indexing more than 50000+ records from database. The same surely applies to
50000+ files (  50000 is just approximate number given as indication  )
I tried a new and deprecated syntax  without success.having   "Out of
memory" in both cases.
Memory consumption grows to 1G and then program bombs.

I didn't have such a problem with 1.4.3 , having indexed up to 1 500 000
records from database.

Remarkably when I use interop calling Lucene from a  COM  client memory is
released for 1.9 for some reason.

Any comments ?

TIA
Michael

Re: 1.9 indexing

Posted by Jeff Rodenburg <je...@gmail.com>.
I believe the plan relative to Lucene 1.9 is to focus community development
using the 1.1 Framework.  There will be a transition in the future to the
2.0 Framework, but for now I believe we're targeting the 1.1 Framework for
Lucene 1.9.

-- j

On 5/5/06, George Aroush <ge...@aroush.net> wrote:
>
> Hi Michael,
>
> The problem you noticed with 1.9 is not due to .NET 2.0, but due to a bug
> in
> Lucene 1.9
>
> Regards,
>
> -- George
>
> -----Original Message-----
> From: Michael Mitiaguin [mailto:mitiaguin@gmail.com]
> Sent: Friday, May 05, 2006 9:20 AM
> To: lucene-net-user@incubator.apache.org
> Subject: 1.9 indexing
>
> I noticed in development list a comment about using Lucene.Net 1.9 with
> .Net
> 2 without  any problem. Quite clearly there is  a memory problem when
> indexing more than 50000+ records from database. The same surely applies
> to
> 50000+ files (  50000 is just approximate number given as indication  )
> I tried a new and deprecated syntax  without success.having   "Out of
> memory" in both cases.
> Memory consumption grows to 1G and then program bombs.
>
> I didn't have such a problem with 1.4.3 , having indexed up to 1 500 000
> records from database.
>
> Remarkably when I use interop calling Lucene from a  COM  client memory is
> released for 1.9 for some reason.
>
> Any comments ?
>
> TIA
> Michael
>
>

RE: 1.9 indexing

Posted by George Aroush <ge...@aroush.net>.
Hi Michael,

You can tell which version/build you are using by examining the file
AssemblyInfo.cs.  Look for the line with [assembly:
AssemblyVersion("1.9.0.002")] -- the last number, 002, is the build which is
what you should be using.

In the next day or two I will be moving the binary downloads to ASF so this
should help complete the transition to ASF from SourceForge.net.

Also, I am not too far off releasing build 3 of 1.9 which is mainly the port
of the Lucene Test code to C#.

Regards,

-- George Aroush 

-----Original Message-----
From: Michael Mitiaguin [mailto:mitiaguin@gmail.com] 
Sent: Sunday, May 07, 2006 11:39 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: 1.9 indexing

George,

It is hard to say. Firstly,  I am not sure how to  download  the most recent
version now Secondly, I seem to have run directory comparison at some stage
and just applied file changes, but whether it was build 2 against build 1
just don't remember.  I am quite sure though , that the problem existed and
has gone for the same Lucene code , but  different computer.

Regards
Michael

On 5/8/06, George Aroush <ge...@aroush.net> wrote:
>
> Hi Michael,
>
> I'm glad to see the problem disappeared, but it would have been better 
> to know what caused it.
>
> May I ask, what build version of 1.9 are you using?  Is it build 1 or 2?
> There are known memory leaks in build 1 which I fixed in build 2.
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Michael Mitiaguin [mailto:mitiaguin@gmail.com]
> Sent: Sunday, May 07, 2006 10:15 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: 1.9 indexing
>
> Sorry for being too quick to attribute a memory problem to Lucene. My 
> box has changed and now I have a difficulty to reproduce a problem. 
> Test case worked successfully , then running the old code which caused 
> a memory grief didn't create any problem either.
> It is hard to explain what made  it  disbehave , but I am a happy man now.
>
> Michael
>
>
> On 5/8/06, Jeff Rodenburg <je...@gmail.com> wrote:
> >
> > Michael -
> >
> > Would you mind posting a test case that reproduces the problem?
> >
> > -- j
> >
> > On 5/6/06, Michael Mitiaguin <mi...@gmail.com> wrote:
> > >
> > > Workaround in my case was to use a COM client which is not 
> > > suitable for general use. Another workaround is to write Java 
> > > indexing code
> > > :) , but
> > if
> > > there is such a problem in Java I am flabbergasted then.
> > > Also bug seems to be critical and does exist ( according to George 
> > > ) irrespective of 1.1 or 2 framework.
> > > So even the focus on 1.1 still without fixing that,  Lucene.Net 
> > > 1.9 can hardly be considered usable
> > > 1.9 was accessible on SourceForge for quite while and if it was a 
> > > known problem just wondering whether the cause still remains unknown.
> > >
> > > Michael
> > >
> > > On 5/6/06, Eyal Post <ey...@epocalipse.com> wrote:
> > > >
> > > > Where's the leak? Is there a simple workaround?
> > > >
> > > > Eyal
> > > >
> > > >
> > >
> > >
> >
> >
>
>


Re: 1.9 indexing

Posted by Michael Mitiaguin <mi...@gmail.com>.
George,

It is hard to say. Firstly,  I am not sure how to  download  the most recent
version now
Secondly, I seem to have run directory comparison at some stage and just
applied file changes, but whether it was build 2 against build 1 just don't
remember.  I am quite sure though , that the problem existed and has gone
for the same Lucene code , but  different computer.

Regards
Michael

On 5/8/06, George Aroush <ge...@aroush.net> wrote:
>
> Hi Michael,
>
> I'm glad to see the problem disappeared, but it would have been better to
> know what caused it.
>
> May I ask, what build version of 1.9 are you using?  Is it build 1 or 2?
> There are known memory leaks in build 1 which I fixed in build 2.
>
> Regards,
>
> -- George Aroush
>
>
> -----Original Message-----
> From: Michael Mitiaguin [mailto:mitiaguin@gmail.com]
> Sent: Sunday, May 07, 2006 10:15 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Re: 1.9 indexing
>
> Sorry for being too quick to attribute a memory problem to Lucene. My box
> has changed and now I have a difficulty to reproduce a problem. Test case
> worked successfully , then running the old code which caused a memory
> grief
> didn't create any problem either.
> It is hard to explain what made  it  disbehave , but I am a happy man now.
>
> Michael
>
>
> On 5/8/06, Jeff Rodenburg <je...@gmail.com> wrote:
> >
> > Michael -
> >
> > Would you mind posting a test case that reproduces the problem?
> >
> > -- j
> >
> > On 5/6/06, Michael Mitiaguin <mi...@gmail.com> wrote:
> > >
> > > Workaround in my case was to use a COM client which is not suitable
> > > for general use. Another workaround is to write Java indexing code
> > > :) , but
> > if
> > > there is such a problem in Java I am flabbergasted then.
> > > Also bug seems to be critical and does exist ( according to George )
> > > irrespective of 1.1 or 2 framework.
> > > So even the focus on 1.1 still without fixing that,  Lucene.Net 1.9
> > > can hardly be considered usable
> > > 1.9 was accessible on SourceForge for quite while and if it was a
> > > known problem just wondering whether the cause still remains unknown.
> > >
> > > Michael
> > >
> > > On 5/6/06, Eyal Post <ey...@epocalipse.com> wrote:
> > > >
> > > > Where's the leak? Is there a simple workaround?
> > > >
> > > > Eyal
> > > >
> > > >
> > >
> > >
> >
> >
>
>

RE: 1.9 indexing

Posted by George Aroush <ge...@aroush.net>.
Hi Michael,

I'm glad to see the problem disappeared, but it would have been better to
know what caused it.

May I ask, what build version of 1.9 are you using?  Is it build 1 or 2?
There are known memory leaks in build 1 which I fixed in build 2.

Regards,

-- George Aroush
 

-----Original Message-----
From: Michael Mitiaguin [mailto:mitiaguin@gmail.com] 
Sent: Sunday, May 07, 2006 10:15 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: 1.9 indexing

Sorry for being too quick to attribute a memory problem to Lucene. My box
has changed and now I have a difficulty to reproduce a problem. Test case
worked successfully , then running the old code which caused a memory grief
didn't create any problem either.
It is hard to explain what made  it  disbehave , but I am a happy man now.

Michael


On 5/8/06, Jeff Rodenburg <je...@gmail.com> wrote:
>
> Michael -
>
> Would you mind posting a test case that reproduces the problem?
>
> -- j
>
> On 5/6/06, Michael Mitiaguin <mi...@gmail.com> wrote:
> >
> > Workaround in my case was to use a COM client which is not suitable 
> > for general use. Another workaround is to write Java indexing code 
> > :) , but
> if
> > there is such a problem in Java I am flabbergasted then.
> > Also bug seems to be critical and does exist ( according to George ) 
> > irrespective of 1.1 or 2 framework.
> > So even the focus on 1.1 still without fixing that,  Lucene.Net 1.9 
> > can hardly be considered usable
> > 1.9 was accessible on SourceForge for quite while and if it was a 
> > known problem just wondering whether the cause still remains unknown.
> >
> > Michael
> >
> > On 5/6/06, Eyal Post <ey...@epocalipse.com> wrote:
> > >
> > > Where's the leak? Is there a simple workaround?
> > >
> > > Eyal
> > >
> > >
> >
> >
>
>


Re: 1.9 indexing

Posted by Michael Mitiaguin <mi...@gmail.com>.
Sorry for being too quick to attribute a memory problem to Lucene. My box
has changed and now I have a difficulty to reproduce a problem. Test case
worked successfully , then running the old code which caused a memory grief
didn't create any problem either.
It is hard to explain what made  it  disbehave , but I am a happy man now.

Michael


On 5/8/06, Jeff Rodenburg <je...@gmail.com> wrote:
>
> Michael -
>
> Would you mind posting a test case that reproduces the problem?
>
> -- j
>
> On 5/6/06, Michael Mitiaguin <mi...@gmail.com> wrote:
> >
> > Workaround in my case was to use a COM client which is not suitable for
> > general use. Another workaround is to write Java indexing code :) , but
> if
> > there is such a problem in Java I am flabbergasted then.
> > Also bug seems to be critical and does exist ( according to George )
> > irrespective of 1.1 or 2 framework.
> > So even the focus on 1.1 still without fixing that,  Lucene.Net 1.9 can
> > hardly be considered usable
> > 1.9 was accessible on SourceForge for quite while and if it was a known
> > problem just wondering whether the cause still remains unknown.
> >
> > Michael
> >
> > On 5/6/06, Eyal Post <ey...@epocalipse.com> wrote:
> > >
> > > Where's the leak? Is there a simple workaround?
> > >
> > > Eyal
> > >
> > >
> >
> >
>
>

Re: 1.9 indexing

Posted by Jeff Rodenburg <je...@gmail.com>.
Michael -

Would you mind posting a test case that reproduces the problem?

-- j

On 5/6/06, Michael Mitiaguin <mi...@gmail.com> wrote:
>
> Workaround in my case was to use a COM client which is not suitable for
> general use. Another workaround is to write Java indexing code :) , but if
> there is such a problem in Java I am flabbergasted then.
> Also bug seems to be critical and does exist ( according to George )
> irrespective of 1.1 or 2 framework.
> So even the focus on 1.1 still without fixing that,  Lucene.Net 1.9 can
> hardly be considered usable
> 1.9 was accessible on SourceForge for quite while and if it was a known
> problem just wondering whether the cause still remains unknown.
>
> Michael
>
> On 5/6/06, Eyal Post <ey...@epocalipse.com> wrote:
> >
> > Where's the leak? Is there a simple workaround?
> >
> > Eyal
> >
> >
>
>

Re: 1.9 indexing

Posted by Michael Mitiaguin <mi...@gmail.com>.
Workaround in my case was to use a COM client which is not suitable for
general use. Another workaround is to write Java indexing code :) , but if
there is such a problem in Java I am flabbergasted then.
Also bug seems to be critical and does exist ( according to George )
irrespective of 1.1 or 2 framework.
So even the focus on 1.1 still without fixing that,  Lucene.Net 1.9 can
hardly be considered usable
1.9 was accessible on SourceForge for quite while and if it was a known
problem just wondering whether the cause still remains unknown.

Michael

On 5/6/06, Eyal Post <ey...@epocalipse.com> wrote:
>
> Where's the leak? Is there a simple workaround?
>
> Eyal
>
>

RE: 1.9 indexing

Posted by Eyal Post <ey...@epocalipse.com>.
Where's the leak? Is there a simple workaround?

Eyal 

> -----Original Message-----
> From: George Aroush [mailto:george@aroush.net] 
> Sent: Friday, May 05, 2006 17:46 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: 1.9 indexing
> 
> Hi Michael,
> 
> The problem you noticed with 1.9 is not due to .NET 2.0, but 
> due to a bug in Lucene 1.9
> 
> Regards,
> 
> -- George 
> 
> -----Original Message-----
> From: Michael Mitiaguin [mailto:mitiaguin@gmail.com]
> Sent: Friday, May 05, 2006 9:20 AM
> To: lucene-net-user@incubator.apache.org
> Subject: 1.9 indexing
> 
> I noticed in development list a comment about using 
> Lucene.Net 1.9 with .Net
> 2 without  any problem. Quite clearly there is  a memory 
> problem when indexing more than 50000+ records from database. 
> The same surely applies to
> 50000+ files (  50000 is just approximate number given as 
> indication  )
> I tried a new and deprecated syntax  without success.having   "Out of
> memory" in both cases.
> Memory consumption grows to 1G and then program bombs.
> 
> I didn't have such a problem with 1.4.3 , having indexed up 
> to 1 500 000 records from database.
> 
> Remarkably when I use interop calling Lucene from a  COM  
> client memory is released for 1.9 for some reason.
> 
> Any comments ?
> 
> TIA
> Michael
> 
> 
> 


RE: 1.9 indexing

Posted by George Aroush <ge...@aroush.net>.
Hi Michael,

The problem you noticed with 1.9 is not due to .NET 2.0, but due to a bug in
Lucene 1.9

Regards,

-- George 

-----Original Message-----
From: Michael Mitiaguin [mailto:mitiaguin@gmail.com] 
Sent: Friday, May 05, 2006 9:20 AM
To: lucene-net-user@incubator.apache.org
Subject: 1.9 indexing

I noticed in development list a comment about using Lucene.Net 1.9 with .Net
2 without  any problem. Quite clearly there is  a memory problem when
indexing more than 50000+ records from database. The same surely applies to
50000+ files (  50000 is just approximate number given as indication  )
I tried a new and deprecated syntax  without success.having   "Out of
memory" in both cases.
Memory consumption grows to 1G and then program bombs.

I didn't have such a problem with 1.4.3 , having indexed up to 1 500 000
records from database.

Remarkably when I use interop calling Lucene from a  COM  client memory is
released for 1.9 for some reason.

Any comments ?

TIA
Michael