You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Maik Schreiber <bl...@blizzy.de> on 2005/03/16 22:32:57 UTC
Interfaces
Hi,
I know the issue of using interfaces in Lucene has been discussed a few
times before. I'd like to bring that to the table once again, though,
because I think all those discussions were missing an important point: unit
testing.
Without coding to interfaces it gets quite complicated to test for calls
being made to an IndexWriter, for example. Of course it's possible to
subclass IndexWriter and intercept all relevant calls, it still remains
being a PITA.
Consider the following JUnit test method for testing a hypothetical
Foo.foo(IndexWriter, Document) method that does nothing more than
IndexWriter.addDocument(doc):
public void testFoo() throws IOException {
final List addedDocuments = new ArrayList();
Directory dir = new RAMDirectory();
Analyzer analyzer = new StandardAnalyzer();
IndexWriter writer = new IndexWriter(dir, analyzer, true) {
public void addDocument(Document doc) throws IOException {
super.addDocument(doc);
addedDocuments.add(doc);
}
};
Document doc = new Document();
new Foo().foo(writer, doc);
assertEquals(1, addedDocuments.size());
assertSame(doc, addedDocuments.get(0));
}
As you can see, you have to go at relatively great lengths just to test for
a single method call. You need to create a Directory for the IndexWriter,
and the IndexWriter instance itself. The IndexWriter really adds the
document to the directory. In addition to that, we're taking notes of the
added documents, so we can assert those later on.
This complexity even grows when doing other this: Say you want to test a
call to IndexReader. The problem is that IndexReader.open() needs to have a
valid index lying around. So to set up the environment for that requirement
you first need to create a fake index in a RAMDirectory using an
IndexWriter, then use that index to test IndexReader calls.
Now this is the same test method as above, but this time using EasyMock and
an interface IIndexWriter (which IndexWriter hypothetically implements):
public void testFoo() throws IOException {
MockControl writerControl =
MockControl.createControl(IIndexWriter.class);
IIndexWriter writer = (IIndexWriter) writerControl.getMock();
Document doc = new Document();
writer.addDocument(doc);
writerControl.setVoidCallable();
writerControl.replay();
new Foo().foo(writer, doc);
writerControl.verify();
}
This time, it's all straightforward: No Directory created, no IndexWriter
created, no document added. We just tell the mocked-up IndexWriter what
method calls to expect and what to return (or do nothing, or throw an
exception). Also there's no taking notes of added documents because EasyMock
does that already.
The net benefit is that there are no useless objects created. Those could
even harm the test results in the worst case. Also the test should run
faster now, which can really add up when you have hundreds or thousands of
tests.
--
Maik Schreiber * http://www.blizzy.de
GPG public key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1F11D713
Key fingerprint: CF19 AFCE 6E3D 5443 9599 18B5 5640 1F11 D713
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Doug Cutting <cu...@apache.org>.
Erik Hatcher wrote:
> Ultimately, though, the decision to refactor the codebase to use
> interfaces more pervasively lies with Doug.
Actually the decision lies not with me, but with the Lucene PMC as a
group, according to Apache's voting process:
http://www.apache.org/foundation/voting.html
But, like other PMC members, I do have veto power. Fortunately I've
never had to use it. We've always come to agreement through reasoned
argument.
In this particular case, I do not object to making interfaces for
IndexReader and IndexWriter. What we need is a well-designed,
back-compatible, fully-javadoc'd patch that implements this. Folks
(including myself) may have reasonable objections to the first version.
So it will take some patience and determination to make such a change
to these core classes.
My guess at the best way to approach this is to add new interfaces
(e.g., named IndexReadable and IndexWritable) and to then have
IndexReader and IndexWriter implement these. That makes
back-compatibility easier. Then one could add factories too. Someone
proposed IndexReaderWriterFactory as a name. Why not just call this
IndexFactory?
Finally, this change, on its own, is not a high-priority for me. Long
term, I hope we can re-structure these APIs to make them extensible in
more interesting ways. Consider point 11 in section 1.2 of
http://wiki.apache.org/jakarta-lucene/Lucene2Whiteboard as an example.
This sort of thing would probably best be implemented by decomposing
IndexWriter and IndexReader into sets of interfaces. But until we do
that I don't think I would make use of interfaces here. So if others
would make good use of these interfaces, then the onus is on them to
make the change.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
I didn't mean to imply that using interfaces themselves were a
potential performance issue.... what I meant was all the IoC mechanisms
that might be tossed in as factory overhead to construct things
indirectly.
Erik
On Mar 18, 2005, at 10:22 AM, Maik Schreiber wrote:
>> However, the primary focus of Lucene is and always will be
>> performance. It would be important to demonstrate that resource
>> utilization and performance are not adversely affected by such
>> changes.
>
> I can't see why interfaces should affect performance or resource
> utilization in a tangible way.
>
> For interface calls, the resulting bytecode would use invokeinterface
> instead of invokevirtual calls, but the speed difference is absolutely
> negligible (in the range of 500 nanoseconds per call at most during my
> tests with JDK 1.5.0_02 and JProfiler 3.3).
>
> Resource utilization would take up some bytes for the additional
> interfaces and generated code from the JIT compiler, but that's about
> it.
>
> --
> Maik Schreiber * http://www.blizzy.de
>
> GPG public key:
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1F11D713
> Key fingerprint: CF19 AFCE 6E3D 5443 9599 18B5 5640 1F11 D713
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Maik Schreiber <bl...@blizzy.de>.
> However, the primary focus of
> Lucene is and always will be performance. It would be important to
> demonstrate that resource utilization and performance are not adversely
> affected by such changes.
I can't see why interfaces should affect performance or resource utilization
in a tangible way.
For interface calls, the resulting bytecode would use invokeinterface
instead of invokevirtual calls, but the speed difference is absolutely
negligible (in the range of 500 nanoseconds per call at most during my tests
with JDK 1.5.0_02 and JProfiler 3.3).
Resource utilization would take up some bytes for the additional interfaces
and generated code from the JIT compiler, but that's about it.
--
Maik Schreiber * http://www.blizzy.de
GPG public key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1F11D713
Key fingerprint: CF19 AFCE 6E3D 5443 9599 18B5 5640 1F11 D713
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Mar 17, 2005, at 9:15 AM, Maik Schreiber wrote:
>> Pragmatically, have you ever had addDocument fail? If not, then what
>> peace of mind are you getting from such a test?
>
> The test I've shown is not supposed to test if addDocument() fails or
> not, but to test if addDocument() is invoked at all. The tested method
> was Foo.foo(), which could be doing more things that could possibly
> fail before addDocument() is invoked at all.
Point taken.
Ease of unit testing is a reasonable argument for refactoring Lucene's
API to be more interface-based - agreed. However, the primary focus of
Lucene is and always will be performance. It would be important to
demonstrate that resource utilization and performance are not adversely
affected by such changes.
Again, Doug needs to be the one to approve such a refactoring.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Maik Schreiber <bl...@blizzy.de>.
> Pragmatically,
> have you ever had addDocument fail? If not, then what peace of mind are
> you getting from such a test?
The test I've shown is not supposed to test if addDocument() fails or not,
but to test if addDocument() is invoked at all. The tested method was
Foo.foo(), which could be doing more things that could possibly fail before
addDocument() is invoked at all.
--
Maik Schreiber * http://www.blizzy.de
GPG public key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1F11D713
Key fingerprint: CF19 AFCE 6E3D 5443 9599 18B5 5640 1F11 D713
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Todd VanderVeen <td...@part.net>.
Erik Hatcher wrote:
> I think, though I'm not speaking for anyone here but myself, that the
> Lucene team is open to API improvements that _do not adversely affect
> performance_ and that have _a real benefit_.
>
> While I'm as IoC and design pattern savvy as the next developer, I'm
> also highly pragmatic. I've not been convinced by any of the examples
> thus far. If a unit test needs to detect if a document has been
> added, you can check the size of the index before and after for
> example, rather than doing some mock trickery to hook the addDocument
> call. Or you could use AspectJ to do this if you really wanted to get
> fancy. The unit test example shown is really a unit test appropriate
> to Lucene itself, and not to a project using Lucene, it seems.
> Pragmatically, have you ever had addDocument fail? If not, then what
> peace of mind are you getting from such a test?
>
> Ultimately, though, the decision to refactor the codebase to use
> interfaces more pervasively lies with Doug.
>
> Erik
>
I have seen this and several other similar requests dismissed with
similar reasoning. I, myself, once made similar requests of the abstract
classes in the analysis package. Yes, there are lots of ways to skin a
cat, but I would ask that a different question be asked, "'What is
risked in giving the flexibility to the user base to do it their way?"
These requests are often simply an appeal for flexibility. I can
understand the maintainers concerns about increasingly non-uniform use
of the API and associated cost of helping those experimenting. I don't
know how comfortable the maintainers are with simply letting these
people work it out for themselves or with the help of others in the
forum, but I hope these concerns are weighed against the possible
benefits to the people doing the myriad and sundry things they do with
open source projects. Flexibility is a real benefit.
Cheers,
Todd VanderVeen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Interfaces
Posted by Robert Engels <re...@ix.netcom.com>.
I only speak from current project experience, where we've developed a
"search server" utilizing lucene , utilizing custom indexing formats in
order to support high-speed incremental indexing. It was necessary to extend
IndexReader in order to have the rest of the searching framework work
properly, even though I use NO METHODS in the base class. Seems silly.
-----Original Message-----
From: Erik Hatcher [mailto:erik@ehatchersolutions.com]
Sent: Thursday, March 17, 2005 5:31 AM
To: java-dev@lucene.apache.org
Subject: Re: Interfaces
I think, though I'm not speaking for anyone here but myself, that the
Lucene team is open to API improvements that _do not adversely affect
performance_ and that have _a real benefit_.
While I'm as IoC and design pattern savvy as the next developer, I'm
also highly pragmatic. I've not been convinced by any of the examples
thus far. If a unit test needs to detect if a document has been added,
you can check the size of the index before and after for example,
rather than doing some mock trickery to hook the addDocument call. Or
you could use AspectJ to do this if you really wanted to get fancy.
The unit test example shown is really a unit test appropriate to Lucene
itself, and not to a project using Lucene, it seems. Pragmatically,
have you ever had addDocument fail? If not, then what peace of mind
are you getting from such a test?
Ultimately, though, the decision to refactor the codebase to use
interfaces more pervasively lies with Doug.
Erik
On Mar 17, 2005, at 2:40 AM, Konstantin Priblouda wrote:
>
> --- Robert Engels <re...@ix.netcom.com> wrote:
>
>> I've been pushing for making IndexReader and
>> IndexWriter interfaces for a
>> long time (and changing lucene core to use them),
>> but there seems a
>> reluctance to do so (although I am not sure why).
>> The current factory
>> methods could be easily moved to a IndexReaderWriter
>> factory class. Probably
>> a top-level interface IndexReaderWriter with methods
>> like isWritable() for
>> "read only" indices, and methods getReader(),
>> getWriter().
>
>
> Hi guys,
> what about making lucene a bit more IoC Friendly?
> ( I speak of providing objects that can be created wia
> constructor, and not by factory )
>
> I'm tired of writing adapter classes...
>
> regards,
>
> ----[ Konstantin Pribluda ( ko5tik ) ]----------------
> Plugins for xdoclet-2 are released. check it out at:
> http://www.sourceforge.net/projects/xdoclet-plugins/
> ----[ http://www.pribluda.de ]------------------------
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Small Business - Try our new resources site!
> http://smallbusiness.yahoo.com/resources/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Interfaces
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
I think, though I'm not speaking for anyone here but myself, that the
Lucene team is open to API improvements that _do not adversely affect
performance_ and that have _a real benefit_.
While I'm as IoC and design pattern savvy as the next developer, I'm
also highly pragmatic. I've not been convinced by any of the examples
thus far. If a unit test needs to detect if a document has been added,
you can check the size of the index before and after for example,
rather than doing some mock trickery to hook the addDocument call. Or
you could use AspectJ to do this if you really wanted to get fancy.
The unit test example shown is really a unit test appropriate to Lucene
itself, and not to a project using Lucene, it seems. Pragmatically,
have you ever had addDocument fail? If not, then what peace of mind
are you getting from such a test?
Ultimately, though, the decision to refactor the codebase to use
interfaces more pervasively lies with Doug.
Erik
On Mar 17, 2005, at 2:40 AM, Konstantin Priblouda wrote:
>
> --- Robert Engels <re...@ix.netcom.com> wrote:
>
>> I've been pushing for making IndexReader and
>> IndexWriter interfaces for a
>> long time (and changing lucene core to use them),
>> but there seems a
>> reluctance to do so (although I am not sure why).
>> The current factory
>> methods could be easily moved to a IndexReaderWriter
>> factory class. Probably
>> a top-level interface IndexReaderWriter with methods
>> like isWritable() for
>> "read only" indices, and methods getReader(),
>> getWriter().
>
>
> Hi guys,
> what about making lucene a bit more IoC Friendly?
> ( I speak of providing objects that can be created wia
> constructor, and not by factory )
>
> I'm tired of writing adapter classes...
>
> regards,
>
> ----[ Konstantin Pribluda ( ko5tik ) ]----------------
> Plugins for xdoclet-2 are released. check it out at:
> http://www.sourceforge.net/projects/xdoclet-plugins/
> ----[ http://www.pribluda.de ]------------------------
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Small Business - Try our new resources site!
> http://smallbusiness.yahoo.com/resources/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Interfaces
Posted by Konstantin Priblouda <kp...@yahoo.com>.
--- Robert Engels <re...@ix.netcom.com> wrote:
> I've been pushing for making IndexReader and
> IndexWriter interfaces for a
> long time (and changing lucene core to use them),
> but there seems a
> reluctance to do so (although I am not sure why).
> The current factory
> methods could be easily moved to a IndexReaderWriter
> factory class. Probably
> a top-level interface IndexReaderWriter with methods
> like isWritable() for
> "read only" indices, and methods getReader(),
> getWriter().
Hi guys,
what about making lucene a bit more IoC Friendly?
( I speak of providing objects that can be created wia
constructor, and not by factory )
I'm tired of writing adapter classes...
regards,
----[ Konstantin Pribluda ( ko5tik ) ]----------------
Plugins for xdoclet-2 are released. check it out at:
http://www.sourceforge.net/projects/xdoclet-plugins/
----[ http://www.pribluda.de ]------------------------
__________________________________
Do you Yahoo!?
Yahoo! Small Business - Try our new resources site!
http://smallbusiness.yahoo.com/resources/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: Interfaces
Posted by Robert Engels <re...@ix.netcom.com>.
I've been pushing for making IndexReader and IndexWriter interfaces for a
long time (and changing lucene core to use them), but there seems a
reluctance to do so (although I am not sure why). The current factory
methods could be easily moved to a IndexReaderWriter factory class. Probably
a top-level interface IndexReaderWriter with methods like isWritable() for
"read only" indices, and methods getReader(), getWriter().
With a decent IDE a single developer could make all of the changes in about
a half a day, but it would need support of the "team", since the patch would
be huge.
Robert Engels
-----Original Message-----
From: Maik Schreiber [mailto:blizzy-keyword-lucene_dev.6564c8@blizzy.de]
Sent: Wednesday, March 16, 2005 3:33 PM
To: java-dev@lucene.apache.org
Subject: Interfaces
Hi,
I know the issue of using interfaces in Lucene has been discussed a few
times before. I'd like to bring that to the table once again, though,
because I think all those discussions were missing an important point: unit
testing.
Without coding to interfaces it gets quite complicated to test for calls
being made to an IndexWriter, for example. Of course it's possible to
subclass IndexWriter and intercept all relevant calls, it still remains
being a PITA.
Consider the following JUnit test method for testing a hypothetical
Foo.foo(IndexWriter, Document) method that does nothing more than
IndexWriter.addDocument(doc):
public void testFoo() throws IOException {
final List addedDocuments = new ArrayList();
Directory dir = new RAMDirectory();
Analyzer analyzer = new StandardAnalyzer();
IndexWriter writer = new IndexWriter(dir, analyzer, true) {
public void addDocument(Document doc) throws IOException {
super.addDocument(doc);
addedDocuments.add(doc);
}
};
Document doc = new Document();
new Foo().foo(writer, doc);
assertEquals(1, addedDocuments.size());
assertSame(doc, addedDocuments.get(0));
}
As you can see, you have to go at relatively great lengths just to test for
a single method call. You need to create a Directory for the IndexWriter,
and the IndexWriter instance itself. The IndexWriter really adds the
document to the directory. In addition to that, we're taking notes of the
added documents, so we can assert those later on.
This complexity even grows when doing other this: Say you want to test a
call to IndexReader. The problem is that IndexReader.open() needs to have a
valid index lying around. So to set up the environment for that requirement
you first need to create a fake index in a RAMDirectory using an
IndexWriter, then use that index to test IndexReader calls.
Now this is the same test method as above, but this time using EasyMock and
an interface IIndexWriter (which IndexWriter hypothetically implements):
public void testFoo() throws IOException {
MockControl writerControl =
MockControl.createControl(IIndexWriter.class);
IIndexWriter writer = (IIndexWriter) writerControl.getMock();
Document doc = new Document();
writer.addDocument(doc);
writerControl.setVoidCallable();
writerControl.replay();
new Foo().foo(writer, doc);
writerControl.verify();
}
This time, it's all straightforward: No Directory created, no IndexWriter
created, no document added. We just tell the mocked-up IndexWriter what
method calls to expect and what to return (or do nothing, or throw an
exception). Also there's no taking notes of added documents because EasyMock
does that already.
The net benefit is that there are no useless objects created. Those could
even harm the test results in the worst case. Also the test should run
faster now, which can really add up when you have hundreds or thousands of
tests.
--
Maik Schreiber * http://www.blizzy.de
GPG public key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1F11D713
Key fingerprint: CF19 AFCE 6E3D 5443 9599 18B5 5640 1F11 D713
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org