You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Joseph Ottinger <jo...@enigmastation.com> on 2005/01/19 20:27:13 UTC
LuceneRAR project announcement
After babbling endlessly about an RDMS directory and my lack of success
with it, I've created a project on java.net to create a Lucene JCA
component, to allow J2EE components to interact with a Lucene service.
It's at https://lucenerar.dev.java.net/ currently.
It's very crude. In current form, in fact, it's able to add a *specific
string* and search with *a specific query*, which is entirely useless -
it's in "the plumbing works!" stage, really.
But I'd like to throw it out for your awareness and consideration, because
while there's definitely a feature list I want to have (add, search, and
delete), I'm definitely cognizant that my use cases may not satisfy the
community-at-large's needs, and thus I'd love to know what you think such
a component should do, and how, if possible.
-----------------------------------------------------------------------
Joseph B. Ottinger http://enigmastation.com
IT Consultant joeo@enigmastation.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: LuceneRAR project announcement
Posted by Joseph Ottinger <jo...@enigmastation.com>.
First off, Erik, thank you for taking an interest in any way. As I've said
before, I'm not trying to represent myelf as a Lucene expert, so having
someone point out flaws is goooooooood.
On Wed, 19 Jan 2005, Erik Hatcher wrote:
> >> Could you elaborate on some use cases?
> >
> > Sure, and I'll pick the one that's been driving me along:
> >
> > I have a set of J2EE servers, all of which can generate new content for
> > search, and all of which will be performing searches. They're on
> > separate
> > machines. Sharing directories isn't my idea of "doing J2EE correctly."
>
> "doing J2EE correctly" is a funny phrase. If sharing directories
> works and gets the job done right, on time, under budget, can be
> adjusted later if needed, and has been reasonably well tested, then
> you've done it "right". And since its in Java and not on a cell phone,
> its basically "J2EE".
Absolutely. I'm not trying to make fun of pragmatic, working solutions.
Nor am I sneering at those who've done it by sharing filesystems or
whatever.
> Also, what about using Lucene over RMI using the RemoteSearchable
> facility built-in?
Well, I'd prefer to avoid RMI. App servers typically have far better
transport layers than raw RMI, internally, and JCA can leverage that.
> > Therefore, I chose to represent Lucene as an enterprise service, one
> > communicated to via a remote service instead, so that every module can
> > communicate with Lucene without realising the communication layer...
> > for
> > the most part.
>
> And this is where I think the abstraction leaks.
>
> The Nutch project has a very scalable "enterprise" approach to this
> type of remote service also.
*nod* I'll look it up.
> > Plus, I no longer violate my purist's sensibilities.
>
> Ah, now we get to the real rationale! :)
>
> I'm not giving you, personally, a hard time, really ... but rather this
> purist approach, where "purist" means fitting into the acronyms under
> the J2EE umbrella. I've been there myself, read the specs, and cringed
> when I saw file system access from a session bean, and so on.
Well, in all honesty, there IS a small factor of "Gee, I can use an
acronym here!" involved. It's not ALL that's involved, of course - I think
the connector's transparency might be a real benefit for others as well as
satisfying my own "I need a deployed component, and not a service I have
to tune" need.
> > The Hits object could CERTAINLY use optimization - callbacks into the
> > connector would probably be acceptable, for example.
>
> Gotcha. Yes, callbacks would be the right approach with this type of
> abstraction.
Just as a general question... is it efficient to retrieve a Document by,
uh, a sort of Lucene key? (Is there such a thing?) If there is, I can code
up a callback procedure in almost no time. (There are some other issues to
address first, but THAT would be easy to do.)
> >> JCA sounds like an unnecessary abstraction around Lucene - though I'm
> >> open to be convinced otherwise.
> >
> > I'm more than happy to talk about it. If I can fulfill my needs with no
> > code, hey, that's great!
>
> Would RemoteSearchable get you closer to no code?
Dunno, I'll investigate.
> > I just haven't been able to successfully do so
> > yet, and everyone to whom I've spoken who says that they HAVE
> > managed...
> > well, they've almost invariably done so by lowering the bar a great
> > deal
> > in order to accept what Lucene requires.
>
> I'm definitely a skeptic when it comes to generic layers on top of
> Lucene, though there is definitely a yearning for easier management of
> the lower-level details.
On my part, too. If I coul get a Directory reliably and quickly using an
RDMS, I'd have gone that route.
> I'll definitely follow your work with LuceneRAR closely and will do
> what I can to help out in this forum. So take my feedback as
> constructive criticism, but keep up the good work!
Again, no problem - and thank you. The things you bring up are issues I
might not be aware of, so it's good to see them and evaluate them.
-----------------------------------------------------------------------
Joseph B. Ottinger http://enigmastation.com
IT Consultant joeo@enigmastation.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: LuceneRAR project announcement
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jan 19, 2005, at 3:30 PM, Joseph Ottinger wrote:
> On Wed, 19 Jan 2005, Erik Hatcher wrote:
>
>> On Jan 19, 2005, at 2:27 PM, Joseph Ottinger wrote:
>>> After babbling endlessly about an RDMS directory and my lack of
>>> success
>>> with it, I've created a project on java.net to create a Lucene JCA
>>> component, to allow J2EE components to interact with a Lucene
>>> service.
>>> It's at https://lucenerar.dev.java.net/ currently.
>>
>> Could you elaborate on some use cases?
>
> Sure, and I'll pick the one that's been driving me along:
>
> I have a set of J2EE servers, all of which can generate new content for
> search, and all of which will be performing searches. They're on
> separate
> machines. Sharing directories isn't my idea of "doing J2EE correctly."
"doing J2EE correctly" is a funny phrase. If sharing directories
works and gets the job done right, on time, under budget, can be
adjusted later if needed, and has been reasonably well tested, then
you've done it "right". And since its in Java and not on a cell phone,
its basically "J2EE".
Also, what about using Lucene over RMI using the RemoteSearchable
facility built-in?
> Therefore, I chose to represent Lucene as an enterprise service, one
> communicated to via a remote service instead, so that every module can
> communicate with Lucene without realising the communication layer...
> for
> the most part.
And this is where I think the abstraction leaks.
The Nutch project has a very scalable "enterprise" approach to this
type of remote service also.
> Plus, I no longer violate my purist's sensibilities.
Ah, now we get to the real rationale! :)
I'm not giving you, personally, a hard time, really ... but rather this
purist approach, where "purist" means fitting into the acronyms under
the J2EE umbrella. I've been there myself, read the specs, and cringed
when I saw file system access from a session bean, and so on.
> The Hits object could CERTAINLY use optimization - callbacks into the
> connector would probably be acceptable, for example.
Gotcha. Yes, callbacks would be the right approach with this type of
abstraction.
>> JCA sounds like an unnecessary abstraction around Lucene - though I'm
>> open to be convinced otherwise.
>
> I'm more than happy to talk about it. If I can fulfill my needs with no
> code, hey, that's great!
Would RemoteSearchable get you closer to no code?
> I just haven't been able to successfully do so
> yet, and everyone to whom I've spoken who says that they HAVE
> managed...
> well, they've almost invariably done so by lowering the bar a great
> deal
> in order to accept what Lucene requires.
I'm definitely a skeptic when it comes to generic layers on top of
Lucene, though there is definitely a yearning for easier management of
the lower-level details.
I'll definitely follow your work with LuceneRAR closely and will do
what I can to help out in this forum. So take my feedback as
constructive criticism, but keep up the good work!
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: LuceneRAR project announcement
Posted by Joseph Ottinger <jo...@enigmastation.com>.
On Wed, 19 Jan 2005, Erik Hatcher wrote:
> On Jan 19, 2005, at 2:27 PM, Joseph Ottinger wrote:
> > After babbling endlessly about an RDMS directory and my lack of success
> > with it, I've created a project on java.net to create a Lucene JCA
> > component, to allow J2EE components to interact with a Lucene service.
> > It's at https://lucenerar.dev.java.net/ currently.
>
> Could you elaborate on some use cases?
Sure, and I'll pick the one that's been driving me along:
I have a set of J2EE servers, all of which can generate new content for
search, and all of which will be performing searches. They're on separate
machines. Sharing directories isn't my idea of "doing J2EE correctly."
Therefore, I chose to represent Lucene as an enterprise service, one
communicated to via a remote service instead, so that every module can
communicate with Lucene without realising the communication layer... for
the most part. Plus, I no longer violate my purist's sensibilities.
> What drove you to consider JCA rather than some other technique? I'm
> curious why it is important to get all J2EE with it rather than working
> with Lucene much more naturally at a lower level of abstraction.
JCA allows me to provide it as a system service instead of as a dependency
represented at each component layer. An EJB would have served almost as
well, except an EJB has filesystem restrictions that a Connector does not.
> I briefly browsed the source tree from java.net and saw this comment in
> your Hits.java:
>
> "This method loads a LuceneRAR hits object with its equivalent from the
> Apache Lucene Hits object. It basically walks the Lucene Hits object,
> copying values as it goes, so it may not be as light or fast as its
> Apache equivalent"
>
> I'll say!
Haha, it's good to see my propensity for understatement is still alive. :)
The Hits object could CERTAINLY use optimization - callbacks into the
connector would probably be acceptable, for example. The code you were
looking at has a lot of other areas that are, um, surprisingly crippled as
well.
For example, the add() method... well, first, THAT's the signature. Yes,
that's right. It adds constant text. Every time.
Likewise, the super-flexible "search()" -- again, that's the signature. It
searches for "time." That's it. Nothing more. Nothing less.
This is very much a first-cut "can I get it working?" version. I think,
for very limited definitions of "working," the answer is "yes." I
certainly don't think it's got that show-room floor gleam going for it
yet.
> For large result sets, which are more often the norm than the exception
> for a search, you are going to take a huge performance hit doing
> something like this, not to mention possibly even killing the process
> as you run out of RAM.
*nod* As stated, a callback would be far more preferable. Given that
Lucene's internal Hits object is final and nonserializable, at least my
client's Hit object gives me an opportunity to do that.
> JCA sounds like an unnecessary abstraction around Lucene - though I'm
> open to be convinced otherwise.
I'm more than happy to talk about it. If I can fulfill my needs with no
code, hey, that's great! I just haven't been able to successfully do so
yet, and everyone to whom I've spoken who says that they HAVE managed...
well, they've almost invariably done so by lowering the bar a great deal
in order to accept what Lucene requires.
I'm certainly not castigating those who've done this - in fact, in many
ways, I'm very impressed. It's just something I'd prefer not to do, given
any alternative.
-----------------------------------------------------------------------
Joseph B. Ottinger http://enigmastation.com
IT Consultant joeo@enigmastation.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: LuceneRAR project announcement
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jan 19, 2005, at 2:27 PM, Joseph Ottinger wrote:
> After babbling endlessly about an RDMS directory and my lack of success
> with it, I've created a project on java.net to create a Lucene JCA
> component, to allow J2EE components to interact with a Lucene service.
> It's at https://lucenerar.dev.java.net/ currently.
Could you elaborate on some use cases?
What drove you to consider JCA rather than some other technique? I'm
curious why it is important to get all J2EE with it rather than working
with Lucene much more naturally at a lower level of abstraction.
I briefly browsed the source tree from java.net and saw this comment in
your Hits.java:
"This method loads a LuceneRAR hits object with its equivalent from the
Apache Lucene Hits object. It basically walks the Lucene Hits object,
copying values as it goes, so it may not be as light or fast as its
Apache equivalent"
I'll say!
For large result sets, which are more often the norm than the exception
for a search, you are going to take a huge performance hit doing
something like this, not to mention possibly even killing the process
as you run out of RAM.
This brings me back to my first questions - abstractions around Lucene
tend to leak heavily. While it sounds clever to wrap layers around
Hits, the fact of the matter is that searches often return an enormous
amount of results and only the first few are generally needed.
Lucene's Hits takes this into account and fetches on demand from the
index.
Admittedly for Java Development with Ant, I implemented a stateless
session bean that walked the hits and packaged them up to send across
the wire. This was naive and only worked because I never tried it with
a large number of hits. These days I push back from "J2EE" in the
larger "let's add acronyms because we can" sense and opt for much
lighter weight, simpler solutions.
JCA sounds like an unnecessary abstraction around Lucene - though I'm
open to be convinced otherwise.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org