You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucy.apache.org by "Andrew S. Townley" <as...@atownley.org> on 2011/02/25 15:56:44 UTC
[lucy-user] Mostly re: the SWIG part (was Re: [lucy-user] Question about query parsing API
Hi Jens,
On 25 Feb 2011, at 1:41 PM, Jens Wille wrote:
>
> Andrew S. Townley [2011-02-25 13:41]:
>> If so, is there a way to traverse the query parse tree (or get a
>> callback notification when it matches terms)? What I need to be
>> able to do is take an arbitrary fulltext search query and then
>> determine whether any search fields have been requested
>> explicitly. I suppose a more ideal solution would be to have a
>> #target_fields method or something on the query object, but that
>> would just be convenience.
> i understand that problem has been solved in the meantime
> (Ferret::Search::Query#terms, basically)? ;-)
Yes, it seems like you did that as a byproduct of your change to Ferret (https://github.com/blackwinter/ferret). Doubly awesome. Thanks!
>> Also, I mentioned SWIG in passing the other day in a previous
>> message. Would it not be possible to just generate the bindings
>> for Ruby with SWIG?
> re SWIG, there was a clarifying statement by marvin humphrey on
> lucy-dev recently:
>
> <http://mail-archives.apache.org/mod_mbox/incubator-lucy-dev/201102.mbox/%3C20110221215214.GA27653@rectangular.com%3E>
Thanks for the pointer. Archive failure on my part. However, after reading that and this http://s.apache.org/pSd (referenced in the above), I understand the rationale, and think it's cool, but am slightly concerned about how easy it'll be to get it to generate Ruby.
To the larger Lucy community:
I can't find any documentation about Clownfish other than the above JIRA entry. Is there some discussion/project/docs somewhere?
The advantages of SWIG are that it's pretty widely used, and anyone with experience with both the source and the target binding language can usually get something working in hours, not days. It seems to me that for lucy to get the adoption it needs to grow communities in other circles besides Perl, you need to start doing some marketing about how much easier it is for Joe-random-developer to take a Clownfish-enabled project and automatically generate the language bindings of their choice.
As it stands, it seems that anyone interested in using lucy outside of Perl (or actually, just as a C library) needs to:
a) Learn Clownfish
b) Implement Clownfish -> Lang conversion routines (in Perl?)
c) Run it on the lucy source
d) write binding tests
e) tweak bindings and/or conversion, e.g. goto b
f) rinse & repeat
I really like the mechanisms you're using in Clownfish--I really do, but it isn't going to matter much (except for KinoSearch/Perl users) if there aren't people able to easily create language bindings--no matter how much more powerful the plumbing is than SWIG or other approaches.
Please understand I'm not trying to be negative here. I really want Lucy to succeed in its goals. I'm just trying to get a realistic picture of where things are and what's actually involved. I think finishing the incubation process and getting more user and host language developer documentation in place (even a few example blog posts) would go a long way to attracting some support. Right now, except for the mailing list, lucy's just a black box that sounds like it'll be cool--eventually (at least for non-Perl users). ;)
If these documentation resources, Clownfish intro materials or HOW-TO guides exist, then please point me to them. Google wasn't much help here, and I've been through the SVN browser several times now with no luck.
It seems like the core architecture of Lucy and Clownfish are first-rate efforts. Now, you need to think about the right way to market it so that you can build the community. Again, it looks very much like Lucy is the long-term fit for what I'm working on, so I'm very interested in how things develop and what I might have to bite off in terms of additional work to integrate it at any level (C++ or Ruby) into what I have.
Thanks for your time,
ast
--
Andrew S. Townley <as...@atownley.org>
http://atownley.org
Re: [lucy-user] Mostly re: the SWIG part (was Re: [lucy-user] Question
about query parsing API
Posted by Peter Karman <pe...@peknet.com>.
Andrew S. Townley wrote on 2/26/11 7:26 AM:
> I get what you're saying, but here's the rub for me: I'm not really a
> developer anymore--I just play one on TV. ;) At the moment, I'm wearing
> about 10 different hats, and I'm trying to stabilize my own platform so I can
> divest myself of as many development responsibilities as possible. If I
> don't, I won't be able to do a lot of the other work that I need to be
> doing.
>
> I'm confident that I *could* do the Ruby bindings, I'm just not sure that
> taking on yet another development task is very smart right now. Puts me in a
> rather unpleasant dilemma, actually, because I want to help, and I suspect
> that I'm going to hit more issues with Ferret as I go along since it doesn't
> seem to be designed exactly for how I want/need to use it. Unfortunately, I
> can't make any more promises at the moment. :(
>
Completely fair and I appreciate your self-knowledge on this score.
Please do continue to lurk, at least, and consider reviewing the Ruby binding
RFCs when they emerge. One of our primary goals is idiomatically consistent APIs
for each host language, and it helps immensely to have "native speakers"
reviewing the idioms. :)
pek
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-user] Mostly re: the SWIG part (was Re: [lucy-user] Question about query parsing API
Posted by "Andrew S. Townley" <as...@atownley.org>.
On 25 Feb 2011, at 7:45 PM, Nathan Kurz wrote:
> On Fri, Feb 25, 2011 at 6:56 AM, Andrew S. Townley <as...@atownley.org> wrote:
>> The advantages of SWIG are that it's pretty widely used, and anyone with experience with both the source and the target binding language can usually get something working in hours, not days. It seems to me that for lucy to get the adoption it needs to grow communities in other circles besides Perl, you need to start doing some marketing about how much easier it is for Joe-random-developer to take a Clownfish-enabled project and automatically generate the language bindings of their choice.
>
> As Marvin's lost post claimed, SWIG is not really a good fit here.
I wasn't really advocating SWIG as a solution with the above; more that it's a known beast and people can create bindings for something they don't know the internals for relatively easily. I get why you've gone with something like Clownfish.
>> a) Learn Clownfish
>> b) Implement Clownfish -> Lang conversion routines (in Perl?)
>> c) Run it on the lucy source
>> d) write binding tests
>> e) tweak bindings and/or conversion, e.g. goto b
>> f) rinse & repeat
>
> Essentially correct, although it will soon be possible to do this with
> only C and the target language (no Perl required). Also on the bright
> side, most of the heavy lifting only has to be once for Clownfish,
> rather than once per project. Unfortunately, this lifting is heavy
> and has yet to be done, and will initially require working closely
> with Marvin.
>
> Realistically, your hypothetical Joe-random-developer isn't going to
> be able to do this. While turnkey documentation would help, that time
> would probably be better spent just making/polishing the bindings
> themselves for the most likely languages, which I'd guess would be
> Perl, Ruby, and Python. The full documentation can follow after these
> are solid.
>
>> I really like the mechanisms you're using in Clownfish--I really do, but it isn't going to matter much (except for KinoSearch/Perl users) if there aren't people able to easily create language bindings--no matter how much more powerful the plumbing is than SWIG or other approaches.
>
> I think you're on the right path, but the bar is actually a bit
> higher. Lucy is not going to get significant traction in any language
> until the bindings are sufficiently solid that Clownfish doesn't even
> come up. I think it's a great tool for the purpose, but it's a core
> developers tool rather than an end-user tool. It's more akin to
> porting SWIG to a new language than merely adding a SWIG interface.
>
> What's needed is a few more core developers (such as yourself) with a
> strong language-specific itch and a willingness to dive into the C.
> Once that works done (once), Lucy should start to spread pretty
> easily.
I get what you're saying, but here's the rub for me: I'm not really a developer anymore--I just play one on TV. ;) At the moment, I'm wearing about 10 different hats, and I'm trying to stabilize my own platform so I can divest myself of as many development responsibilities as possible. If I don't, I won't be able to do a lot of the other work that I need to be doing.
I'm confident that I *could* do the Ruby bindings, I'm just not sure that taking on yet another development task is very smart right now. Puts me in a rather unpleasant dilemma, actually, because I want to help, and I suspect that I'm going to hit more issues with Ferret as I go along since it doesn't seem to be designed exactly for how I want/need to use it. Unfortunately, I can't make any more promises at the moment. :(
ast
--
Andrew S. Townley <as...@atownley.org>
http://atownley.org
Re: [lucy-user] Mostly re: the SWIG part (was Re: [lucy-user]
Question about query parsing API
Posted by Nathan Kurz <na...@verse.com>.
On Fri, Feb 25, 2011 at 6:56 AM, Andrew S. Townley <as...@atownley.org> wrote:
> The advantages of SWIG are that it's pretty widely used, and anyone with experience with both the source and the target binding language can usually get something working in hours, not days. It seems to me that for lucy to get the adoption it needs to grow communities in other circles besides Perl, you need to start doing some marketing about how much easier it is for Joe-random-developer to take a Clownfish-enabled project and automatically generate the language bindings of their choice.
As Marvin's lost post claimed, SWIG is not really a good fit here.
> a) Learn Clownfish
> b) Implement Clownfish -> Lang conversion routines (in Perl?)
> c) Run it on the lucy source
> d) write binding tests
> e) tweak bindings and/or conversion, e.g. goto b
> f) rinse & repeat
Essentially correct, although it will soon be possible to do this with
only C and the target language (no Perl required). Also on the bright
side, most of the heavy lifting only has to be once for Clownfish,
rather than once per project. Unfortunately, this lifting is heavy
and has yet to be done, and will initially require working closely
with Marvin.
Realistically, your hypothetical Joe-random-developer isn't going to
be able to do this. While turnkey documentation would help, that time
would probably be better spent just making/polishing the bindings
themselves for the most likely languages, which I'd guess would be
Perl, Ruby, and Python. The full documentation can follow after these
are solid.
> I really like the mechanisms you're using in Clownfish--I really do, but it isn't going to matter much (except for KinoSearch/Perl users) if there aren't people able to easily create language bindings--no matter how much more powerful the plumbing is than SWIG or other approaches.
I think you're on the right path, but the bar is actually a bit
higher. Lucy is not going to get significant traction in any language
until the bindings are sufficiently solid that Clownfish doesn't even
come up. I think it's a great tool for the purpose, but it's a core
developers tool rather than an end-user tool. It's more akin to
porting SWIG to a new language than merely adding a SWIG interface.
What's needed is a few more core developers (such as yourself) with a
strong language-specific itch and a willingness to dive into the C.
Once that works done (once), Lucy should start to spread pretty
easily.
Nathan Kurz
nate@verse.com