You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Kerang Lv <lv...@yahoo.com> on 2009/09/14 18:04:27 UTC
Run your Lucene Applications on Google AppEngine with GAELucene
Hi Lucene users,
Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
The main clazz of GAELucene include:
* GAEDirectory - a read only Directory based on google datastore.
* GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
* GAEFileContent - stands for a segment of index file.
* GAECategory - the identifier of different indices.
* GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
* GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
* GAEIndexReaderPool - pool for GAEIndexReader
The following code snippet demonstrates the use of GAELucene do searching:
Query queryObject = parserQuery(request);
GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
IndexSearcher searcher =newIndexSearcher(indexReader);
Hits hits = searcher.search(queryObject);
readerPool.returnReader(indexReader);
You can find more here 'http://code.google.com/p/gaelucene/'
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Kerang Lv <lv...@yahoo.com>.
>Do you plan to support in memory indexes using the memcache api?
I'm afraid not, I prefer to do indexing on another machine before I got a plan that can finish indexing within 30s.
----- Original Message ----
From: Erdinc Yilmazel <er...@gmail.com>
To: java-user@lucene.apache.org
Sent: Tuesday, September 15, 2009 6:05:15 PM
Subject: Re: Run your Lucene Applications on Google AppEngine with GAELucene
This is great news! Are you happy with the the performance of the google
data store? Do you plan to support in memory indexes using the memcache api?
Thanks
On Mon, Sep 14, 2009 at 5:04 PM, Kerang Lv <lv...@yahoo.com> wrote:
> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [
> http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html],
> I implemented a google datastore based Lucene component, GAELucene, which
> can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:
> * GAEDirectory - a read only Directory based on google datastore.
> * GAEFile - stands for an index file, the file's byte content will be
> splited into multi GAEFileContent.
> * GAEFileContent - stands for a segment of index file.
> * GAECategory - the identifier of different indices.
> * GAEIndexInput - a memory-resident IndexInput? implementation like the
> RAMInputStream.
> * GAEIndexReader - wrapper for IndexReader? that cached in
> GAEIndexReaderPool
> * GAEIndexReaderPool - pool for GAEIndexReader
>
> The following code snippet demonstrates the use of GAELucene do searching:
> Query queryObject = parserQuery(request);
> GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
> GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
> IndexSearcher searcher =newIndexSearcher(indexReader);
> Hits hits = searcher.search(queryObject);
> readerPool.returnReader(indexReader);
>
> You can find more here 'http://code.google.com/p/gaelucene/'
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Erdinc Yilmazel <er...@gmail.com>.
This is great news! Are you happy with the the performance of the google
data store? Do you plan to support in memory indexes using the memcache api?
Thanks
On Mon, Sep 14, 2009 at 5:04 PM, Kerang Lv <lv...@yahoo.com> wrote:
> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [
> http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html],
> I implemented a google datastore based Lucene component, GAELucene, which
> can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:
> * GAEDirectory - a read only Directory based on google datastore.
> * GAEFile - stands for an index file, the file's byte content will be
> splited into multi GAEFileContent.
> * GAEFileContent - stands for a segment of index file.
> * GAECategory - the identifier of different indices.
> * GAEIndexInput - a memory-resident IndexInput? implementation like the
> RAMInputStream.
> * GAEIndexReader - wrapper for IndexReader? that cached in
> GAEIndexReaderPool
> * GAEIndexReaderPool - pool for GAEIndexReader
>
> The following code snippet demonstrates the use of GAELucene do searching:
> Query queryObject = parserQuery(request);
> GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
> GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
> IndexSearcher searcher =newIndexSearcher(indexReader);
> Hits hits = searcher.search(queryObject);
> readerPool.returnReader(indexReader);
>
> You can find more here 'http://code.google.com/p/gaelucene/'
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Kerang Lv <lv...@yahoo.com>.
>I think I will try this today evening.
Remember to update your local project from the svn, I fixed some mistakes just now. I apologize for my negligence.
>I think we should put this as one of component in lucene-contrib. What do you say?
Yes, that's a good news.
----- Original Message ----
From: Allahbaksh Mohammedali Asadullah <Al...@infosys.com>
To: "java-user@lucene.apache.org" <ja...@lucene.apache.org>
Sent: Tuesday, September 15, 2009 12:39:16 PM
Subject: RE: Run your Lucene Applications on Google AppEngine with GAELucene
Hi,
This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
Regards,
Allahbaksh
-----Original Message-----
From: Kerang Lv [mailto:lvkrnewer@yahoo.com]
Sent: Monday, September 14, 2009 9:34 PM
To: java-user@lucene.apache.org
Subject: Run your Lucene Applications on Google AppEngine with GAELucene
Hi Lucene users,
Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
The main clazz of GAELucene include:
* GAEDirectory - a read only Directory based on google datastore.
* GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
* GAEFileContent - stands for a segment of index file.
* GAECategory - the identifier of different indices.
* GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
* GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
* GAEIndexReaderPool - pool for GAEIndexReader
The following code snippet demonstrates the use of GAELucene do searching:
Query queryObject = parserQuery(request);
GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
IndexSearcher searcher =newIndexSearcher(indexReader);
Hits hits = searcher.search(queryObject);
readerPool.returnReader(indexReader);
You can find morere 'http://code.google.com/p/gaelucene/'
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Kerang Lv <lv...@yahoo.com>.
Hi Daniel Shane,
The GAEFile/GAEIndexInput was a bit like the RAMFile/RAMInputStream, it can hold the file's byte data in memory, so the performance should be acceptable once most file segments were loaded from the google datastore.
Wish your test result.
----- Original Message ----
From: Daniel Shane <sh...@LEXUM.UMontreal.CA>
To: java-user@lucene.apache.org
Sent: Wednesday, September 16, 2009 10:33:32 PM
Subject: Re: Run your Lucene Applications on Google AppEngine with GAELucene
I question the performance of such an approach. For lucene to be fast, disk access need to be fast, and the transaction stuff with google is not that good.
I'll have to test it out to see, but I anticipate a huge performance hit compared to lucene running with a real HDD access.
Daniel Shane
Allahbaksh Mohammedali Asadullah wrote:
> Hi,
> This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
> Regards,
> Allahbaksh
> -----Original Message-----
> From: Kerang Lv [mailto:lvkrnewer@yahoo.com] Sent: Monday, September 14, 2009 9:34 PM
> To: java-user@lucene.apache.org
> Subject: Run your Lucene Applications on Google AppEngine with GAELucene
>
> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include: * GAEDirectory - a read only Directory based on google datastore. * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent. * GAEFileContent - stands for a segment of index file. * GAECategory - the identifier of different indices. * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream. * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool * GAEIndexReaderPool - pool for GAEIndexReader
> The following code snippet demonstrates the use of GAELucene do searching: Query queryObject = parserQuery(request);
> GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
> GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
> IndexSearcher searcher =newIndexSearcher(indexReader);
> Hits hits = searcher.search(queryObject);
> readerPool.returnReader(indexReader);
>
> You can find morere 'http://code.google.com/p/gaelucene/'
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Daniel Shane <sh...@LEXUM.UMontreal.CA>.
I question the performance of such an approach. For lucene to be fast,
disk access need to be fast, and the transaction stuff with google is
not that good.
I'll have to test it out to see, but I anticipate a huge performance hit
compared to lucene running with a real HDD access.
Daniel Shane
Allahbaksh Mohammedali Asadullah wrote:
> Hi,
> This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
> Regards,
> Allahbaksh
>
>
>
> -----Original Message-----
> From: Kerang Lv [mailto:lvkrnewer@yahoo.com]
> Sent: Monday, September 14, 2009 9:34 PM
> To: java-user@lucene.apache.org
> Subject: Run your Lucene Applications on Google AppEngine with GAELucene
>
> Hi Lucene users,
>
> Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
>
> The main clazz of GAELucene include:
> * GAEDirectory - a read only Directory based on google datastore.
> * GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
> * GAEFileContent - stands for a segment of index file.
> * GAECategory - the identifier of different indices.
> * GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
> * GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
> * GAEIndexReaderPool - pool for GAEIndexReader
>
> The following code snippet demonstrates the use of GAELucene do searching:
> Query queryObject = parserQuery(request);
> GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
> GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
> IndexSearcher searcher =newIndexSearcher(indexReader);
> Hits hits = searcher.search(queryObject);
> readerPool.returnReader(indexReader);
>
> You can find morere 'http://code.google.com/p/gaelucene/'
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Allahbaksh Mohammedali Asadullah <Al...@infosys.com>.
HI Mike,
I think adding this in Lucene 3.0 contrib would be the best we could do. I think we could add it in Lucene 2.9 Release as it would grow the community and we would also able to find some nice practices, bugs, improvement and that would make it better in upcoming release.
Regards,
Allahbaksh
-----Original Message----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Tuesday, September 15, 2009 3:25 PM
To: java-user@lucene.apache.org
Subject: Re: Run your Lucene Applications on Google AppEngine with GAELucene
On Tue, Sep 15, 2009 at 12:39 AM, Allahbaksh Mohammedali Asadullah
<Al...@infosys.com> wrote:
> I think we should put this as one of component in lucene-contrib.
+1, this looks like a great contribution!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Sep 15, 2009 at 12:39 AM, Allahbaksh Mohammedali Asadullah
<Al...@infosys.com> wrote:
> I think we should put this as one of component in lucene-contrib.
+1, this looks like a great contribution!
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Run your Lucene Applications on Google AppEngine with GAELucene
Posted by Allahbaksh Mohammedali Asadullah <Al...@infosys.com>.
Hi,
This is great news and good work. I think I will try this today evening. I think we should put this as one of component in lucene-contrib. What do you say? Committer and owner please comment.
Regards,
Allahbaksh
-----Original Message-----
From: Kerang Lv [mailto:lvkrnewer@yahoo.com]
Sent: Monday, September 14, 2009 9:34 PM
To: java-user@lucene.apache.org
Subject: Run your Lucene Applications on Google AppEngine with GAELucene
Hi Lucene users,
Enlightened by the discussion "Can I run Lucene in google app engine? [http://www.nabble.com/Can-I-run-Lucene-in-google-app-engine--td23017742.html], I implemented a google datastore based Lucene component, GAELucene, which can help you to run search applications on google app engine.
The main clazz of GAELucene include:
* GAEDirectory - a read only Directory based on google datastore.
* GAEFile - stands for an index file, the file's byte content will be splited into multi GAEFileContent.
* GAEFileContent - stands for a segment of index file.
* GAECategory - the identifier of different indices.
* GAEIndexInput - a memory-resident IndexInput? implementation like the RAMInputStream.
* GAEIndexReader - wrapper for IndexReader? that cached in GAEIndexReaderPool
* GAEIndexReaderPool - pool for GAEIndexReader
The following code snippet demonstrates the use of GAELucene do searching:
Query queryObject = parserQuery(request);
GAEIndexReaderPool readerPool =GAEIndexReaderPool.getInstance();
GAEIndexReader indexReader = readerPool.borrowReader(INDEX_CATEGORY_DEMO);
IndexSearcher searcher =newIndexSearcher(indexReader);
Hits hits = searcher.search(queryObject);
readerPool.returnReader(indexReader);
You can find morere 'http://code.google.com/p/gaelucene/'
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org