Solr on AppEngine - patches? & a question

[Nick Lothian <ni...@gmail.com>]

I've got Solr working on Google AppEngine (mostly, anyway - it's read only
ATM, and I haven't done any stability testing).

For those of you who aren't aware, AppEngine doesn't allow file access,
while Solr is fairly dependent on file system access for configuration
files.

Solr allowed me to work around the majority of the file system dependencies
(by implementing a custom ResourceLoader etc), but I needed to alter
SolrCore.java, SolrResourceLoader.java and SolrDispatchFilter.java

Is there any interest in patches to make these classes extensible enough for
use in this scenario?

Secondly, AppEngine also doesn't allow the creation of threads. This was
mostly easy to work around (I created a ExecutorService using the current
thread), except for the CountDownLatch locking stuff in the SolrCore
constructor:

      searcherExecutor.submit(new Callable() {
        public Object call() throws Exception {
          latch.await();
          return null;
        }
      });

I don't completely understand what's going on here. My understanding is that
it's locking the search executor so that it won't run until all the
searchers are correctly initialized, and I'm guessing that SolrCore handles
the case where simultaneous requests to the filter come in
before initialization is complete somehow.

Since I'm running single threaded here, I've worked around that by
commenting it out. Is that sufficient or is there a better way?

Nick

Re: Solr on AppEngine - patches? & a question

[Nick Lothian <ni...@gmail.com>]

The index is a custom directory implementation taken (mostly) from Compass
GAE (which is Apache 2 licensed):
http://fisheye.compass-project.org/fisheye/browse/compass/trunk/src/main/src/org/compass/needle/gae/GoogleAppEngineDirectory.java?r=trunk

I'll try & put some patches together once I get it more stable - my code
works pretty well in the local development AppEngine implementation, but the
real AppEngine is still causing more problems that I'd like.

On Fri, May 7, 2010 at 4:43 AM, Chris Hostetter <ho...@fucit.org>wrote:

>
> : For those of you who aren't aware, AppEngine doesn't allow file access,
>
> I compeltey forgot to ask in my last reply: how are you dealing with the
> index itself?  did you write a custom Directory implementation for dealing
> with the AppEngine datastorage system?
>
> that would also probably be of interest to many people.
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: Solr on AppEngine - patches? & a question

[Chris Hostetter <ho...@fucit.org>]

: For those of you who aren't aware, AppEngine doesn't allow file access,

I compeltey forgot to ask in my last reply: how are you dealing with the 
index itself?  did you write a custom Directory implementation for dealing 
with the AppEngine datastorage system?

that would also probably be of interest to many people.

-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: Solr on AppEngine - patches? & a question

[Chris Hostetter <ho...@fucit.org>]

: Is there any interest in patches to make these classes extensible enough for
: use in this scenario?

I think it would definitely be informative for a lot of people if you 
would be willing/interested in posting your patches -- even if we 
can't/don't commit them for some reason, they can still serve as 
informative starting points for other people, and they might help us spot 
ways the code can be refactored/generalized to make your situation easier.

: Secondly, AppEngine also doesn't allow the creation of threads. This was
: mostly easy to work around (I created a ExecutorService using the current
: thread), except for the CountDownLatch locking stuff in the SolrCore
: constructor:
	...
: I don't completely understand what's going on here. My understanding is that
: it's locking the search executor so that it won't run until all the
: searchers are correctly initialized, and I'm guessing that SolrCore handles
: the case where simultaneous requests to the filter come in
: before initialization is complete somehow.

not exactly -- external requests won't be sent to the core until the 
constructor has finished, but "firstSearcher" queries configured in the 
solrconfig.xml are executed internally, and those need to be blocked 
untill all the components have been initialized.

: Since I'm running single threaded here, I've worked around that by
: commenting it out. Is that sufficient or is there a better way?

I'm not sure ... presumable your "ExecutorService using the current 
thread" approach wouldn't work here since it would defeat the point of 
blocking (but it would be no worse then commenting out -- that's one 
generalization we might support, a way to configure the ExecutorService 
used).  

I have no idea if there would be a better way of dealing with that.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org