You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by danben <da...@gmail.com> on 2009/07/31 21:55:38 UTC
Problem After Modifying CoreContainer
Hi,
I'm developing an application that requires a large number of cores, and
since lazy loading / LRU caching won't be available until 1.5, I decided to
modify CoreContainer to hold me over.
Another requirement is that multiple Solr instances can access the same
cores (on NAS, for instance), so the approach I'm using is to maintain a
local registry / load balancer that assigns "active" cores to different
machines and, when a machine has exceeded its limit, unload cores in LRU
order.
The modifications to CoreContainer are as follows: to avoid having to issue
"create" requests every time we need to load an inactive core, getCore will
attempt to create/open any core that it doesn't find in the cores map,
unless name is "" or "admin", in which case it returns null as per the
original implementation. The create function is overloaded to take a core
name and creates a CoreDescriptor using some defaults added to solr.xml.
Everything works fine until I try to make a core unload request, at which
point I see the following:
org.apache.solr.common.SolrException: Not Found
Not Found
request:
/solr/NewUser0/admin/cores?action=UNLOAD&core=NewUser0&wt=javabin&version=2.2
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:343)
So, my guess is that my modifications are keeping admin cores from being
created properly. Or maybe I don't know what I'm talking about. I stepped
through execution with a debugger and watched it bounce around
SolrDispatchHandler.doFilter() before giving up and forwarding the request
to the RequestDispatcher, but all I could tell from that was that it wanted
something it couldn't find. Can anyone shed some light on what my mods
might be keeping Solr from doing that would cause this problem?
Code pasted below. And don't mind the weird locking, this happens even with
a single thread.
Thanks,
Dan
public SolrCore getCore(String name) {
ReentrantReadWriteLock lock = getCoreLock(name);
try {
lock.readLock().lock();
SolrCore core = cores.get(name);
if (core != null) {
core.open(); // increment the ref count while still
synchronized
return core;
} else if ("".equals(name) || "admin".equals(name)) {
return null;
} else {
try {
lock.readLock().unlock();
lock.writeLock().lock();
SolrCore core1 = cores.get(name);
if (core1 != null) {
return core1;
}
log.info("Autocreating core: '" + name + "'");
core = create(name);
cores.put(name, core);
core.open();
return core;
} catch (IOException e) {
log.error("Autocreating core '" + name + "'", e);
return null;
} catch (ParserConfigurationException e) {
log.error("Autocreating core '" + name + "'", e);
return null;
} catch (SAXException e) {
log.error("Autocreating core '" + name + "'", e);
return null;
} finally {
lock.readLock().lock();
lock.writeLock().unlock();
}
}
} finally {
lock.readLock().unlock();
}
}
public SolrCore create(String coreName) throws IOException,
ParserConfigurationException, SAXException {
if (defaultConfigFile == null || defaultSchemaFile == null) {
throw new RuntimeException("Cannot use autocreate unless both a
default configuration file and a default schema file are specified");
}
CoreDescriptor dcore = new CoreDescriptor(this, coreName,
getInstanceDir(coreName));
dcore.setConfigName(defaultConfigFile);
dcore.setSchemaName(defaultSchemaFile);
return create(dcore);
}
// Eventually this will be overridden to do some intelligent management of a
directory hierarchy so we //don't have hundreds of thousands of cores in the
same directory
public String getInstanceDir(String coreName) {
return coreName;
}
--
View this message in context: http://www.nabble.com/Problem-After-Modifying-CoreContainer-tp24762199p24762199.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem After Modifying CoreContainer
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
did you take a look at https://issues.apache.org/jira/browse/SOLR-1293
which already handles this
On Sat, Aug 1, 2009 at 2:40 AM, danben<da...@gmail.com> wrote:
>
> And, re-examining the URL, this is clearly my fault for improper use of
> SolrJ. Please ignore.
>
>
> danben wrote:
>>
>> Hi,
>>
>> I'm developing an application that requires a large number of cores, and
>> since lazy loading / LRU caching won't be available until 1.5, I decided
>> to modify CoreContainer to hold me over.
>>
>> Another requirement is that multiple Solr instances can access the same
>> cores (on NAS, for instance), so the approach I'm using is to maintain a
>> local registry / load balancer that assigns "active" cores to different
>> machines and, when a machine has exceeded its limit, unload cores in LRU
>> order.
>>
>> The modifications to CoreContainer are as follows: to avoid having to
>> issue "create" requests every time we need to load an inactive core,
>> getCore will attempt to create/open any core that it doesn't find in the
>> cores map, unless name is "" or "admin", in which case it returns null as
>> per the original implementation. The create function is overloaded to
>> take a core name and creates a CoreDescriptor using some defaults added to
>> solr.xml.
>>
>> Everything works fine until I try to make a core unload request, at which
>> point I see the following:
>> org.apache.solr.common.SolrException: Not Found
>>
>> Not Found
>>
>> request:
>> /solr/NewUser0/admin/cores?action=UNLOAD&core=NewUser0&wt=javabin&version=2.2
>> at
>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:343)
>>
>> So, my guess is that my modifications are keeping admin cores from being
>> created properly. Or maybe I don't know what I'm talking about. I
>> stepped through execution with a debugger and watched it bounce around
>> SolrDispatchHandler.doFilter() before giving up and forwarding the request
>> to the RequestDispatcher, but all I could tell from that was that it
>> wanted something it couldn't find. Can anyone shed some light on what my
>> mods might be keeping Solr from doing that would cause this problem?
>>
>> Code pasted below. And don't mind the weird locking, this happens even
>> with a single thread.
>>
>> Thanks,
>> Dan
>>
>> public SolrCore getCore(String name) {
>> ReentrantReadWriteLock lock = getCoreLock(name);
>> try {
>> lock.readLock().lock();
>> SolrCore core = cores.get(name);
>> if (core != null) {
>> core.open(); // increment the ref count while still
>> synchronized
>> return core;
>> } else if ("".equals(name) || "admin".equals(name)) {
>> return null;
>> } else {
>> try {
>> lock.readLock().unlock();
>> lock.writeLock().lock();
>> SolrCore core1 = cores.get(name);
>> if (core1 != null) {
>> return core1;
>> }
>> log.info("Autocreating core: '" + name + "'");
>> core = create(name);
>> cores.put(name, core);
>> core.open();
>> return core;
>> } catch (IOException e) {
>> log.error("Autocreating core '" + name + "'", e);
>> return null;
>> } catch (ParserConfigurationException e) {
>> log.error("Autocreating core '" + name + "'", e);
>> return null;
>> } catch (SAXException e) {
>> log.error("Autocreating core '" + name + "'", e);
>> return null;
>> } finally {
>> lock.readLock().lock();
>> lock.writeLock().unlock();
>> }
>> }
>> } finally {
>> lock.readLock().unlock();
>> }
>> }
>>
>> public SolrCore create(String coreName) throws IOException,
>> ParserConfigurationException, SAXException {
>> if (defaultConfigFile == null || defaultSchemaFile == null) {
>> throw new RuntimeException("Cannot use autocreate unless both a
>> default configuration file and a default schema file are specified");
>> }
>> CoreDescriptor dcore = new CoreDescriptor(this, coreName,
>> getInstanceDir(coreName));
>> dcore.setConfigName(defaultConfigFile);
>> dcore.setSchemaName(defaultSchemaFile);
>> return create(dcore);
>> }
>>
>> // Eventually this will be overridden to do some intelligent management of
>> a directory hierarchy so we //don't have hundreds of thousands of cores in
>> the same directory
>> public String getInstanceDir(String coreName) {
>> return coreName;
>> }
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Problem-After-Modifying-CoreContainer-tp24762199p24763149.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Problem After Modifying CoreContainer
Posted by danben <da...@gmail.com>.
And, re-examining the URL, this is clearly my fault for improper use of
SolrJ. Please ignore.
danben wrote:
>
> Hi,
>
> I'm developing an application that requires a large number of cores, and
> since lazy loading / LRU caching won't be available until 1.5, I decided
> to modify CoreContainer to hold me over.
>
> Another requirement is that multiple Solr instances can access the same
> cores (on NAS, for instance), so the approach I'm using is to maintain a
> local registry / load balancer that assigns "active" cores to different
> machines and, when a machine has exceeded its limit, unload cores in LRU
> order.
>
> The modifications to CoreContainer are as follows: to avoid having to
> issue "create" requests every time we need to load an inactive core,
> getCore will attempt to create/open any core that it doesn't find in the
> cores map, unless name is "" or "admin", in which case it returns null as
> per the original implementation. The create function is overloaded to
> take a core name and creates a CoreDescriptor using some defaults added to
> solr.xml.
>
> Everything works fine until I try to make a core unload request, at which
> point I see the following:
> org.apache.solr.common.SolrException: Not Found
>
> Not Found
>
> request:
> /solr/NewUser0/admin/cores?action=UNLOAD&core=NewUser0&wt=javabin&version=2.2
> at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:343)
>
> So, my guess is that my modifications are keeping admin cores from being
> created properly. Or maybe I don't know what I'm talking about. I
> stepped through execution with a debugger and watched it bounce around
> SolrDispatchHandler.doFilter() before giving up and forwarding the request
> to the RequestDispatcher, but all I could tell from that was that it
> wanted something it couldn't find. Can anyone shed some light on what my
> mods might be keeping Solr from doing that would cause this problem?
>
> Code pasted below. And don't mind the weird locking, this happens even
> with a single thread.
>
> Thanks,
> Dan
>
> public SolrCore getCore(String name) {
> ReentrantReadWriteLock lock = getCoreLock(name);
> try {
> lock.readLock().lock();
> SolrCore core = cores.get(name);
> if (core != null) {
> core.open(); // increment the ref count while still
> synchronized
> return core;
> } else if ("".equals(name) || "admin".equals(name)) {
> return null;
> } else {
> try {
> lock.readLock().unlock();
> lock.writeLock().lock();
> SolrCore core1 = cores.get(name);
> if (core1 != null) {
> return core1;
> }
> log.info("Autocreating core: '" + name + "'");
> core = create(name);
> cores.put(name, core);
> core.open();
> return core;
> } catch (IOException e) {
> log.error("Autocreating core '" + name + "'", e);
> return null;
> } catch (ParserConfigurationException e) {
> log.error("Autocreating core '" + name + "'", e);
> return null;
> } catch (SAXException e) {
> log.error("Autocreating core '" + name + "'", e);
> return null;
> } finally {
> lock.readLock().lock();
> lock.writeLock().unlock();
> }
> }
> } finally {
> lock.readLock().unlock();
> }
> }
>
> public SolrCore create(String coreName) throws IOException,
> ParserConfigurationException, SAXException {
> if (defaultConfigFile == null || defaultSchemaFile == null) {
> throw new RuntimeException("Cannot use autocreate unless both a
> default configuration file and a default schema file are specified");
> }
> CoreDescriptor dcore = new CoreDescriptor(this, coreName,
> getInstanceDir(coreName));
> dcore.setConfigName(defaultConfigFile);
> dcore.setSchemaName(defaultSchemaFile);
> return create(dcore);
> }
>
> // Eventually this will be overridden to do some intelligent management of
> a directory hierarchy so we //don't have hundreds of thousands of cores in
> the same directory
> public String getInstanceDir(String coreName) {
> return coreName;
> }
>
>
--
View this message in context: http://www.nabble.com/Problem-After-Modifying-CoreContainer-tp24762199p24763149.html
Sent from the Solr - User mailing list archive at Nabble.com.