You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Joseph Obernberger <jo...@lovehorsepower.com> on 2015/03/24 19:47:37 UTC
Solr and HDFS configuration
Hi All - does it make sense to run a solr shard on a node within an
Hadoop cluster that is not a data node? In that case all the data that
node processes would need to come over the network, but you get the
benefit of more CPU for things like faceting.
Thank you!
-Joe
Re: Solr 5.0.0 and HDFS
Posted by Joseph Obernberger <jo...@lovehorsepower.com>.
Hi - I've been able to replicate the error, and I'm also getting the
following error that might be related:
null:org.apache.solr.common.SolrException: CLUSTERSTATUS the collection
time out:180s
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:630)
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:582)
at
org.apache.solr.handler.admin.CollectionsHandler.handleClusterStatus(CollectionsHandler.java:932)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:256)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:736)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:261)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
--------
Is there anything I can do to debug this better?
Thank you!
-Joe
On 3/31/2015 3:13 PM, Joseph Obernberger wrote:
> I've tried to replicate the issue starting from new, but so far it
> hasn't happened again.
>
> -Joe
>
> On 3/28/2015 2:10 PM, Mark Miller wrote:
>> Hmm...can you file a JIRA issue with this info?
>>
>> - Mark
>>
>> On Fri, Mar 27, 2015 at 6:09 PM Joseph Obernberger
>> <jo...@lovehorsepower.com>
>> wrote:
>>
>>> I just started up a two shard cluster on two machines using HDFS.
>>> When I
>>> started to index documents, the log shows errors like this. They repeat
>>> when I execute searches. All seems well - searches and indexing appear
>>> to be working.
>>> Possibly a configuration issue?
>>> My HDFS config:
>>> <directoryFactory name="DirectoryFactory"
>>> class="solr.HdfsDirectoryFactory">
>>> <bool name="solr.hdfs.blockcache.enabled">true</bool>
>>> <int name="solr.hdfs.blockcache.slab.count">160</int>
>>> <bool
>>> name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
>>> <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
>>> <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
>>> <bool name="solr.hdfs.blockcache.write.enabled">false</bool>
>>> <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
>>> <int
>>> name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64</int>
>>> <int
>>> name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512</int>
>>> <str
>>> name="solr.hdfs.home">hdfs://nameservice1:8020/solr5</str>
>>> <str
>>> name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</
>>> str>
>>> </directoryFactory>
>>> Thank you!
>>>
>>> -Joe
>>> --------------------------------------------------------
>>>
>>> java.lang.IllegalStateException: file:
>>> BlockDirectory(HdfsDirectory@799d5a0e
>>> lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@49838b82)
>>> appears
>>> both in delegate and in cache: cache=[_25.fnm, _2d.si, _2e.nvd, _2b.si,
>>> _28.tvx, _2c.tvx, _1t.si, _27.nvd, _2b.tvd, _2d_Lucene50_0.pos,
>>> _23.nvd,
>>> _28_Lucene50_0.doc, _28_Lucene50_0.dvd, _2d.fdt, _2c_Lucene50_0.pos,
>>> _23.fdx, _2b_Lucene50_0.doc, _2d.nvm, _28.nvd, _23.fnm,
>>> _2b_Lucene50_0.tim, _2e.fdt, _2d_Lucene50_0.doc, _2b_Lucene50_0.dvd,
>>> _2d_Lucene50_0.dvd, _2b.nvd, _2g.tvx, _28_Lucene50_0.dvm,
>>> _1v_Lucene50_0.tip, _2e_Lucene50_0.dvm, _2e_Lucene50_0.pos, _2g.fdx,
>>> _2e.nvm, _2f.fdx, _1s.tvd, _23.nvm, _27.nvm, _1s_Lucene50_0.tip,
>>> _2c.fnm, _2b.fdt, _2d.fdx, _2c.fdx, _2c.nvm, _2e.fnm,
>>> _2d_Lucene50_0.dvm, _28.nvm, _28.fnm, _2b_Lucene50_0.tip,
>>> _2e_Lucene50_0.dvd, _2c.si, _2f.fdt, _2b.fnm, _2e_Lucene50_0.tip,
>>> _28.si, _28_Lucene50_0.tip, _2f.tvd, _2d_Lucene50_0.tim, _2f.tvx,
>>> _2b_Lucene50_0.pos, _2e.fdx, _28.fdx, _2c_Lucene50_0.dvd, _2g.tvd,
>>> _2c_Lucene50_0.tim, _2b.nvm, _23.fdt, _1s_Lucene50_0.tim,
>>> _28_Lucene50_0.tim, _2c_Lucene50_0.doc, _28.tvd, _2b.tvx, _2c.nvd,
>>> _2b.fdx, _2c_Lucene50_0.tip, _2e_Lucene50_0.doc, _2e_Lucene50_0.tim,
>>> _2c.fdt, _27.tvd, _2d.tvd, _2d.tvx, _28_Lucene50_0.pos,
>>> _2b_Lucene50_0.dvm, _2e.si, _2e.tvd, _2d.fnm, _2c.tvd, _2g.fdt,
>>> _2e.tvx,
>>> _28.fdt, _2d_Lucene50_0.tip, _2c_Lucene50_0.dvm,
>>> _2d.nvd],delegate=[_10.fdt, _10.fdx, _10.fnm, _10.nvd, _10.nvm, _10.si,
>>> _10.tvd, _10.tvx, _10_Lucene50_0.doc, _10_Lucene50_0.dvd,
>>> _10_Lucene50_0.dvm, _10_Lucene50_0.pos, _10_Lucene50_0.tim,
>>> _10_Lucene50_0.tip, _11.fdt, _11.fdx, _11.fnm, _11.nvd, _11.nvm,
>>> _11.si,
>>> _11.tvd, _11.tvx, _11_Lucene50_0.doc, _11_Lucene50_0.dvd,
>>> _11_Lucene50_0.dvm, _11_Lucene50_0.pos, _11_Lucene50_0.tim,
>>> _11_Lucene50_0.tip, _12.fdt, _12.fdx, _12.fnm, _12.nvd, _12.nvm,
>>> _12.si,
>>> _12.tvd, _12.tvx, _12_Lucene50_0.doc, _12_Lucene50_0.dvd,
>>> _12_Lucene50_0.dvm, _12_Lucene50_0.pos, _12_Lucene50_0.tim,
>>> _12_Lucene50_0.tip, _13.fdt, _13.fdx, _13.fnm, _13.nvd, _13.nvm,
>>> _13.si,
>>> _13.tvd, _13.tvx, _13_Lucene50_0.doc, _13_Lucene50_0.dvd,
>>> _13_Lucene50_0.dvm, _13_Lucene50_0.pos, _13_Lucene50_0.tim,
>>> _13_Lucene50_0.tip, _14.fdt, _14.fdx, _14.fnm, _14.nvd, _14.nvm,
>>> _14.si,
>>> _14.tvd, _14.tvx, _14_Lucene50_0.doc, _14_Lucene50_0.dvd,
>>> _14_Lucene50_0.dvm, _14_Lucene50_0.pos, _14_Lucene50_0.tim,
>>> _14_Lucene50_0.tip, _15.fdt, _15.fdx, _15.fnm, _15.nvd, _15.nvm,
>>> _15.si,
>>> _15.tvd, _15.tvx, _15_Lucene50_0.doc, _15_Lucene50_0.dvd,
>>> _15_Lucene50_0.dvm, _15_Lucene50_0.pos, _15_Lucene50_0.tim,
>>> _15_Lucene50_0.tip, _1f.fdt, _1f.fdx, _1f.fnm, _1f.nvd, _1f.nvm,
>>> _1f.si,
>>> _1f.tvd, _1f.tvx, _1f_Lucene50_0.doc, _1f_Lucene50_0.dvd,
>>> _1f_Lucene50_0.dvm, _1f_Lucene50_0.pos, _1f_Lucene50_0.tim,
>>> _1f_Lucene50_0.tip, _1g.fdt, _1g.fdx, _1g.fnm, _1g.nvd, _1g.nvm,
>>> _1g.si,
>>> _1g.tvd, _1g.tvx, _1g_Lucene50_0.doc, _1g_Lucene50_0.dvd,
>>> _1g_Lucene50_0.dvm, _1g_Lucene50_0.pos, _1g_Lucene50_0.tim,
>>> _1g_Lucene50_0.tip, _1h.fdt, _1h.fdx, _1h.fnm, _1h.nvd, _1h.nvm,
>>> _1h.si,
>>> _1h.tvd, _1h.tvx, _1h_Lucene50_0.doc, _1h_Lucene50_0.dvd,
>>> _1h_Lucene50_0.dvm, _1h_Lucene50_0.pos, _1h_Lucene50_0.tim,
>>> _1h_Lucene50_0.tip, _1i.fdt, _1i.fdx, _1i.fnm, _1i.nvd, _1i.nvm,
>>> _1i.si,
>>> _1i.tvd, _1i.tvx, _1i_Lucene50_0.doc, _1i_Lucene50_0.dvd,
>>> _1i_Lucene50_0.dvm, _1i_Lucene50_0.pos, _1i_Lucene50_0.tim,
>>> _1i_Lucene50_0.tip, _1j.fdt, _1j.fdx, _1j.fnm, _1j.nvd, _1j.nvm,
>>> _1j.si,
>>> _1j.tvd, _1j.tvx, _1j_Lucene50_0.doc, _1j_Lucene50_0.dvd,
>>> _1j_Lucene50_0.dvm, _1j_Lucene50_0.pos, _1j_Lucene50_0.tim,
>>> _1j_Lucene50_0.tip, _1k.fdt, _1k.fdx, _1k.fnm, _1k.nvd, _1k.nvm,
>>> _1k.si,
>>> _1k.tvd, _1k.tvx, _1k_Lucene50_0.doc, _1k_Lucene50_0.dvd,
>>> _1k_Lucene50_0.dvm, _1k_Lucene50_0.pos, _1k_Lucene50_0.tim,
>>> _1k_Lucene50_0.tip, _1l.fdt, _1l.fdx, _1l.fnm, _1l.nvd, _1l.nvm,
>>> _1l.si,
>>> _1l.tvd, _1l.tvx, _1l_Lucene50_0.doc, _1l_Lucene50_0.dvd,
>>> _1l_Lucene50_0.dvm, _1l_Lucene50_0.pos, _1l_Lucene50_0.tim,
>>> _1l_Lucene50_0.tip, _1m.fdt, _1m.fdx, _1m.fnm, _1m.nvd, _1m.nvm,
>>> _1m.si,
>>> _1m.tvd, _1m.tvx, _1m_Lucene50_0.doc, _1m_Lucene50_0.dvd,
>>> _1m_Lucene50_0.dvm, _1m_Lucene50_0.pos, _1m_Lucene50_0.tim,
>>> _1m_Lucene50_0.tip, _1n.fdt, _1n.fdx, _1n.fnm, _1n.nvd, _1n.nvm,
>>> _1n.si,
>>> _1n.tvd, _1n.tvx, _1n_Lucene50_0.doc, _1n_Lucene50_0.dvd,
>>> _1n_Lucene50_0.dvm, _1n_Lucene50_0.pos, _1n_Lucene50_0.tim,
>>> _1n_Lucene50_0.tip, _1p.fdt, _1p.fdx, _1p.fnm, _1p.nvd, _1p.nvm,
>>> _1p.si,
>>> _1p.tvd, _1p.tvx, _1p_Lucene50_0.doc, _1p_Lucene50_0.dvd,
>>> _1p_Lucene50_0.dvm, _1p_Lucene50_0.pos, _1p_Lucene50_0.tim,
>>> _1p_Lucene50_0.tip, _1s.fdt, _1s.fdx, _1s.fnm, _1s.nvd, _1s.nvm,
>>> _1s.si,
>>> _1s.tvx, _1s_Lucene50_0.doc, _1s_Lucene50_0.dvd, _1s_Lucene50_0.dvm,
>>> _1s_Lucene50_0.pos, _1t.fdt, _1t.fdx, _1t.fnm, _1t.nvd, _1t.nvm,
>>> _1t.tvd, _1t.tvx, _1t_Lucene50_0.doc, _1t_Lucene50_0.dvd,
>>> _1t_Lucene50_0.dvm, _1t_Lucene50_0.pos, _1t_Lucene50_0.tim,
>>> _1t_Lucene50_0.tip, _1v.fdt, _1v.fdx, _1v.fnm, _1v.nvd, _1v.nvm,
>>> _1v.si,
>>> _1v.tvd, _1v.tvx, _1v_Lucene50_0.doc, _1v_Lucene50_0.dvd,
>>> _1v_Lucene50_0.dvm, _1v_Lucene50_0.pos, _1v_Lucene50_0.tim,
>>> _1v_Lucene50_0.tip, _22.fdt, _22.fdx, _22.fnm, _22.nvd, _22.nvm,
>>> _22.si,
>>> _22.tvd, _22.tvx, _22_Lucene50_0.doc, _22_Lucene50_0.dvd,
>>> _22_Lucene50_0.dvm, _22_Lucene50_0.pos, _22_Lucene50_0.tim,
>>> _22_Lucene50_0.tip, _23.si, _23.tvd, _23.tvx, _23_Lucene50_0.doc,
>>> _23_Lucene50_0.dvd, _23_Lucene50_0.dvm, _23_Lucene50_0.pos,
>>> _23_Lucene50_0.tim, _23_Lucene50_0.tip, _24.fdt, _24.fdx, _24.fnm,
>>> _24.nvd, _24.nvm, _24.si, _24.tvd, _24.tvx, _24_Lucene50_0.doc,
>>> _24_Lucene50_0.dvd, _24_Lucene50_0.dvm, _24_Lucene50_0.pos,
>>> _24_Lucene50_0.tim, _24_Lucene50_0.tip, _25.fdt, _25.fdx, _25.nvd,
>>> _25.nvm, _25.si, _25.tvd, _25.tvx, _25_Lucene50_0.doc,
>>> _25_Lucene50_0.dvd, _25_Lucene50_0.dvm, _25_Lucene50_0.pos,
>>> _25_Lucene50_0.tim, _25_Lucene50_0.tip, _26.fdt, _26.fdx, _26.fnm,
>>> _26.nvd, _26.nvm, _26.si, _26.tvd, _26.tvx, _26_Lucene50_0.doc,
>>> _26_Lucene50_0.dvd, _26_Lucene50_0.dvm, _26_Lucene50_0.pos,
>>> _26_Lucene50_0.tim, _26_Lucene50_0.tip, _27.fdt, _27.fdx, _27.fnm,
>>> _27.si, _27.tvx, _27_Lucene50_0.doc, _27_Lucene50_0.dvd,
>>> _27_Lucene50_0.dvm, _27_Lucene50_0.pos, _27_Lucene50_0.tim,
>>> _27_Lucene50_0.tip, _u.fdt, _u.fdx, _u.fnm, _u.nvd, _u.nvm, _u.si,
>>> _u.tvd, _u.tvx, _u_Lucene50_0.doc, _u_Lucene50_0.dvd,
>>> _u_Lucene50_0.dvm,
>>> _u_Lucene50_0.pos, _u_Lucene50_0.tim, _u_Lucene50_0.tip, _v.fdt,
>>> _v.fdx,
>>> _v.fnm, _v.nvd, _v.nvm, _v.si, _v.tvd, _v.tvx, _v_Lucene50_0.doc,
>>> _v_Lucene50_0.dvd, _v_Lucene50_0.dvm, _v_Lucene50_0.pos,
>>> _v_Lucene50_0.tim, _v_Lucene50_0.tip, _w.fdt, _w.fdx, _w.fnm, _w.nvd,
>>> _w.nvm, _w.si, _w.tvd, _w.tvx, _w_Lucene50_0.doc, _w_Lucene50_0.dvd,
>>> _w_Lucene50_0.dvm, _w_Lucene50_0.pos, _w_Lucene50_0.tim,
>>> _w_Lucene50_0.tip, _x.fdt, _x.fdx, _x.fnm, _x.nvd, _x.nvm, _x.si,
>>> _x.tvd, _x.tvx, _x_Lucene50_0.doc, _x_Lucene50_0.dvd,
>>> _x_Lucene50_0.dvm,
>>> _x_Lucene50_0.pos, _x_Lucene50_0.tim, _x_Lucene50_0.tip, _y.fdt,
>>> _y.fdx,
>>> _y.fnm, _y.nvd, _y.nvm, _y.si, _y.tvd, _y.tvx, _y_Lucene50_0.doc,
>>> _y_Lucene50_0.dvd, _y_Lucene50_0.dvm, _y_Lucene50_0.pos,
>>> _y_Lucene50_0.tim, _y_Lucene50_0.tip, _z.fdt, _z.fdx, _z.fnm, _z.nvd,
>>> _z.nvm, _z.si, _z.tvd, _z.tvx, _z_Lucene50_0.doc, _z_Lucene50_0.dvd,
>>> _z_Lucene50_0.dvm, _z_Lucene50_0.pos, _z_Lucene50_0.tim,
>>> _z_Lucene50_0.tip, pending_segments_b, segments_9, segments_a,
>>> write.lock]
>>> at
>>> org.apache.lucene.store.NRTCachingDirectory.listAll(
>>> NRTCachingDirectory.java:103)
>>> at
>>> org.apache.solr.core.DirectoryFactory.sizeOfDirectory(
>>> DirectoryFactory.java:208)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.getIndexSize(
>>> ReplicationHandler.java:583)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.getStatistics(
>>> ReplicationHandler.java:619)
>>> at
>>> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(
>>> SolrInfoMBeanHandler.java:165)
>>> at
>>> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(
>>> SolrInfoMBeanHandler.java:135)
>>> at
>>> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(
>>> SolrInfoMBeanHandler.java:66)
>>> at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(
>>> RequestHandlerBase.java:144)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(
>>> SolrDispatchFilter.java:777)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>>> SolrDispatchFilter.java:413)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>>> SolrDispatchFilter.java:204)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>>> doFilter(ServletHandler.java:1419)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>>>
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>>> ScopedHandler.java:137)
>>> at
>>> org.eclipse.jetty.security.SecurityHandler.handle(
>>> SecurityHandler.java:557)
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.
>>> doHandle(SessionHandler.java:231)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.
>>> doHandle(ContextHandler.java:1075)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>>>
>>> at
>>> org.eclipse.jetty.server.session.SessionHandler.
>>> doScope(SessionHandler.java:193)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandler.
>>> doScope(ContextHandler.java:1009)
>>> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>>> ScopedHandler.java:135)
>>> at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
>>> ContextHandlerCollection.java:255)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerCollection.
>>> handle(HandlerCollection.java:154)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>>> HandlerWrapper.java:116)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
>>> AbstractHttpConnection.java:489)
>>> at
>>> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(
>>> BlockingHttpConnection.java:53)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection.content(
>>> AbstractHttpConnection.java:953)
>>> at
>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(
>>> AbstractHttpConnection.java:1014)
>>> at
>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
>>> at
>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>>> at
>>> org.eclipse.jetty.server.BlockingHttpConnection.handle(
>>> BlockingHttpConnection.java:72)
>>> at
>>> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(
>>> SocketConnector.java:264)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>>> QueuedThreadPool.java:608)
>>> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>>> QueuedThreadPool.java:543)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>
>
Re: Solr 5.0.0 and HDFS
Posted by Joseph Obernberger <jo...@lovehorsepower.com>.
I've tried to replicate the issue starting from new, but so far it
hasn't happened again.
-Joe
On 3/28/2015 2:10 PM, Mark Miller wrote:
> Hmm...can you file a JIRA issue with this info?
>
> - Mark
>
> On Fri, Mar 27, 2015 at 6:09 PM Joseph Obernberger <jo...@lovehorsepower.com>
> wrote:
>
>> I just started up a two shard cluster on two machines using HDFS. When I
>> started to index documents, the log shows errors like this. They repeat
>> when I execute searches. All seems well - searches and indexing appear
>> to be working.
>> Possibly a configuration issue?
>> My HDFS config:
>> <directoryFactory name="DirectoryFactory"
>> class="solr.HdfsDirectoryFactory">
>> <bool name="solr.hdfs.blockcache.enabled">true</bool>
>> <int name="solr.hdfs.blockcache.slab.count">160</int>
>> <bool
>> name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
>> <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
>> <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
>> <bool name="solr.hdfs.blockcache.write.enabled">false</bool>
>> <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
>> <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64</int>
>> <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512</int>
>> <str name="solr.hdfs.home">hdfs://nameservice1:8020/solr5</str>
>> <str name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</
>> str>
>> </directoryFactory>
>> Thank you!
>>
>> -Joe
>> --------------------------------------------------------
>>
>> java.lang.IllegalStateException: file:
>> BlockDirectory(HdfsDirectory@799d5a0e
>> lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@49838b82) appears
>> both in delegate and in cache: cache=[_25.fnm, _2d.si, _2e.nvd, _2b.si,
>> _28.tvx, _2c.tvx, _1t.si, _27.nvd, _2b.tvd, _2d_Lucene50_0.pos, _23.nvd,
>> _28_Lucene50_0.doc, _28_Lucene50_0.dvd, _2d.fdt, _2c_Lucene50_0.pos,
>> _23.fdx, _2b_Lucene50_0.doc, _2d.nvm, _28.nvd, _23.fnm,
>> _2b_Lucene50_0.tim, _2e.fdt, _2d_Lucene50_0.doc, _2b_Lucene50_0.dvd,
>> _2d_Lucene50_0.dvd, _2b.nvd, _2g.tvx, _28_Lucene50_0.dvm,
>> _1v_Lucene50_0.tip, _2e_Lucene50_0.dvm, _2e_Lucene50_0.pos, _2g.fdx,
>> _2e.nvm, _2f.fdx, _1s.tvd, _23.nvm, _27.nvm, _1s_Lucene50_0.tip,
>> _2c.fnm, _2b.fdt, _2d.fdx, _2c.fdx, _2c.nvm, _2e.fnm,
>> _2d_Lucene50_0.dvm, _28.nvm, _28.fnm, _2b_Lucene50_0.tip,
>> _2e_Lucene50_0.dvd, _2c.si, _2f.fdt, _2b.fnm, _2e_Lucene50_0.tip,
>> _28.si, _28_Lucene50_0.tip, _2f.tvd, _2d_Lucene50_0.tim, _2f.tvx,
>> _2b_Lucene50_0.pos, _2e.fdx, _28.fdx, _2c_Lucene50_0.dvd, _2g.tvd,
>> _2c_Lucene50_0.tim, _2b.nvm, _23.fdt, _1s_Lucene50_0.tim,
>> _28_Lucene50_0.tim, _2c_Lucene50_0.doc, _28.tvd, _2b.tvx, _2c.nvd,
>> _2b.fdx, _2c_Lucene50_0.tip, _2e_Lucene50_0.doc, _2e_Lucene50_0.tim,
>> _2c.fdt, _27.tvd, _2d.tvd, _2d.tvx, _28_Lucene50_0.pos,
>> _2b_Lucene50_0.dvm, _2e.si, _2e.tvd, _2d.fnm, _2c.tvd, _2g.fdt, _2e.tvx,
>> _28.fdt, _2d_Lucene50_0.tip, _2c_Lucene50_0.dvm,
>> _2d.nvd],delegate=[_10.fdt, _10.fdx, _10.fnm, _10.nvd, _10.nvm, _10.si,
>> _10.tvd, _10.tvx, _10_Lucene50_0.doc, _10_Lucene50_0.dvd,
>> _10_Lucene50_0.dvm, _10_Lucene50_0.pos, _10_Lucene50_0.tim,
>> _10_Lucene50_0.tip, _11.fdt, _11.fdx, _11.fnm, _11.nvd, _11.nvm, _11.si,
>> _11.tvd, _11.tvx, _11_Lucene50_0.doc, _11_Lucene50_0.dvd,
>> _11_Lucene50_0.dvm, _11_Lucene50_0.pos, _11_Lucene50_0.tim,
>> _11_Lucene50_0.tip, _12.fdt, _12.fdx, _12.fnm, _12.nvd, _12.nvm, _12.si,
>> _12.tvd, _12.tvx, _12_Lucene50_0.doc, _12_Lucene50_0.dvd,
>> _12_Lucene50_0.dvm, _12_Lucene50_0.pos, _12_Lucene50_0.tim,
>> _12_Lucene50_0.tip, _13.fdt, _13.fdx, _13.fnm, _13.nvd, _13.nvm, _13.si,
>> _13.tvd, _13.tvx, _13_Lucene50_0.doc, _13_Lucene50_0.dvd,
>> _13_Lucene50_0.dvm, _13_Lucene50_0.pos, _13_Lucene50_0.tim,
>> _13_Lucene50_0.tip, _14.fdt, _14.fdx, _14.fnm, _14.nvd, _14.nvm, _14.si,
>> _14.tvd, _14.tvx, _14_Lucene50_0.doc, _14_Lucene50_0.dvd,
>> _14_Lucene50_0.dvm, _14_Lucene50_0.pos, _14_Lucene50_0.tim,
>> _14_Lucene50_0.tip, _15.fdt, _15.fdx, _15.fnm, _15.nvd, _15.nvm, _15.si,
>> _15.tvd, _15.tvx, _15_Lucene50_0.doc, _15_Lucene50_0.dvd,
>> _15_Lucene50_0.dvm, _15_Lucene50_0.pos, _15_Lucene50_0.tim,
>> _15_Lucene50_0.tip, _1f.fdt, _1f.fdx, _1f.fnm, _1f.nvd, _1f.nvm, _1f.si,
>> _1f.tvd, _1f.tvx, _1f_Lucene50_0.doc, _1f_Lucene50_0.dvd,
>> _1f_Lucene50_0.dvm, _1f_Lucene50_0.pos, _1f_Lucene50_0.tim,
>> _1f_Lucene50_0.tip, _1g.fdt, _1g.fdx, _1g.fnm, _1g.nvd, _1g.nvm, _1g.si,
>> _1g.tvd, _1g.tvx, _1g_Lucene50_0.doc, _1g_Lucene50_0.dvd,
>> _1g_Lucene50_0.dvm, _1g_Lucene50_0.pos, _1g_Lucene50_0.tim,
>> _1g_Lucene50_0.tip, _1h.fdt, _1h.fdx, _1h.fnm, _1h.nvd, _1h.nvm, _1h.si,
>> _1h.tvd, _1h.tvx, _1h_Lucene50_0.doc, _1h_Lucene50_0.dvd,
>> _1h_Lucene50_0.dvm, _1h_Lucene50_0.pos, _1h_Lucene50_0.tim,
>> _1h_Lucene50_0.tip, _1i.fdt, _1i.fdx, _1i.fnm, _1i.nvd, _1i.nvm, _1i.si,
>> _1i.tvd, _1i.tvx, _1i_Lucene50_0.doc, _1i_Lucene50_0.dvd,
>> _1i_Lucene50_0.dvm, _1i_Lucene50_0.pos, _1i_Lucene50_0.tim,
>> _1i_Lucene50_0.tip, _1j.fdt, _1j.fdx, _1j.fnm, _1j.nvd, _1j.nvm, _1j.si,
>> _1j.tvd, _1j.tvx, _1j_Lucene50_0.doc, _1j_Lucene50_0.dvd,
>> _1j_Lucene50_0.dvm, _1j_Lucene50_0.pos, _1j_Lucene50_0.tim,
>> _1j_Lucene50_0.tip, _1k.fdt, _1k.fdx, _1k.fnm, _1k.nvd, _1k.nvm, _1k.si,
>> _1k.tvd, _1k.tvx, _1k_Lucene50_0.doc, _1k_Lucene50_0.dvd,
>> _1k_Lucene50_0.dvm, _1k_Lucene50_0.pos, _1k_Lucene50_0.tim,
>> _1k_Lucene50_0.tip, _1l.fdt, _1l.fdx, _1l.fnm, _1l.nvd, _1l.nvm, _1l.si,
>> _1l.tvd, _1l.tvx, _1l_Lucene50_0.doc, _1l_Lucene50_0.dvd,
>> _1l_Lucene50_0.dvm, _1l_Lucene50_0.pos, _1l_Lucene50_0.tim,
>> _1l_Lucene50_0.tip, _1m.fdt, _1m.fdx, _1m.fnm, _1m.nvd, _1m.nvm, _1m.si,
>> _1m.tvd, _1m.tvx, _1m_Lucene50_0.doc, _1m_Lucene50_0.dvd,
>> _1m_Lucene50_0.dvm, _1m_Lucene50_0.pos, _1m_Lucene50_0.tim,
>> _1m_Lucene50_0.tip, _1n.fdt, _1n.fdx, _1n.fnm, _1n.nvd, _1n.nvm, _1n.si,
>> _1n.tvd, _1n.tvx, _1n_Lucene50_0.doc, _1n_Lucene50_0.dvd,
>> _1n_Lucene50_0.dvm, _1n_Lucene50_0.pos, _1n_Lucene50_0.tim,
>> _1n_Lucene50_0.tip, _1p.fdt, _1p.fdx, _1p.fnm, _1p.nvd, _1p.nvm, _1p.si,
>> _1p.tvd, _1p.tvx, _1p_Lucene50_0.doc, _1p_Lucene50_0.dvd,
>> _1p_Lucene50_0.dvm, _1p_Lucene50_0.pos, _1p_Lucene50_0.tim,
>> _1p_Lucene50_0.tip, _1s.fdt, _1s.fdx, _1s.fnm, _1s.nvd, _1s.nvm, _1s.si,
>> _1s.tvx, _1s_Lucene50_0.doc, _1s_Lucene50_0.dvd, _1s_Lucene50_0.dvm,
>> _1s_Lucene50_0.pos, _1t.fdt, _1t.fdx, _1t.fnm, _1t.nvd, _1t.nvm,
>> _1t.tvd, _1t.tvx, _1t_Lucene50_0.doc, _1t_Lucene50_0.dvd,
>> _1t_Lucene50_0.dvm, _1t_Lucene50_0.pos, _1t_Lucene50_0.tim,
>> _1t_Lucene50_0.tip, _1v.fdt, _1v.fdx, _1v.fnm, _1v.nvd, _1v.nvm, _1v.si,
>> _1v.tvd, _1v.tvx, _1v_Lucene50_0.doc, _1v_Lucene50_0.dvd,
>> _1v_Lucene50_0.dvm, _1v_Lucene50_0.pos, _1v_Lucene50_0.tim,
>> _1v_Lucene50_0.tip, _22.fdt, _22.fdx, _22.fnm, _22.nvd, _22.nvm, _22.si,
>> _22.tvd, _22.tvx, _22_Lucene50_0.doc, _22_Lucene50_0.dvd,
>> _22_Lucene50_0.dvm, _22_Lucene50_0.pos, _22_Lucene50_0.tim,
>> _22_Lucene50_0.tip, _23.si, _23.tvd, _23.tvx, _23_Lucene50_0.doc,
>> _23_Lucene50_0.dvd, _23_Lucene50_0.dvm, _23_Lucene50_0.pos,
>> _23_Lucene50_0.tim, _23_Lucene50_0.tip, _24.fdt, _24.fdx, _24.fnm,
>> _24.nvd, _24.nvm, _24.si, _24.tvd, _24.tvx, _24_Lucene50_0.doc,
>> _24_Lucene50_0.dvd, _24_Lucene50_0.dvm, _24_Lucene50_0.pos,
>> _24_Lucene50_0.tim, _24_Lucene50_0.tip, _25.fdt, _25.fdx, _25.nvd,
>> _25.nvm, _25.si, _25.tvd, _25.tvx, _25_Lucene50_0.doc,
>> _25_Lucene50_0.dvd, _25_Lucene50_0.dvm, _25_Lucene50_0.pos,
>> _25_Lucene50_0.tim, _25_Lucene50_0.tip, _26.fdt, _26.fdx, _26.fnm,
>> _26.nvd, _26.nvm, _26.si, _26.tvd, _26.tvx, _26_Lucene50_0.doc,
>> _26_Lucene50_0.dvd, _26_Lucene50_0.dvm, _26_Lucene50_0.pos,
>> _26_Lucene50_0.tim, _26_Lucene50_0.tip, _27.fdt, _27.fdx, _27.fnm,
>> _27.si, _27.tvx, _27_Lucene50_0.doc, _27_Lucene50_0.dvd,
>> _27_Lucene50_0.dvm, _27_Lucene50_0.pos, _27_Lucene50_0.tim,
>> _27_Lucene50_0.tip, _u.fdt, _u.fdx, _u.fnm, _u.nvd, _u.nvm, _u.si,
>> _u.tvd, _u.tvx, _u_Lucene50_0.doc, _u_Lucene50_0.dvd, _u_Lucene50_0.dvm,
>> _u_Lucene50_0.pos, _u_Lucene50_0.tim, _u_Lucene50_0.tip, _v.fdt, _v.fdx,
>> _v.fnm, _v.nvd, _v.nvm, _v.si, _v.tvd, _v.tvx, _v_Lucene50_0.doc,
>> _v_Lucene50_0.dvd, _v_Lucene50_0.dvm, _v_Lucene50_0.pos,
>> _v_Lucene50_0.tim, _v_Lucene50_0.tip, _w.fdt, _w.fdx, _w.fnm, _w.nvd,
>> _w.nvm, _w.si, _w.tvd, _w.tvx, _w_Lucene50_0.doc, _w_Lucene50_0.dvd,
>> _w_Lucene50_0.dvm, _w_Lucene50_0.pos, _w_Lucene50_0.tim,
>> _w_Lucene50_0.tip, _x.fdt, _x.fdx, _x.fnm, _x.nvd, _x.nvm, _x.si,
>> _x.tvd, _x.tvx, _x_Lucene50_0.doc, _x_Lucene50_0.dvd, _x_Lucene50_0.dvm,
>> _x_Lucene50_0.pos, _x_Lucene50_0.tim, _x_Lucene50_0.tip, _y.fdt, _y.fdx,
>> _y.fnm, _y.nvd, _y.nvm, _y.si, _y.tvd, _y.tvx, _y_Lucene50_0.doc,
>> _y_Lucene50_0.dvd, _y_Lucene50_0.dvm, _y_Lucene50_0.pos,
>> _y_Lucene50_0.tim, _y_Lucene50_0.tip, _z.fdt, _z.fdx, _z.fnm, _z.nvd,
>> _z.nvm, _z.si, _z.tvd, _z.tvx, _z_Lucene50_0.doc, _z_Lucene50_0.dvd,
>> _z_Lucene50_0.dvm, _z_Lucene50_0.pos, _z_Lucene50_0.tim,
>> _z_Lucene50_0.tip, pending_segments_b, segments_9, segments_a, write.lock]
>> at
>> org.apache.lucene.store.NRTCachingDirectory.listAll(
>> NRTCachingDirectory.java:103)
>> at
>> org.apache.solr.core.DirectoryFactory.sizeOfDirectory(
>> DirectoryFactory.java:208)
>> at
>> org.apache.solr.handler.ReplicationHandler.getIndexSize(
>> ReplicationHandler.java:583)
>> at
>> org.apache.solr.handler.ReplicationHandler.getStatistics(
>> ReplicationHandler.java:619)
>> at
>> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(
>> SolrInfoMBeanHandler.java:165)
>> at
>> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(
>> SolrInfoMBeanHandler.java:135)
>> at
>> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(
>> SolrInfoMBeanHandler.java:66)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(
>> RequestHandlerBase.java:144)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(
>> SolrDispatchFilter.java:777)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:413)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:204)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1419)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:137)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(
>> SecurityHandler.java:557)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.
>> doHandle(SessionHandler.java:231)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1075)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.
>> doScope(SessionHandler.java:193)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1009)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:135)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
>> ContextHandlerCollection.java:255)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.
>> handle(HandlerCollection.java:154)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:116)
>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
>> AbstractHttpConnection.java:489)
>> at
>> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(
>> BlockingHttpConnection.java:53)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection.content(
>> AbstractHttpConnection.java:953)
>> at
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(
>> AbstractHttpConnection.java:1014)
>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
>> at
>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>> at
>> org.eclipse.jetty.server.BlockingHttpConnection.handle(
>> BlockingHttpConnection.java:72)
>> at
>> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(
>> SocketConnector.java:264)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:608)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
Re: Solr 5.0.0 and HDFS
Posted by Mark Miller <ma...@gmail.com>.
Hmm...can you file a JIRA issue with this info?
- Mark
On Fri, Mar 27, 2015 at 6:09 PM Joseph Obernberger <jo...@lovehorsepower.com>
wrote:
> I just started up a two shard cluster on two machines using HDFS. When I
> started to index documents, the log shows errors like this. They repeat
> when I execute searches. All seems well - searches and indexing appear
> to be working.
> Possibly a configuration issue?
> My HDFS config:
> <directoryFactory name="DirectoryFactory"
> class="solr.HdfsDirectoryFactory">
> <bool name="solr.hdfs.blockcache.enabled">true</bool>
> <int name="solr.hdfs.blockcache.slab.count">160</int>
> <bool
> name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
> <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
> <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
> <bool name="solr.hdfs.blockcache.write.enabled">false</bool>
> <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
> <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64</int>
> <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512</int>
> <str name="solr.hdfs.home">hdfs://nameservice1:8020/solr5</str>
> <str name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</
> str>
> </directoryFactory>
> Thank you!
>
> -Joe
> --------------------------------------------------------
>
> java.lang.IllegalStateException: file:
> BlockDirectory(HdfsDirectory@799d5a0e
> lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@49838b82) appears
> both in delegate and in cache: cache=[_25.fnm, _2d.si, _2e.nvd, _2b.si,
> _28.tvx, _2c.tvx, _1t.si, _27.nvd, _2b.tvd, _2d_Lucene50_0.pos, _23.nvd,
> _28_Lucene50_0.doc, _28_Lucene50_0.dvd, _2d.fdt, _2c_Lucene50_0.pos,
> _23.fdx, _2b_Lucene50_0.doc, _2d.nvm, _28.nvd, _23.fnm,
> _2b_Lucene50_0.tim, _2e.fdt, _2d_Lucene50_0.doc, _2b_Lucene50_0.dvd,
> _2d_Lucene50_0.dvd, _2b.nvd, _2g.tvx, _28_Lucene50_0.dvm,
> _1v_Lucene50_0.tip, _2e_Lucene50_0.dvm, _2e_Lucene50_0.pos, _2g.fdx,
> _2e.nvm, _2f.fdx, _1s.tvd, _23.nvm, _27.nvm, _1s_Lucene50_0.tip,
> _2c.fnm, _2b.fdt, _2d.fdx, _2c.fdx, _2c.nvm, _2e.fnm,
> _2d_Lucene50_0.dvm, _28.nvm, _28.fnm, _2b_Lucene50_0.tip,
> _2e_Lucene50_0.dvd, _2c.si, _2f.fdt, _2b.fnm, _2e_Lucene50_0.tip,
> _28.si, _28_Lucene50_0.tip, _2f.tvd, _2d_Lucene50_0.tim, _2f.tvx,
> _2b_Lucene50_0.pos, _2e.fdx, _28.fdx, _2c_Lucene50_0.dvd, _2g.tvd,
> _2c_Lucene50_0.tim, _2b.nvm, _23.fdt, _1s_Lucene50_0.tim,
> _28_Lucene50_0.tim, _2c_Lucene50_0.doc, _28.tvd, _2b.tvx, _2c.nvd,
> _2b.fdx, _2c_Lucene50_0.tip, _2e_Lucene50_0.doc, _2e_Lucene50_0.tim,
> _2c.fdt, _27.tvd, _2d.tvd, _2d.tvx, _28_Lucene50_0.pos,
> _2b_Lucene50_0.dvm, _2e.si, _2e.tvd, _2d.fnm, _2c.tvd, _2g.fdt, _2e.tvx,
> _28.fdt, _2d_Lucene50_0.tip, _2c_Lucene50_0.dvm,
> _2d.nvd],delegate=[_10.fdt, _10.fdx, _10.fnm, _10.nvd, _10.nvm, _10.si,
> _10.tvd, _10.tvx, _10_Lucene50_0.doc, _10_Lucene50_0.dvd,
> _10_Lucene50_0.dvm, _10_Lucene50_0.pos, _10_Lucene50_0.tim,
> _10_Lucene50_0.tip, _11.fdt, _11.fdx, _11.fnm, _11.nvd, _11.nvm, _11.si,
> _11.tvd, _11.tvx, _11_Lucene50_0.doc, _11_Lucene50_0.dvd,
> _11_Lucene50_0.dvm, _11_Lucene50_0.pos, _11_Lucene50_0.tim,
> _11_Lucene50_0.tip, _12.fdt, _12.fdx, _12.fnm, _12.nvd, _12.nvm, _12.si,
> _12.tvd, _12.tvx, _12_Lucene50_0.doc, _12_Lucene50_0.dvd,
> _12_Lucene50_0.dvm, _12_Lucene50_0.pos, _12_Lucene50_0.tim,
> _12_Lucene50_0.tip, _13.fdt, _13.fdx, _13.fnm, _13.nvd, _13.nvm, _13.si,
> _13.tvd, _13.tvx, _13_Lucene50_0.doc, _13_Lucene50_0.dvd,
> _13_Lucene50_0.dvm, _13_Lucene50_0.pos, _13_Lucene50_0.tim,
> _13_Lucene50_0.tip, _14.fdt, _14.fdx, _14.fnm, _14.nvd, _14.nvm, _14.si,
> _14.tvd, _14.tvx, _14_Lucene50_0.doc, _14_Lucene50_0.dvd,
> _14_Lucene50_0.dvm, _14_Lucene50_0.pos, _14_Lucene50_0.tim,
> _14_Lucene50_0.tip, _15.fdt, _15.fdx, _15.fnm, _15.nvd, _15.nvm, _15.si,
> _15.tvd, _15.tvx, _15_Lucene50_0.doc, _15_Lucene50_0.dvd,
> _15_Lucene50_0.dvm, _15_Lucene50_0.pos, _15_Lucene50_0.tim,
> _15_Lucene50_0.tip, _1f.fdt, _1f.fdx, _1f.fnm, _1f.nvd, _1f.nvm, _1f.si,
> _1f.tvd, _1f.tvx, _1f_Lucene50_0.doc, _1f_Lucene50_0.dvd,
> _1f_Lucene50_0.dvm, _1f_Lucene50_0.pos, _1f_Lucene50_0.tim,
> _1f_Lucene50_0.tip, _1g.fdt, _1g.fdx, _1g.fnm, _1g.nvd, _1g.nvm, _1g.si,
> _1g.tvd, _1g.tvx, _1g_Lucene50_0.doc, _1g_Lucene50_0.dvd,
> _1g_Lucene50_0.dvm, _1g_Lucene50_0.pos, _1g_Lucene50_0.tim,
> _1g_Lucene50_0.tip, _1h.fdt, _1h.fdx, _1h.fnm, _1h.nvd, _1h.nvm, _1h.si,
> _1h.tvd, _1h.tvx, _1h_Lucene50_0.doc, _1h_Lucene50_0.dvd,
> _1h_Lucene50_0.dvm, _1h_Lucene50_0.pos, _1h_Lucene50_0.tim,
> _1h_Lucene50_0.tip, _1i.fdt, _1i.fdx, _1i.fnm, _1i.nvd, _1i.nvm, _1i.si,
> _1i.tvd, _1i.tvx, _1i_Lucene50_0.doc, _1i_Lucene50_0.dvd,
> _1i_Lucene50_0.dvm, _1i_Lucene50_0.pos, _1i_Lucene50_0.tim,
> _1i_Lucene50_0.tip, _1j.fdt, _1j.fdx, _1j.fnm, _1j.nvd, _1j.nvm, _1j.si,
> _1j.tvd, _1j.tvx, _1j_Lucene50_0.doc, _1j_Lucene50_0.dvd,
> _1j_Lucene50_0.dvm, _1j_Lucene50_0.pos, _1j_Lucene50_0.tim,
> _1j_Lucene50_0.tip, _1k.fdt, _1k.fdx, _1k.fnm, _1k.nvd, _1k.nvm, _1k.si,
> _1k.tvd, _1k.tvx, _1k_Lucene50_0.doc, _1k_Lucene50_0.dvd,
> _1k_Lucene50_0.dvm, _1k_Lucene50_0.pos, _1k_Lucene50_0.tim,
> _1k_Lucene50_0.tip, _1l.fdt, _1l.fdx, _1l.fnm, _1l.nvd, _1l.nvm, _1l.si,
> _1l.tvd, _1l.tvx, _1l_Lucene50_0.doc, _1l_Lucene50_0.dvd,
> _1l_Lucene50_0.dvm, _1l_Lucene50_0.pos, _1l_Lucene50_0.tim,
> _1l_Lucene50_0.tip, _1m.fdt, _1m.fdx, _1m.fnm, _1m.nvd, _1m.nvm, _1m.si,
> _1m.tvd, _1m.tvx, _1m_Lucene50_0.doc, _1m_Lucene50_0.dvd,
> _1m_Lucene50_0.dvm, _1m_Lucene50_0.pos, _1m_Lucene50_0.tim,
> _1m_Lucene50_0.tip, _1n.fdt, _1n.fdx, _1n.fnm, _1n.nvd, _1n.nvm, _1n.si,
> _1n.tvd, _1n.tvx, _1n_Lucene50_0.doc, _1n_Lucene50_0.dvd,
> _1n_Lucene50_0.dvm, _1n_Lucene50_0.pos, _1n_Lucene50_0.tim,
> _1n_Lucene50_0.tip, _1p.fdt, _1p.fdx, _1p.fnm, _1p.nvd, _1p.nvm, _1p.si,
> _1p.tvd, _1p.tvx, _1p_Lucene50_0.doc, _1p_Lucene50_0.dvd,
> _1p_Lucene50_0.dvm, _1p_Lucene50_0.pos, _1p_Lucene50_0.tim,
> _1p_Lucene50_0.tip, _1s.fdt, _1s.fdx, _1s.fnm, _1s.nvd, _1s.nvm, _1s.si,
> _1s.tvx, _1s_Lucene50_0.doc, _1s_Lucene50_0.dvd, _1s_Lucene50_0.dvm,
> _1s_Lucene50_0.pos, _1t.fdt, _1t.fdx, _1t.fnm, _1t.nvd, _1t.nvm,
> _1t.tvd, _1t.tvx, _1t_Lucene50_0.doc, _1t_Lucene50_0.dvd,
> _1t_Lucene50_0.dvm, _1t_Lucene50_0.pos, _1t_Lucene50_0.tim,
> _1t_Lucene50_0.tip, _1v.fdt, _1v.fdx, _1v.fnm, _1v.nvd, _1v.nvm, _1v.si,
> _1v.tvd, _1v.tvx, _1v_Lucene50_0.doc, _1v_Lucene50_0.dvd,
> _1v_Lucene50_0.dvm, _1v_Lucene50_0.pos, _1v_Lucene50_0.tim,
> _1v_Lucene50_0.tip, _22.fdt, _22.fdx, _22.fnm, _22.nvd, _22.nvm, _22.si,
> _22.tvd, _22.tvx, _22_Lucene50_0.doc, _22_Lucene50_0.dvd,
> _22_Lucene50_0.dvm, _22_Lucene50_0.pos, _22_Lucene50_0.tim,
> _22_Lucene50_0.tip, _23.si, _23.tvd, _23.tvx, _23_Lucene50_0.doc,
> _23_Lucene50_0.dvd, _23_Lucene50_0.dvm, _23_Lucene50_0.pos,
> _23_Lucene50_0.tim, _23_Lucene50_0.tip, _24.fdt, _24.fdx, _24.fnm,
> _24.nvd, _24.nvm, _24.si, _24.tvd, _24.tvx, _24_Lucene50_0.doc,
> _24_Lucene50_0.dvd, _24_Lucene50_0.dvm, _24_Lucene50_0.pos,
> _24_Lucene50_0.tim, _24_Lucene50_0.tip, _25.fdt, _25.fdx, _25.nvd,
> _25.nvm, _25.si, _25.tvd, _25.tvx, _25_Lucene50_0.doc,
> _25_Lucene50_0.dvd, _25_Lucene50_0.dvm, _25_Lucene50_0.pos,
> _25_Lucene50_0.tim, _25_Lucene50_0.tip, _26.fdt, _26.fdx, _26.fnm,
> _26.nvd, _26.nvm, _26.si, _26.tvd, _26.tvx, _26_Lucene50_0.doc,
> _26_Lucene50_0.dvd, _26_Lucene50_0.dvm, _26_Lucene50_0.pos,
> _26_Lucene50_0.tim, _26_Lucene50_0.tip, _27.fdt, _27.fdx, _27.fnm,
> _27.si, _27.tvx, _27_Lucene50_0.doc, _27_Lucene50_0.dvd,
> _27_Lucene50_0.dvm, _27_Lucene50_0.pos, _27_Lucene50_0.tim,
> _27_Lucene50_0.tip, _u.fdt, _u.fdx, _u.fnm, _u.nvd, _u.nvm, _u.si,
> _u.tvd, _u.tvx, _u_Lucene50_0.doc, _u_Lucene50_0.dvd, _u_Lucene50_0.dvm,
> _u_Lucene50_0.pos, _u_Lucene50_0.tim, _u_Lucene50_0.tip, _v.fdt, _v.fdx,
> _v.fnm, _v.nvd, _v.nvm, _v.si, _v.tvd, _v.tvx, _v_Lucene50_0.doc,
> _v_Lucene50_0.dvd, _v_Lucene50_0.dvm, _v_Lucene50_0.pos,
> _v_Lucene50_0.tim, _v_Lucene50_0.tip, _w.fdt, _w.fdx, _w.fnm, _w.nvd,
> _w.nvm, _w.si, _w.tvd, _w.tvx, _w_Lucene50_0.doc, _w_Lucene50_0.dvd,
> _w_Lucene50_0.dvm, _w_Lucene50_0.pos, _w_Lucene50_0.tim,
> _w_Lucene50_0.tip, _x.fdt, _x.fdx, _x.fnm, _x.nvd, _x.nvm, _x.si,
> _x.tvd, _x.tvx, _x_Lucene50_0.doc, _x_Lucene50_0.dvd, _x_Lucene50_0.dvm,
> _x_Lucene50_0.pos, _x_Lucene50_0.tim, _x_Lucene50_0.tip, _y.fdt, _y.fdx,
> _y.fnm, _y.nvd, _y.nvm, _y.si, _y.tvd, _y.tvx, _y_Lucene50_0.doc,
> _y_Lucene50_0.dvd, _y_Lucene50_0.dvm, _y_Lucene50_0.pos,
> _y_Lucene50_0.tim, _y_Lucene50_0.tip, _z.fdt, _z.fdx, _z.fnm, _z.nvd,
> _z.nvm, _z.si, _z.tvd, _z.tvx, _z_Lucene50_0.doc, _z_Lucene50_0.dvd,
> _z_Lucene50_0.dvm, _z_Lucene50_0.pos, _z_Lucene50_0.tim,
> _z_Lucene50_0.tip, pending_segments_b, segments_9, segments_a, write.lock]
> at
> org.apache.lucene.store.NRTCachingDirectory.listAll(
> NRTCachingDirectory.java:103)
> at
> org.apache.solr.core.DirectoryFactory.sizeOfDirectory(
> DirectoryFactory.java:208)
> at
> org.apache.solr.handler.ReplicationHandler.getIndexSize(
> ReplicationHandler.java:583)
> at
> org.apache.solr.handler.ReplicationHandler.getStatistics(
> ReplicationHandler.java:619)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(
> SolrInfoMBeanHandler.java:165)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(
> SolrInfoMBeanHandler.java:135)
> at
> org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(
> SolrInfoMBeanHandler.java:66)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:144)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(
> SolrDispatchFilter.java:777)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:413)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:204)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:557)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
> AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(
> BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(
> AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(
> AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(
> BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(
> SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
Solr 5.0.0 and HDFS
Posted by Joseph Obernberger <jo...@lovehorsepower.com>.
I just started up a two shard cluster on two machines using HDFS. When I
started to index documents, the log shows errors like this. They repeat
when I execute searches. All seems well - searches and indexing appear
to be working.
Possibly a configuration issue?
My HDFS config:
<directoryFactory name="DirectoryFactory"
class="solr.HdfsDirectoryFactory">
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">160</int>
<bool
name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.blockcache.write.enabled">false</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512</int>
<str name="solr.hdfs.home">hdfs://nameservice1:8020/solr5</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</str>
</directoryFactory>
Thank you!
-Joe
--------------------------------------------------------
java.lang.IllegalStateException: file:
BlockDirectory(HdfsDirectory@799d5a0e
lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@49838b82) appears
both in delegate and in cache: cache=[_25.fnm, _2d.si, _2e.nvd, _2b.si,
_28.tvx, _2c.tvx, _1t.si, _27.nvd, _2b.tvd, _2d_Lucene50_0.pos, _23.nvd,
_28_Lucene50_0.doc, _28_Lucene50_0.dvd, _2d.fdt, _2c_Lucene50_0.pos,
_23.fdx, _2b_Lucene50_0.doc, _2d.nvm, _28.nvd, _23.fnm,
_2b_Lucene50_0.tim, _2e.fdt, _2d_Lucene50_0.doc, _2b_Lucene50_0.dvd,
_2d_Lucene50_0.dvd, _2b.nvd, _2g.tvx, _28_Lucene50_0.dvm,
_1v_Lucene50_0.tip, _2e_Lucene50_0.dvm, _2e_Lucene50_0.pos, _2g.fdx,
_2e.nvm, _2f.fdx, _1s.tvd, _23.nvm, _27.nvm, _1s_Lucene50_0.tip,
_2c.fnm, _2b.fdt, _2d.fdx, _2c.fdx, _2c.nvm, _2e.fnm,
_2d_Lucene50_0.dvm, _28.nvm, _28.fnm, _2b_Lucene50_0.tip,
_2e_Lucene50_0.dvd, _2c.si, _2f.fdt, _2b.fnm, _2e_Lucene50_0.tip,
_28.si, _28_Lucene50_0.tip, _2f.tvd, _2d_Lucene50_0.tim, _2f.tvx,
_2b_Lucene50_0.pos, _2e.fdx, _28.fdx, _2c_Lucene50_0.dvd, _2g.tvd,
_2c_Lucene50_0.tim, _2b.nvm, _23.fdt, _1s_Lucene50_0.tim,
_28_Lucene50_0.tim, _2c_Lucene50_0.doc, _28.tvd, _2b.tvx, _2c.nvd,
_2b.fdx, _2c_Lucene50_0.tip, _2e_Lucene50_0.doc, _2e_Lucene50_0.tim,
_2c.fdt, _27.tvd, _2d.tvd, _2d.tvx, _28_Lucene50_0.pos,
_2b_Lucene50_0.dvm, _2e.si, _2e.tvd, _2d.fnm, _2c.tvd, _2g.fdt, _2e.tvx,
_28.fdt, _2d_Lucene50_0.tip, _2c_Lucene50_0.dvm,
_2d.nvd],delegate=[_10.fdt, _10.fdx, _10.fnm, _10.nvd, _10.nvm, _10.si,
_10.tvd, _10.tvx, _10_Lucene50_0.doc, _10_Lucene50_0.dvd,
_10_Lucene50_0.dvm, _10_Lucene50_0.pos, _10_Lucene50_0.tim,
_10_Lucene50_0.tip, _11.fdt, _11.fdx, _11.fnm, _11.nvd, _11.nvm, _11.si,
_11.tvd, _11.tvx, _11_Lucene50_0.doc, _11_Lucene50_0.dvd,
_11_Lucene50_0.dvm, _11_Lucene50_0.pos, _11_Lucene50_0.tim,
_11_Lucene50_0.tip, _12.fdt, _12.fdx, _12.fnm, _12.nvd, _12.nvm, _12.si,
_12.tvd, _12.tvx, _12_Lucene50_0.doc, _12_Lucene50_0.dvd,
_12_Lucene50_0.dvm, _12_Lucene50_0.pos, _12_Lucene50_0.tim,
_12_Lucene50_0.tip, _13.fdt, _13.fdx, _13.fnm, _13.nvd, _13.nvm, _13.si,
_13.tvd, _13.tvx, _13_Lucene50_0.doc, _13_Lucene50_0.dvd,
_13_Lucene50_0.dvm, _13_Lucene50_0.pos, _13_Lucene50_0.tim,
_13_Lucene50_0.tip, _14.fdt, _14.fdx, _14.fnm, _14.nvd, _14.nvm, _14.si,
_14.tvd, _14.tvx, _14_Lucene50_0.doc, _14_Lucene50_0.dvd,
_14_Lucene50_0.dvm, _14_Lucene50_0.pos, _14_Lucene50_0.tim,
_14_Lucene50_0.tip, _15.fdt, _15.fdx, _15.fnm, _15.nvd, _15.nvm, _15.si,
_15.tvd, _15.tvx, _15_Lucene50_0.doc, _15_Lucene50_0.dvd,
_15_Lucene50_0.dvm, _15_Lucene50_0.pos, _15_Lucene50_0.tim,
_15_Lucene50_0.tip, _1f.fdt, _1f.fdx, _1f.fnm, _1f.nvd, _1f.nvm, _1f.si,
_1f.tvd, _1f.tvx, _1f_Lucene50_0.doc, _1f_Lucene50_0.dvd,
_1f_Lucene50_0.dvm, _1f_Lucene50_0.pos, _1f_Lucene50_0.tim,
_1f_Lucene50_0.tip, _1g.fdt, _1g.fdx, _1g.fnm, _1g.nvd, _1g.nvm, _1g.si,
_1g.tvd, _1g.tvx, _1g_Lucene50_0.doc, _1g_Lucene50_0.dvd,
_1g_Lucene50_0.dvm, _1g_Lucene50_0.pos, _1g_Lucene50_0.tim,
_1g_Lucene50_0.tip, _1h.fdt, _1h.fdx, _1h.fnm, _1h.nvd, _1h.nvm, _1h.si,
_1h.tvd, _1h.tvx, _1h_Lucene50_0.doc, _1h_Lucene50_0.dvd,
_1h_Lucene50_0.dvm, _1h_Lucene50_0.pos, _1h_Lucene50_0.tim,
_1h_Lucene50_0.tip, _1i.fdt, _1i.fdx, _1i.fnm, _1i.nvd, _1i.nvm, _1i.si,
_1i.tvd, _1i.tvx, _1i_Lucene50_0.doc, _1i_Lucene50_0.dvd,
_1i_Lucene50_0.dvm, _1i_Lucene50_0.pos, _1i_Lucene50_0.tim,
_1i_Lucene50_0.tip, _1j.fdt, _1j.fdx, _1j.fnm, _1j.nvd, _1j.nvm, _1j.si,
_1j.tvd, _1j.tvx, _1j_Lucene50_0.doc, _1j_Lucene50_0.dvd,
_1j_Lucene50_0.dvm, _1j_Lucene50_0.pos, _1j_Lucene50_0.tim,
_1j_Lucene50_0.tip, _1k.fdt, _1k.fdx, _1k.fnm, _1k.nvd, _1k.nvm, _1k.si,
_1k.tvd, _1k.tvx, _1k_Lucene50_0.doc, _1k_Lucene50_0.dvd,
_1k_Lucene50_0.dvm, _1k_Lucene50_0.pos, _1k_Lucene50_0.tim,
_1k_Lucene50_0.tip, _1l.fdt, _1l.fdx, _1l.fnm, _1l.nvd, _1l.nvm, _1l.si,
_1l.tvd, _1l.tvx, _1l_Lucene50_0.doc, _1l_Lucene50_0.dvd,
_1l_Lucene50_0.dvm, _1l_Lucene50_0.pos, _1l_Lucene50_0.tim,
_1l_Lucene50_0.tip, _1m.fdt, _1m.fdx, _1m.fnm, _1m.nvd, _1m.nvm, _1m.si,
_1m.tvd, _1m.tvx, _1m_Lucene50_0.doc, _1m_Lucene50_0.dvd,
_1m_Lucene50_0.dvm, _1m_Lucene50_0.pos, _1m_Lucene50_0.tim,
_1m_Lucene50_0.tip, _1n.fdt, _1n.fdx, _1n.fnm, _1n.nvd, _1n.nvm, _1n.si,
_1n.tvd, _1n.tvx, _1n_Lucene50_0.doc, _1n_Lucene50_0.dvd,
_1n_Lucene50_0.dvm, _1n_Lucene50_0.pos, _1n_Lucene50_0.tim,
_1n_Lucene50_0.tip, _1p.fdt, _1p.fdx, _1p.fnm, _1p.nvd, _1p.nvm, _1p.si,
_1p.tvd, _1p.tvx, _1p_Lucene50_0.doc, _1p_Lucene50_0.dvd,
_1p_Lucene50_0.dvm, _1p_Lucene50_0.pos, _1p_Lucene50_0.tim,
_1p_Lucene50_0.tip, _1s.fdt, _1s.fdx, _1s.fnm, _1s.nvd, _1s.nvm, _1s.si,
_1s.tvx, _1s_Lucene50_0.doc, _1s_Lucene50_0.dvd, _1s_Lucene50_0.dvm,
_1s_Lucene50_0.pos, _1t.fdt, _1t.fdx, _1t.fnm, _1t.nvd, _1t.nvm,
_1t.tvd, _1t.tvx, _1t_Lucene50_0.doc, _1t_Lucene50_0.dvd,
_1t_Lucene50_0.dvm, _1t_Lucene50_0.pos, _1t_Lucene50_0.tim,
_1t_Lucene50_0.tip, _1v.fdt, _1v.fdx, _1v.fnm, _1v.nvd, _1v.nvm, _1v.si,
_1v.tvd, _1v.tvx, _1v_Lucene50_0.doc, _1v_Lucene50_0.dvd,
_1v_Lucene50_0.dvm, _1v_Lucene50_0.pos, _1v_Lucene50_0.tim,
_1v_Lucene50_0.tip, _22.fdt, _22.fdx, _22.fnm, _22.nvd, _22.nvm, _22.si,
_22.tvd, _22.tvx, _22_Lucene50_0.doc, _22_Lucene50_0.dvd,
_22_Lucene50_0.dvm, _22_Lucene50_0.pos, _22_Lucene50_0.tim,
_22_Lucene50_0.tip, _23.si, _23.tvd, _23.tvx, _23_Lucene50_0.doc,
_23_Lucene50_0.dvd, _23_Lucene50_0.dvm, _23_Lucene50_0.pos,
_23_Lucene50_0.tim, _23_Lucene50_0.tip, _24.fdt, _24.fdx, _24.fnm,
_24.nvd, _24.nvm, _24.si, _24.tvd, _24.tvx, _24_Lucene50_0.doc,
_24_Lucene50_0.dvd, _24_Lucene50_0.dvm, _24_Lucene50_0.pos,
_24_Lucene50_0.tim, _24_Lucene50_0.tip, _25.fdt, _25.fdx, _25.nvd,
_25.nvm, _25.si, _25.tvd, _25.tvx, _25_Lucene50_0.doc,
_25_Lucene50_0.dvd, _25_Lucene50_0.dvm, _25_Lucene50_0.pos,
_25_Lucene50_0.tim, _25_Lucene50_0.tip, _26.fdt, _26.fdx, _26.fnm,
_26.nvd, _26.nvm, _26.si, _26.tvd, _26.tvx, _26_Lucene50_0.doc,
_26_Lucene50_0.dvd, _26_Lucene50_0.dvm, _26_Lucene50_0.pos,
_26_Lucene50_0.tim, _26_Lucene50_0.tip, _27.fdt, _27.fdx, _27.fnm,
_27.si, _27.tvx, _27_Lucene50_0.doc, _27_Lucene50_0.dvd,
_27_Lucene50_0.dvm, _27_Lucene50_0.pos, _27_Lucene50_0.tim,
_27_Lucene50_0.tip, _u.fdt, _u.fdx, _u.fnm, _u.nvd, _u.nvm, _u.si,
_u.tvd, _u.tvx, _u_Lucene50_0.doc, _u_Lucene50_0.dvd, _u_Lucene50_0.dvm,
_u_Lucene50_0.pos, _u_Lucene50_0.tim, _u_Lucene50_0.tip, _v.fdt, _v.fdx,
_v.fnm, _v.nvd, _v.nvm, _v.si, _v.tvd, _v.tvx, _v_Lucene50_0.doc,
_v_Lucene50_0.dvd, _v_Lucene50_0.dvm, _v_Lucene50_0.pos,
_v_Lucene50_0.tim, _v_Lucene50_0.tip, _w.fdt, _w.fdx, _w.fnm, _w.nvd,
_w.nvm, _w.si, _w.tvd, _w.tvx, _w_Lucene50_0.doc, _w_Lucene50_0.dvd,
_w_Lucene50_0.dvm, _w_Lucene50_0.pos, _w_Lucene50_0.tim,
_w_Lucene50_0.tip, _x.fdt, _x.fdx, _x.fnm, _x.nvd, _x.nvm, _x.si,
_x.tvd, _x.tvx, _x_Lucene50_0.doc, _x_Lucene50_0.dvd, _x_Lucene50_0.dvm,
_x_Lucene50_0.pos, _x_Lucene50_0.tim, _x_Lucene50_0.tip, _y.fdt, _y.fdx,
_y.fnm, _y.nvd, _y.nvm, _y.si, _y.tvd, _y.tvx, _y_Lucene50_0.doc,
_y_Lucene50_0.dvd, _y_Lucene50_0.dvm, _y_Lucene50_0.pos,
_y_Lucene50_0.tim, _y_Lucene50_0.tip, _z.fdt, _z.fdx, _z.fnm, _z.nvd,
_z.nvm, _z.si, _z.tvd, _z.tvx, _z_Lucene50_0.doc, _z_Lucene50_0.dvd,
_z_Lucene50_0.dvm, _z_Lucene50_0.pos, _z_Lucene50_0.tim,
_z_Lucene50_0.tip, pending_segments_b, segments_9, segments_a, write.lock]
at
org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:103)
at
org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:208)
at
org.apache.solr.handler.ReplicationHandler.getIndexSize(ReplicationHandler.java:583)
at
org.apache.solr.handler.ReplicationHandler.getStatistics(ReplicationHandler.java:619)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.addMBean(SolrInfoMBeanHandler.java:165)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getMBeanInfo(SolrInfoMBeanHandler.java:135)
at
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:66)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Re: Solr and HDFS configuration
Posted by Michael Della Bitta <mi...@appinions.com>.
The ultimate answer is that you need to test your configuration with your
expected workflow.
However, the thing that mitigates the remote IO factor (hopefully) is that
the Solr HDFS stuff features a blockcache that should (when tuned
correctly) cache in RAM the blocks your Solr process needs the most.
Solr on HDFS currently doesn't have any sort of rack locality like there is
with say HBase colocated on the HDFS nodes. So you can expect that even
with Solr installed on the same nodes as your datanodes for HDFS, that
there will be remote IO.
Michael Della Bitta
Senior Software Engineer
o: +1 646 532 3062
appinions inc.
“The Science of Influence Marketing”
18 East 41st Street
New York, NY 10017
t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>
On Tue, Mar 24, 2015 at 2:47 PM, Joseph Obernberger <joeo@lovehorsepower.com
> wrote:
> Hi All - does it make sense to run a solr shard on a node within an Hadoop
> cluster that is not a data node? In that case all the data that node
> processes would need to come over the network, but you get the benefit of
> more CPU for things like faceting.
> Thank you!
>
> -Joe
>