You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by 梁景明 <fu...@gmail.com> on 2009/09/24 03:57:48 UTC
Can solr build on top of HBase
hi, i use hbase and solr ,now i have a large data need to index ,it means
solr-index will be large,
as the data increases,it will be more larger than now.
so solrconfig.xml 's <dataDir>/solrhome/data</dataDir> ,can i used it from
api ,and point to my
distrabuted hbase data storage,
and if the index is too large ,will it be slow?
thanks.
Re: Can solr build on top of HBase
Posted by Grant Ingersoll <gs...@apache.org>.
I don't think using HDFS or HBase will perform for this kind of thing
at all. If you are that large, you should look into distributing your
index into shards and using Solr's distributed search capabilities.
-Grant
On Sep 24, 2009, at 3:25 AM, 梁景明 wrote:
> hi, thanks, and now i can index data from hbase to the solr server
> using
> nutch core.
> but the indexdata will be local storage,that 's what i worry
> about,to be too
> large in local.
>
> MountableHDFS i never use it ,i am not sure weather solr can write
> the index
> into HDFS,i doubt it
> can work without implements Writable in HDFS.
>
> and i think the point is the reading and writing the indexfile in
> HDFS just
> like it in local filesystem ,
> can u make a new index file format witch can use in the HDFS, if it
> can ,i
> think that will
> be a great helpful to distrabuted index.
>
> if solr built on top of lucene , will it be easy to implement the
> HDFS file
> format?
>
> 2009/9/24 Amit Nithian <an...@gmail.com>
>
>> Would FUSE (http://wiki.apache.org/hadoop/MountableHDFS) be of use?
>> I wonder if you could take the data from HBase and index it into a
>> Lucene
>> index stored on HDFS.
>>
>> 2009/9/23 Noble Paul നോബിള് नोब्ळ्
>> <no...@corp.aol.com>
>>
>>> can hbase be mounted on the filesystem? Solr can only read data
>>> from a
>>> filesystem
>>>
>>> On Thu, Sep 24, 2009 at 7:27 AM, 梁景明 <fu...@gmail.com>
>>> wrote:
>>>> hi, i use hbase and solr ,now i have a large data need to
>>>> index ,it
>>> means
>>>> solr-index will be large,
>>>> as the data increases,it will be more larger than now.
>>>> so solrconfig.xml 's <dataDir>/solrhome/data</dataDir> ,can i
>>>> used it
>>> from
>>>> api ,and point to my
>>>> distrabuted hbase data storage,
>>>> and if the index is too large ,will it be slow?
>>>> thanks.
>>>>
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>>
>>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: Can solr build on top of HBase
Posted by 梁景明 <fu...@gmail.com>.
hi, thanks, and now i can index data from hbase to the solr server using
nutch core.
but the indexdata will be local storage,that 's what i worry about,to be too
large in local.
MountableHDFS i never use it ,i am not sure weather solr can write the index
into HDFS,i doubt it
can work without implements Writable in HDFS.
and i think the point is the reading and writing the indexfile in HDFS just
like it in local filesystem ,
can u make a new index file format witch can use in the HDFS, if it can ,i
think that will
be a great helpful to distrabuted index.
if solr built on top of lucene , will it be easy to implement the HDFS file
format?
2009/9/24 Amit Nithian <an...@gmail.com>
> Would FUSE (http://wiki.apache.org/hadoop/MountableHDFS) be of use?
> I wonder if you could take the data from HBase and index it into a Lucene
> index stored on HDFS.
>
> 2009/9/23 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>
> > can hbase be mounted on the filesystem? Solr can only read data from a
> > filesystem
> >
> > On Thu, Sep 24, 2009 at 7:27 AM, 梁景明 <fu...@gmail.com> wrote:
> > > hi, i use hbase and solr ,now i have a large data need to index ,it
> > means
> > > solr-index will be large,
> > > as the data increases,it will be more larger than now.
> > > so solrconfig.xml 's <dataDir>/solrhome/data</dataDir> ,can i used it
> > from
> > > api ,and point to my
> > > distrabuted hbase data storage,
> > > and if the index is too large ,will it be slow?
> > > thanks.
> > >
> >
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul | Principal Engineer| AOL | http://aol.com
> >
>
Re: Can solr build on top of HBase
Posted by Amit Nithian <an...@gmail.com>.
Would FUSE (http://wiki.apache.org/hadoop/MountableHDFS) be of use?
I wonder if you could take the data from HBase and index it into a Lucene
index stored on HDFS.
2009/9/23 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
> can hbase be mounted on the filesystem? Solr can only read data from a
> filesystem
>
> On Thu, Sep 24, 2009 at 7:27 AM, 梁景明 <fu...@gmail.com> wrote:
> > hi, i use hbase and solr ,now i have a large data need to index ,it
> means
> > solr-index will be large,
> > as the data increases,it will be more larger than now.
> > so solrconfig.xml 's <dataDir>/solrhome/data</dataDir> ,can i used it
> from
> > api ,and point to my
> > distrabuted hbase data storage,
> > and if the index is too large ,will it be slow?
> > thanks.
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
Re: Can solr build on top of HBase
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
can hbase be mounted on the filesystem? Solr can only read data from a
filesystem
On Thu, Sep 24, 2009 at 7:27 AM, 梁景明 <fu...@gmail.com> wrote:
> hi, i use hbase and solr ,now i have a large data need to index ,it means
> solr-index will be large,
> as the data increases,it will be more larger than now.
> so solrconfig.xml 's <dataDir>/solrhome/data</dataDir> ,can i used it from
> api ,and point to my
> distrabuted hbase data storage,
> and if the index is too large ,will it be slow?
> thanks.
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com