You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by rahul gidwani <ch...@apache.org> on 2025/08/14 22:47:42 UTC
bulk upload ability
hello friends,
I am new to solr, I had a question, does solr support a bulk upload feature
where you can add segments to an existing collection in real-time. Sort of
like how HBase and Cassandra do?
If there is no feature or ability to do this, is there anything in the code
that would prevent this feature from being implemented?
Thank you
rahul
Re: bulk upload ability
Posted by Robi Petersen <ro...@gmail.com>.
I worked for Rakuten and they had a custom stack: Cassandra Hadoop Solr.
Index into Cassandra, Hadoop crunches docs into segments with a plugin
which get merged into the live Solr index.
On Thu, Aug 14, 2025, 7:02 PM Gus Heck <gu...@gmail.com> wrote:
> Only good way. ;) I've seen folks feed one doc at a time and force a
> commit once per doc. It's not transactional per se, but defintely not bulk
> either.
>
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
> On Thu, Aug 14, 2025, 9:14 PM Walter Underwood <wu...@wunderwood.org>
> wrote:
>
> > Short version, bulk upload is the only way to get data into Solr. There
> is
> > no transactional interface.
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> > > On Aug 14, 2025, at 5:53 PM, Gus Heck <gu...@gmail.com> wrote:
> > >
> > > Hi and welcome :)
> > >
> > > There are a variety of interfaces that you can use to send multiple
> > > documents at a time. (you can start on this, page, more info on the
> next
> > > few pages as well
> > >
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html
> > > )
> > >
> > > Sending in batches using those interfaces is standard practice.
> > >
> > > If you mean pre-calculating the indexed data to minimize load on the
> > > server, then pre-indexed fields are one option. (
> > >
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/external-files-processes.html#the-preanalyzedfield-type
> > )
> > > - for that JesterJ (a project I (mostly) wrote) has some built in
> > support:
> > >
> > >
> >
> https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/PreAnalyzeFields.java
> > > - I've used that particular processor in one project successfully so
> far.
> > >
> > > I've also heard of folks indexing on one system and then copying or
> > > replicating indexes to a destination system. (this is a custom
> engineered
> > > type of thing)
> > >
> > > All of the above is subject to our commit intervals and/or manual
> commit
> > > requests (typically only use the manual requests in special cases with
> > > careful planning)
> > >
> > > So there are several possibilities (and also some I haven't mentioned
> > > involving streaming expressions), but it would help to have a more
> > detailed
> > > description of the problem you are trying to solve (as opposed to
> asking
> > > after the solution you expect to need).
> > >
> > > -Gus
> > >
> > > http://www.needhamsoftware.com (work)
> > > https://a.co/d/b2sZLD9 (my fantasy fiction book)
> >
> >
>
Re: bulk upload ability
Posted by Walter Underwood <wu...@wunderwood.org>.
Sending one document at a time is still using the bulk load API, just inefficiently. :-)
It really is not transactional. No rollback, no isolation, Just not ACID at all.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Aug 14, 2025, at 7:00 PM, Gus Heck <gu...@gmail.com> wrote:
>
> Only good way. ;) I've seen folks feed one doc at a time and force a
> commit once per doc. It's not transactional per se, but defintely not bulk
> either.
>
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
> On Thu, Aug 14, 2025, 9:14 PM Walter Underwood <wu...@wunderwood.org>
> wrote:
>
>> Short version, bulk upload is the only way to get data into Solr. There is
>> no transactional interface.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>> On Aug 14, 2025, at 5:53 PM, Gus Heck <gu...@gmail.com> wrote:
>>>
>>> Hi and welcome :)
>>>
>>> There are a variety of interfaces that you can use to send multiple
>>> documents at a time. (you can start on this, page, more info on the next
>>> few pages as well
>>>
>> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html
>>> )
>>>
>>> Sending in batches using those interfaces is standard practice.
>>>
>>> If you mean pre-calculating the indexed data to minimize load on the
>>> server, then pre-indexed fields are one option. (
>>>
>> https://solr.apache.org/guide/solr/latest/indexing-guide/external-files-processes.html#the-preanalyzedfield-type
>> )
>>> - for that JesterJ (a project I (mostly) wrote) has some built in
>> support:
>>>
>>>
>> https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/PreAnalyzeFields.java
>>> - I've used that particular processor in one project successfully so far.
>>>
>>> I've also heard of folks indexing on one system and then copying or
>>> replicating indexes to a destination system. (this is a custom engineered
>>> type of thing)
>>>
>>> All of the above is subject to our commit intervals and/or manual commit
>>> requests (typically only use the manual requests in special cases with
>>> careful planning)
>>>
>>> So there are several possibilities (and also some I haven't mentioned
>>> involving streaming expressions), but it would help to have a more
>> detailed
>>> description of the problem you are trying to solve (as opposed to asking
>>> after the solution you expect to need).
>>>
>>> -Gus
>>>
>>> http://www.needhamsoftware.com (work)
>>> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>>
>>
Re: bulk upload ability
Posted by Gus Heck <gu...@gmail.com>.
Only good way. ;) I've seen folks feed one doc at a time and force a
commit once per doc. It's not transactional per se, but defintely not bulk
either.
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)
On Thu, Aug 14, 2025, 9:14 PM Walter Underwood <wu...@wunderwood.org>
wrote:
> Short version, bulk upload is the only way to get data into Solr. There is
> no transactional interface.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Aug 14, 2025, at 5:53 PM, Gus Heck <gu...@gmail.com> wrote:
> >
> > Hi and welcome :)
> >
> > There are a variety of interfaces that you can use to send multiple
> > documents at a time. (you can start on this, page, more info on the next
> > few pages as well
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html
> > )
> >
> > Sending in batches using those interfaces is standard practice.
> >
> > If you mean pre-calculating the indexed data to minimize load on the
> > server, then pre-indexed fields are one option. (
> >
> https://solr.apache.org/guide/solr/latest/indexing-guide/external-files-processes.html#the-preanalyzedfield-type
> )
> > - for that JesterJ (a project I (mostly) wrote) has some built in
> support:
> >
> >
> https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/PreAnalyzeFields.java
> > - I've used that particular processor in one project successfully so far.
> >
> > I've also heard of folks indexing on one system and then copying or
> > replicating indexes to a destination system. (this is a custom engineered
> > type of thing)
> >
> > All of the above is subject to our commit intervals and/or manual commit
> > requests (typically only use the manual requests in special cases with
> > careful planning)
> >
> > So there are several possibilities (and also some I haven't mentioned
> > involving streaming expressions), but it would help to have a more
> detailed
> > description of the problem you are trying to solve (as opposed to asking
> > after the solution you expect to need).
> >
> > -Gus
> >
> > http://www.needhamsoftware.com (work)
> > https://a.co/d/b2sZLD9 (my fantasy fiction book)
>
>
Re: bulk upload ability
Posted by Walter Underwood <wu...@wunderwood.org>.
Short version, bulk upload is the only way to get data into Solr. There is no transactional interface.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Aug 14, 2025, at 5:53 PM, Gus Heck <gu...@gmail.com> wrote:
>
> Hi and welcome :)
>
> There are a variety of interfaces that you can use to send multiple
> documents at a time. (you can start on this, page, more info on the next
> few pages as well
> https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html
> )
>
> Sending in batches using those interfaces is standard practice.
>
> If you mean pre-calculating the indexed data to minimize load on the
> server, then pre-indexed fields are one option. (
> https://solr.apache.org/guide/solr/latest/indexing-guide/external-files-processes.html#the-preanalyzedfield-type)
> - for that JesterJ (a project I (mostly) wrote) has some built in support:
>
> https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/PreAnalyzeFields.java
> - I've used that particular processor in one project successfully so far.
>
> I've also heard of folks indexing on one system and then copying or
> replicating indexes to a destination system. (this is a custom engineered
> type of thing)
>
> All of the above is subject to our commit intervals and/or manual commit
> requests (typically only use the manual requests in special cases with
> careful planning)
>
> So there are several possibilities (and also some I haven't mentioned
> involving streaming expressions), but it would help to have a more detailed
> description of the problem you are trying to solve (as opposed to asking
> after the solution you expect to need).
>
> -Gus
>
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
Re: bulk upload ability
Posted by Gus Heck <gu...@gmail.com>.
Hi and welcome :)
There are a variety of interfaces that you can use to send multiple
documents at a time. (you can start on this, page, more info on the next
few pages as well
https://solr.apache.org/guide/solr/latest/indexing-guide/indexing-with-update-handlers.html
)
Sending in batches using those interfaces is standard practice.
If you mean pre-calculating the indexed data to minimize load on the
server, then pre-indexed fields are one option. (
https://solr.apache.org/guide/solr/latest/indexing-guide/external-files-processes.html#the-preanalyzedfield-type)
- for that JesterJ (a project I (mostly) wrote) has some built in support:
https://github.com/nsoft/jesterj/blob/master/code/ingest/src/main/java/org/jesterj/ingest/processors/PreAnalyzeFields.java
- I've used that particular processor in one project successfully so far.
I've also heard of folks indexing on one system and then copying or
replicating indexes to a destination system. (this is a custom engineered
type of thing)
All of the above is subject to our commit intervals and/or manual commit
requests (typically only use the manual requests in special cases with
careful planning)
So there are several possibilities (and also some I haven't mentioned
involving streaming expressions), but it would help to have a more detailed
description of the problem you are trying to solve (as opposed to asking
after the solution you expect to need).
-Gus
http://www.needhamsoftware.com (work)
https://a.co/d/b2sZLD9 (my fantasy fiction book)