Export big extract from Solr to [My]SQL

[Per Steffensen <st...@designware.dk>]

Hi

I want to make extracts from my Solr to MySQL. Any tools around that can 
help med perform such a task? I find a lot about data-import from SQL 
when googling, but nothing about export/extract. It is not all of the 
data in Solr I need to extract. It is only documents that full fill a 
normal Solr query, but the number of documents fulfilling it will 
(potentially) be huge.

Regards, Per Steffensen

Re: Export big extract from Solr to [My]SQL

[simon <mt...@gmail.com>]

The cursor-based deep paging in 4.7+ works very well and the performance on
large extracts (for us, maybe  up to 100K documents) is excellent, though
it will obviously depend on the number and size of fields that you need to
 pull. I wrote a Perl module to do the extractions from Solr without
problems (and DBI takes care of  writing to a database).

I'm probably going to rewrite in Python since the final destination of many
of our extracts is Tableau,  which has  a Python API for creating TDEs
(Tableau data extracts)

regards

-Simon


On Fri, May 2, 2014 at 7:43 AM, Siegfried Goeschl <sg...@gmx.at> wrote:

> Hi Per,
>
> basically I see three options
>
> * use a lot of memory to scope with huge result sets
> * user result set paging
> * SOLR 4.7 supports cursors (https://issues.apache.org/
> jira/browse/SOLR-5463)
>
> Cheers,
>
> Siegfried Goeschl
>
>
> On 02.05.14 13:32, Per Steffensen wrote:
>
>> Hi
>>
>> I want to make extracts from my Solr to MySQL. Any tools around that can
>> help med perform such a task? I find a lot about data-import from SQL
>> when googling, but nothing about export/extract. It is not all of the
>> data in Solr I need to extract. It is only documents that full fill a
>> normal Solr query, but the number of documents fulfilling it will
>> (potentially) be huge.
>>
>> Regards, Per Steffensen
>>
>
>

Re: Export big extract from Solr to [My]SQL

[Siegfried Goeschl <sg...@gmx.at>]

Hi Per,

basically I see three options

* use a lot of memory to scope with huge result sets
* user result set paging
* SOLR 4.7 supports cursors 
(https://issues.apache.org/jira/browse/SOLR-5463)

Cheers,

Siegfried Goeschl

On 02.05.14 13:32, Per Steffensen wrote:
> Hi
>
> I want to make extracts from my Solr to MySQL. Any tools around that can
> help med perform such a task? I find a lot about data-import from SQL
> when googling, but nothing about export/extract. It is not all of the
> data in Solr I need to extract. It is only documents that full fill a
> normal Solr query, but the number of documents fulfilling it will
> (potentially) be huge.
>
> Regards, Per Steffensen