You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Anuj Bhargava <an...@gmail.com> on 2024/01/12 06:05:39 UTC

Solr-8.11.2 Import of Large Data Issue

I am using solr-8.11.2. My MySql table has 13001335 which is around 259.9
GB.

solr.in.sh has the following entry -
SOLR_JAVA_MEM="-Xms15g -Xmx15g"
SOLR_OPTS="$SOLR_OPTS -Dsolr.disable.shardsWhitelist=true"


<dataConfig>
<dataSource type="JdbcDataSource"
            driver="com.mysql.jdbc.Driver"
            batchSize="-1"
            autoReconnect="true"
            useSSL="false"
            socketTimeout="0"
            connectTimeout="0"
            encoding="UTF-8"

url="jdbc:mysql://xxx.xxx.xx.xxx:3306/archive_dbase?zeroDateTimeBehavior=convertToNull"
            user="archive_user"
            password="password66"/>
<document>
<entity name="newarchive" pk="ID"
  query="SELECT * FROM newarchive"
  deltaImportQuery="SELECT * FROM newarchive
    WHERE posting_id = '${dataimporter.delta.posting_id}'"
  deltaQuery="SELECT posting_id FROM newarchive
    WHERE Deadline > '${dataimporter.last_index_time}'">
</entity>
</document>
</dataConfig>

Whenever I import the MySql stops after sometime and then restarts. Not all
records are imported. What changes do I need to make?

Regards,

Anuj

Re: Solr-8.11.2 Import of Large Data Issue

Posted by kshitij tyagi <ks...@gmail.com>.
Hi anuj

It is recommended to create a proper pipeline for indexing if data set is
huge else this problem will be recurring

Regards
Kshitij

On Fri, 12 Jan 2024 at 10:06 AM Anuj Bhargava <an...@gmail.com> wrote:

> I am using solr-8.11.2. My MySql table has 13001335 which is around 259.9
> GB.
>
> solr.in.sh has the following entry -
> SOLR_JAVA_MEM="-Xms15g -Xmx15g"
> SOLR_OPTS="$SOLR_OPTS -Dsolr.disable.shardsWhitelist=true"
>
>
> <dataConfig>
> <dataSource type="JdbcDataSource"
>             driver="com.mysql.jdbc.Driver"
>             batchSize="-1"
>             autoReconnect="true"
>             useSSL="false"
>             socketTimeout="0"
>             connectTimeout="0"
>             encoding="UTF-8"
>
>
> url="jdbc:mysql://xxx.xxx.xx.xxx:3306/archive_dbase?zeroDateTimeBehavior=convertToNull"
>             user="archive_user"
>             password="password66"/>
> <document>
> <entity name="newarchive" pk="ID"
>   query="SELECT * FROM newarchive"
>   deltaImportQuery="SELECT * FROM newarchive
>     WHERE posting_id = '${dataimporter.delta.posting_id}'"
>   deltaQuery="SELECT posting_id FROM newarchive
>     WHERE Deadline > '${dataimporter.last_index_time}'">
> </entity>
> </document>
> </dataConfig>
>
> Whenever I import the MySql stops after sometime and then restarts. Not all
> records are imported. What changes do I need to make?
>
> Regards,
>
> Anuj
>