automatic delta imports?

[Mark Juszczec <ma...@gmail.com>]

Hello all

I'm seeing the following in my web server log file:

[2011-12-19 08:57:00.016] [customersIndex] webapp=/solr path=/dataimport
params={command=delta-import&commit=true&optimize=true} status=0 QTime=3
[2011-12-19 08:57:00.018] Starting Delta Import
[2011-12-19 08:57:00.018] Read dataimport.properties
[2011-12-19 08:57:00.019] Starting delta collection.
[2011-12-19 08:57:00.019] Running ModifiedRowKey() for Entity: CUSTOMERS
[2011-12-19 08:57:00.019] [ordersIndex] webapp=/solr path=/dataimport
params={command=delta-import&commit=true&optimize=true} status=0 QTime=1
[2011-12-19 08:57:00.023] Starting Delta Import
[2011-12-19 08:57:00.023] Creating a connection for entity CUSTOMERS with
URL: jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST =
somehost)(PORT = 1521))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME =
someservice)))
[2011-12-19 08:57:00.023] Read dataimport.properties
[2011-12-19 08:57:00.024] Starting delta collection.
[2011-12-19 08:57:00.024] Running ModifiedRowKey() for Entity: item2
[2011-12-19 08:57:00.025] Creating a connection for entity item2 with URL:
jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST =
somehost)(PORT
= 1521))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = someservice)))
[2011-12-19 08:57:01.005] Time taken for getConnection(): 982
[2011-12-19 08:57:01.010] Time taken for getConnection(): 985
[2011-12-19 08:57:01.216] Completed ModifiedRowKey for Entity: CUSTOMERS
rows obtained : 0
[2011-12-19 08:57:01.217] Completed DeletedRowKey for Entity: CUSTOMERS
rows obtained : 0
[2011-12-19 08:57:01.217] Completed parentDeltaQuery for Entity: CUSTOMERS
[2011-12-19 08:57:01.217] Delta Import completed successfully
[2011-12-19 08:57:01.217] {} 0 3
[2011-12-19 08:57:01.217] Time taken = 0:0:1.199
[2011-12-19 08:57:02.226] Completed ModifiedRowKey for Entity: item2 rows
obtained : 0
[2011-12-19 08:57:02.227] Completed DeletedRowKey for Entity: item2 rows
obtained : 0
[2011-12-19 08:57:02.227] Completed parentDeltaQuery for Entity: item2
[2011-12-19 08:57:02.227] Delta Import completed successfully
[2011-12-19 08:57:02.227] {} 0 1
[2011-12-19 08:57:02.227] Time taken = 0:0:2.204

It sure looks like its an automatic delta import running on my 2 indexes.
 But I thought the docs said Solr didn't provide for automatic delta
imports?  Is this a new feature in 3.2?  I've got multiple 3.2 instances
running and this is the only one who's logs show this message.  Have I
turned something on accidentally?  If so, what config files contain these
settings?  I want to turn this on for the other solr instances.

Mark

Re: automatic delta imports?

[Shawn Heisey <so...@elyograg.org>]

On 12/19/2011 7:09 AM, Mark Juszczec wrote:
> Hello all
>
> I'm seeing the following in my web server log file:
>
> [2011-12-19 08:57:00.016] [customersIndex] webapp=/solr path=/dataimport
> params={command=delta-import&commit=true&optimize=true} status=0 QTime=3
> [2011-12-19 08:57:00.018] Starting Delta Import
> [2011-12-19 08:57:00.018] Read dataimport.properties
> [2011-12-19 08:57:00.019] Starting delta collection.
> [2011-12-19 08:57:00.019] Running ModifiedRowKey() for Entity: CUSTOMERS
> [2011-12-19 08:57:00.019] [ordersIndex] webapp=/solr path=/dataimport
> params={command=delta-import&commit=true&optimize=true} status=0 QTime=1
> [2011-12-19 08:57:00.023] Starting Delta Import
> [2011-12-19 08:57:00.023] Creating a connection for entity CUSTOMERS with
> URL: jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST =
> somehost)(PORT = 1521))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME =
> someservice)))
> [2011-12-19 08:57:00.023] Read dataimport.properties
> [2011-12-19 08:57:00.024] Starting delta collection.
> [2011-12-19 08:57:00.024] Running ModifiedRowKey() for Entity: item2
> [2011-12-19 08:57:00.025] Creating a connection for entity item2 with URL:
> jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST =
> somehost)(PORT
> = 1521))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = someservice)))
> [2011-12-19 08:57:01.005] Time taken for getConnection(): 982
> [2011-12-19 08:57:01.010] Time taken for getConnection(): 985
> [2011-12-19 08:57:01.216] Completed ModifiedRowKey for Entity: CUSTOMERS
> rows obtained : 0
> [2011-12-19 08:57:01.217] Completed DeletedRowKey for Entity: CUSTOMERS
> rows obtained : 0
> [2011-12-19 08:57:01.217] Completed parentDeltaQuery for Entity: CUSTOMERS
> [2011-12-19 08:57:01.217] Delta Import completed successfully
> [2011-12-19 08:57:01.217] {} 0 3
> [2011-12-19 08:57:01.217] Time taken = 0:0:1.199
> [2011-12-19 08:57:02.226] Completed ModifiedRowKey for Entity: item2 rows
> obtained : 0
> [2011-12-19 08:57:02.227] Completed DeletedRowKey for Entity: item2 rows
> obtained : 0
> [2011-12-19 08:57:02.227] Completed parentDeltaQuery for Entity: item2
> [2011-12-19 08:57:02.227] Delta Import completed successfully
> [2011-12-19 08:57:02.227] {} 0 1
> [2011-12-19 08:57:02.227] Time taken = 0:0:2.204
>
> It sure looks like its an automatic delta import running on my 2 indexes.
>   But I thought the docs said Solr didn't provide for automatic delta
> imports?  Is this a new feature in 3.2?  I've got multiple 3.2 instances
> running and this is the only one who's logs show this message.  Have I
> turned something on accidentally?  If so, what config files contain these
> settings?  I want to turn this on for the other solr instances.

I'm fairly sure there's no ability for Solr itself to initiate data 
indexing, it has to be some other process, possibly one run from cron or 
scheduled tasks.  I'm running 3.5.0, and have not heard of such a thing 
even in 4.0 (trunk).  The first and sixth lines of your logfile show 
requests to the /dataimport URL, one for a core named customersIndex, 
the other for a core named ordersIndex.

These lines are produced by Solr, your container may have a separate log 
for incoming HTTP requests, which should show the IP address of the 
sender.  From that you will be able to direct your search for the 
application that is sending them.

Thanks,
Shawn