You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Husain, Yavar" <yh...@firstam.com> on 2011/11/23 11:57:49 UTC
DIH Strange Problem
I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
...
data-config.xml:
....
<dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/>
<document>
.....
.....
Logs:
INFO: Server startup in 2016 ms
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO: Read dataimport.properties
Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1322041133719
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
</PRE>
<BR>
******************************************************************************************<BR>This message may contain confidential or proprietary information intended only for the use of the<BR>addressee(s) named above or may contain information that is legally privileged. If you are<BR>not the intended addressee, or the person responsible for delivering it to the intended addressee,<BR>you are hereby notified that reading, disseminating, distributing or copying this message is strictly<BR>prohibited. If you have received this message by mistake, please immediately notify us by<BR>replying to the message and delete the original message and any copies immediately thereafter.<BR>
<BR>
Thank you.~<BR>
******************************************************************************************<BR>
FAFLD<BR>
<PRE>
Re: DIH Strange Problem
Posted by Ian Grainger <ia...@isfluent.com>.
Aha! That sounds like it might be it!
On Mon, Nov 28, 2011 at 4:16 PM, Husain, Yavar <yh...@firstam.com> wrote:
>
> Thanks Kai for sharing this. Ian encountered the same problem so marking him in the mail too.
> ________________________________________
> From: Kai Gülzau [kguelzau@novomind.com]
> Sent: Monday, November 28, 2011 6:55 PM
> To: solr-user@lucene.apache.org
> Subject: RE: DIH Strange Problem
>
> Do you use Java 6 update 29? There is a known issue with the latest mssql driver:
>
> http://blogs.msdn.com/b/jdbcteam/archive/2011/11/07/supported-java-versions-november-2011.aspx
>
> "In addition, there are known connection failure issues with Java 6 update 29, and the developer preview (non production) versions of Java 6 update 30 and Java 6 update 30 build 12. We are in contact with Java on these issues and we will update this blog once we have more information."
>
> Should work with update 28.
>
> Kai
>
> -----Original Message-----
> From: Husain, Yavar [mailto:yhusain@firstam.com]
> Sent: Monday, November 28, 2011 1:02 PM
> To: solr-user@lucene.apache.org; Shawn Heisey
> Subject: RE: DIH Strange Problem
>
> I figured out the solution and Microsoft and not Solr is the problem here :):
>
> I downloaded and build latest Solr (3.4) from sources and finally hit following line of code in Solr (where I put my debug statement) :
>
> if(url != null){
> LOG.info("Yavar: getting handle to driver manager:");
> c = DriverManager.getConnection(url, initProps);
> LOG.info("Yavar: got handle to driver manager:"); }
>
> The call to Driver Manager was not returning. Here was the error!! The Driver we were using was Microsoft Type 4 JDBC driver for SQL Server. I downloaded another driver called jTDS jDBC driver and installed that. Problem got fixed!!!
>
> So please follow the following steps:
>
> 1. Download jTDS jDBC driver from http://jtds.sourceforge.net/ 2. Put the driver jar file into your Solr/lib directory where you had put Microsoft JDBC driver.
> 3. In the data-config.xml use this statement: driver="net.sourceforge.jtds.jdbc.Driver"
> 4. Also in data-config.xml mention url like this: "url="jdbc:jTDS:sqlserver://localhost:1433;databaseName=XXX"
> 5. Now run your indexing.
>
> It should solve the problem.
>
> -----Original Message-----
> From: Husain, Yavar
> Sent: Thursday, November 24, 2011 12:38 PM
> To: solr-user@lucene.apache.org; Shawn Heisey
> Subject: RE: DIH Strange Problem
>
> Hi
>
> Thanks for your replies.
>
> I carried out these 2 steps (it did not solve my problem):
>
> 1. I tried setting responseBuffering to adaptive. Did not work.
> 2. For checking Database connection I wrote a simple java program to connect to database and fetch some results with the same driver that I use for solr. It worked. So it does not seem to be a problem with the connection.
>
> Now I am stuck where Tomcat log says: "Creating a connection for entity ....." and does nothing, I mean after this log we usually get the "getConnection() took x millisecond" however I dont get that ,I can just see the time moving with no records getting fetched.
>
> Original Problem listed again:
>
>
> I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
> ...
> data-config.xml:
> ....
> <dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
> url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/> <document> .....
> .....
>
> Logs:
>
> INFO: Server startup in 2016 ms
> Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
> INFO: Starting Full Import
> Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11 Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
> INFO: Read dataimport.properties
> Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
> INFO: [] REMOVING ALL DOCUMENTS FROM INDEX Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
> Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
> INFO: newest commit = 1322041133719
> Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
>
>
> -----Original Message-----
> From: Shawn Heisey [mailto:solr@elyograg.org]
> Sent: Wednesday, November 23, 2011 7:36 PM
> To: solr-user@lucene.apache.org
> Subject: Re: DIH Strange Problem
>
> On 11/23/2011 5:21 AM, Chantal Ackermann wrote:
>> Hi Yavar,
>>
>> my experience with similar problems was that there was something wrong
>> with the database connection or the database.
>>
>> Chantal
>
> It's also possible that your JDBC driver might be trying to buffer the
> entire result set. There's a link on the wiki specifically for this
> problem on MS SQL server. Hopefully it's that, but Chantal could be
> right too.
>
> http://wiki.apache.org/solr/DataImportHandlerFaq
>
> Here's the URL to the specific paragraph, but it's likely that it won't
> survive the email trip in a clickable form:
>
> http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F
>
> Thanks,
> Shawn
>
> ******************************************************************************************
> This message may contain confidential or proprietary information intended only for the use of the
> addressee(s) named above or may contain information that is legally privileged. If you are
> not the intended addressee, or the person responsible for delivering it to the intended addressee,
> you are hereby notified that reading, disseminating, distributing or copying this message is strictly
> prohibited. If you have received this message by mistake, please immediately notify us by
> replying to the message and delete the original message and any copies immediately thereafter.
>
> Thank you.-
> ******************************************************************************************
> FAFLD
--
Ian
ian@isfluent.com
+44 (0)1223 257903
RE: DIH Strange Problem
Posted by "Husain, Yavar" <yh...@firstam.com>.
Thanks Kai for sharing this. Ian encountered the same problem so marking him in the mail too.
________________________________________
From: Kai Gülzau [kguelzau@novomind.com]
Sent: Monday, November 28, 2011 6:55 PM
To: solr-user@lucene.apache.org
Subject: RE: DIH Strange Problem
Do you use Java 6 update 29? There is a known issue with the latest mssql driver:
http://blogs.msdn.com/b/jdbcteam/archive/2011/11/07/supported-java-versions-november-2011.aspx
"In addition, there are known connection failure issues with Java 6 update 29, and the developer preview (non production) versions of Java 6 update 30 and Java 6 update 30 build 12. We are in contact with Java on these issues and we will update this blog once we have more information."
Should work with update 28.
Kai
-----Original Message-----
From: Husain, Yavar [mailto:yhusain@firstam.com]
Sent: Monday, November 28, 2011 1:02 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
I figured out the solution and Microsoft and not Solr is the problem here :):
I downloaded and build latest Solr (3.4) from sources and finally hit following line of code in Solr (where I put my debug statement) :
if(url != null){
LOG.info("Yavar: getting handle to driver manager:");
c = DriverManager.getConnection(url, initProps);
LOG.info("Yavar: got handle to driver manager:"); }
The call to Driver Manager was not returning. Here was the error!! The Driver we were using was Microsoft Type 4 JDBC driver for SQL Server. I downloaded another driver called jTDS jDBC driver and installed that. Problem got fixed!!!
So please follow the following steps:
1. Download jTDS jDBC driver from http://jtds.sourceforge.net/ 2. Put the driver jar file into your Solr/lib directory where you had put Microsoft JDBC driver.
3. In the data-config.xml use this statement: driver="net.sourceforge.jtds.jdbc.Driver"
4. Also in data-config.xml mention url like this: "url="jdbc:jTDS:sqlserver://localhost:1433;databaseName=XXX"
5. Now run your indexing.
It should solve the problem.
-----Original Message-----
From: Husain, Yavar
Sent: Thursday, November 24, 2011 12:38 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
Hi
Thanks for your replies.
I carried out these 2 steps (it did not solve my problem):
1. I tried setting responseBuffering to adaptive. Did not work.
2. For checking Database connection I wrote a simple java program to connect to database and fetch some results with the same driver that I use for solr. It worked. So it does not seem to be a problem with the connection.
Now I am stuck where Tomcat log says: "Creating a connection for entity ....." and does nothing, I mean after this log we usually get the "getConnection() took x millisecond" however I dont get that ,I can just see the time moving with no records getting fetched.
Original Problem listed again:
I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
...
data-config.xml:
....
<dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/> <document> .....
.....
Logs:
INFO: Server startup in 2016 ms
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11 Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO: Read dataimport.properties
Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1322041133719
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, November 23, 2011 7:36 PM
To: solr-user@lucene.apache.org
Subject: Re: DIH Strange Problem
On 11/23/2011 5:21 AM, Chantal Ackermann wrote:
> Hi Yavar,
>
> my experience with similar problems was that there was something wrong
> with the database connection or the database.
>
> Chantal
It's also possible that your JDBC driver might be trying to buffer the
entire result set. There's a link on the wiki specifically for this
problem on MS SQL server. Hopefully it's that, but Chantal could be
right too.
http://wiki.apache.org/solr/DataImportHandlerFaq
Here's the URL to the specific paragraph, but it's likely that it won't
survive the email trip in a clickable form:
http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F
Thanks,
Shawn
******************************************************************************************
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.
Thank you.-
******************************************************************************************
FAFLD
RE: DIH Strange Problem
Posted by Kai Gülzau <kg...@novomind.com>.
Do you use Java 6 update 29? There is a known issue with the latest mssql driver:
http://blogs.msdn.com/b/jdbcteam/archive/2011/11/07/supported-java-versions-november-2011.aspx
"In addition, there are known connection failure issues with Java 6 update 29, and the developer preview (non production) versions of Java 6 update 30 and Java 6 update 30 build 12. We are in contact with Java on these issues and we will update this blog once we have more information."
Should work with update 28.
Kai
-----Original Message-----
From: Husain, Yavar [mailto:yhusain@firstam.com]
Sent: Monday, November 28, 2011 1:02 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
I figured out the solution and Microsoft and not Solr is the problem here :):
I downloaded and build latest Solr (3.4) from sources and finally hit following line of code in Solr (where I put my debug statement) :
if(url != null){
LOG.info("Yavar: getting handle to driver manager:");
c = DriverManager.getConnection(url, initProps);
LOG.info("Yavar: got handle to driver manager:"); }
The call to Driver Manager was not returning. Here was the error!! The Driver we were using was Microsoft Type 4 JDBC driver for SQL Server. I downloaded another driver called jTDS jDBC driver and installed that. Problem got fixed!!!
So please follow the following steps:
1. Download jTDS jDBC driver from http://jtds.sourceforge.net/ 2. Put the driver jar file into your Solr/lib directory where you had put Microsoft JDBC driver.
3. In the data-config.xml use this statement: driver="net.sourceforge.jtds.jdbc.Driver"
4. Also in data-config.xml mention url like this: "url="jdbc:jTDS:sqlserver://localhost:1433;databaseName=XXX"
5. Now run your indexing.
It should solve the problem.
-----Original Message-----
From: Husain, Yavar
Sent: Thursday, November 24, 2011 12:38 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
Hi
Thanks for your replies.
I carried out these 2 steps (it did not solve my problem):
1. I tried setting responseBuffering to adaptive. Did not work.
2. For checking Database connection I wrote a simple java program to connect to database and fetch some results with the same driver that I use for solr. It worked. So it does not seem to be a problem with the connection.
Now I am stuck where Tomcat log says: "Creating a connection for entity ....." and does nothing, I mean after this log we usually get the "getConnection() took x millisecond" however I dont get that ,I can just see the time moving with no records getting fetched.
Original Problem listed again:
I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
...
data-config.xml:
....
<dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/> <document> .....
.....
Logs:
INFO: Server startup in 2016 ms
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11 Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO: Read dataimport.properties
Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1322041133719
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, November 23, 2011 7:36 PM
To: solr-user@lucene.apache.org
Subject: Re: DIH Strange Problem
On 11/23/2011 5:21 AM, Chantal Ackermann wrote:
> Hi Yavar,
>
> my experience with similar problems was that there was something wrong
> with the database connection or the database.
>
> Chantal
It's also possible that your JDBC driver might be trying to buffer the
entire result set. There's a link on the wiki specifically for this
problem on MS SQL server. Hopefully it's that, but Chantal could be
right too.
http://wiki.apache.org/solr/DataImportHandlerFaq
Here's the URL to the specific paragraph, but it's likely that it won't
survive the email trip in a clickable form:
http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F
Thanks,
Shawn
******************************************************************************************
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.
Thank you.-
******************************************************************************************
FAFLD
RE: DIH Strange Problem
Posted by "Husain, Yavar" <yh...@firstam.com>.
I figured out the solution and Microsoft and not Solr is the problem here :):
I downloaded and build latest Solr (3.4) from sources and finally hit following line of code in Solr (where I put my debug statement) :
if(url != null){
LOG.info("Yavar: getting handle to driver manager:");
c = DriverManager.getConnection(url, initProps);
LOG.info("Yavar: got handle to driver manager:");
}
The call to Driver Manager was not returning. Here was the error!! The Driver we were using was Microsoft Type 4 JDBC driver for SQL Server. I downloaded another driver called jTDS jDBC driver and installed that. Problem got fixed!!!
So please follow the following steps:
1. Download jTDS jDBC driver from http://jtds.sourceforge.net/
2. Put the driver jar file into your Solr/lib directory where you had put Microsoft JDBC driver.
3. In the data-config.xml use this statement: driver="net.sourceforge.jtds.jdbc.Driver"
4. Also in data-config.xml mention url like this: "url="jdbc:jTDS:sqlserver://localhost:1433;databaseName=XXX"
5. Now run your indexing.
It should solve the problem.
-----Original Message-----
From: Husain, Yavar
Sent: Thursday, November 24, 2011 12:38 PM
To: solr-user@lucene.apache.org; Shawn Heisey
Subject: RE: DIH Strange Problem
Hi
Thanks for your replies.
I carried out these 2 steps (it did not solve my problem):
1. I tried setting responseBuffering to adaptive. Did not work.
2. For checking Database connection I wrote a simple java program to connect to database and fetch some results with the same driver that I use for solr. It worked. So it does not seem to be a problem with the connection.
Now I am stuck where Tomcat log says: "Creating a connection for entity ....." and does nothing, I mean after this log we usually get the "getConnection() took x millisecond" however I dont get that ,I can just see the time moving with no records getting fetched.
Original Problem listed again:
I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
...
data-config.xml:
....
<dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/>
<document>
.....
.....
Logs:
INFO: Server startup in 2016 ms
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO: Read dataimport.properties
Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1322041133719
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, November 23, 2011 7:36 PM
To: solr-user@lucene.apache.org
Subject: Re: DIH Strange Problem
On 11/23/2011 5:21 AM, Chantal Ackermann wrote:
> Hi Yavar,
>
> my experience with similar problems was that there was something wrong
> with the database connection or the database.
>
> Chantal
It's also possible that your JDBC driver might be trying to buffer the
entire result set. There's a link on the wiki specifically for this
problem on MS SQL server. Hopefully it's that, but Chantal could be
right too.
http://wiki.apache.org/solr/DataImportHandlerFaq
Here's the URL to the specific paragraph, but it's likely that it won't
survive the email trip in a clickable form:
http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F
Thanks,
Shawn
******************************************************************************************
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.
Thank you.-
******************************************************************************************
FAFLD
RE: DIH Strange Problem
Posted by "Husain, Yavar" <yh...@firstam.com>.
Hi
Thanks for your replies.
I carried out these 2 steps (it did not solve my problem):
1. I tried setting responseBuffering to adaptive. Did not work.
2. For checking Database connection I wrote a simple java program to connect to database and fetch some results with the same driver that I use for solr. It worked. So it does not seem to be a problem with the connection.
Now I am stuck where Tomcat log says: "Creating a connection for entity ....." and does nothing, I mean after this log we usually get the "getConnection() took x millisecond" however I dont get that ,I can just see the time moving with no records getting fetched.
Original Problem listed again:
I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
...
data-config.xml:
....
<dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/>
<document>
.....
.....
Logs:
INFO: Server startup in 2016 ms
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO: Read dataimport.properties
Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1322041133719
Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, November 23, 2011 7:36 PM
To: solr-user@lucene.apache.org
Subject: Re: DIH Strange Problem
On 11/23/2011 5:21 AM, Chantal Ackermann wrote:
> Hi Yavar,
>
> my experience with similar problems was that there was something wrong
> with the database connection or the database.
>
> Chantal
It's also possible that your JDBC driver might be trying to buffer the
entire result set. There's a link on the wiki specifically for this
problem on MS SQL server. Hopefully it's that, but Chantal could be
right too.
http://wiki.apache.org/solr/DataImportHandlerFaq
Here's the URL to the specific paragraph, but it's likely that it won't
survive the email trip in a clickable form:
http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F
Thanks,
Shawn
******************************************************************************************
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.
Thank you.-
******************************************************************************************
FAFLD
Re: DIH Strange Problem
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/23/2011 5:21 AM, Chantal Ackermann wrote:
> Hi Yavar,
>
> my experience with similar problems was that there was something wrong
> with the database connection or the database.
>
> Chantal
It's also possible that your JDBC driver might be trying to buffer the
entire result set. There's a link on the wiki specifically for this
problem on MS SQL server. Hopefully it's that, but Chantal could be
right too.
http://wiki.apache.org/solr/DataImportHandlerFaq
Here's the URL to the specific paragraph, but it's likely that it won't
survive the email trip in a clickable form:
http://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F
Thanks,
Shawn
Re: DIH Strange Problem
Posted by Chantal Ackermann <ch...@btelligent.de>.
Hi Yavar,
my experience with similar problems was that there was something wrong
with the database connection or the database.
Chantal
On Wed, 2011-11-23 at 11:57 +0100, Husain, Yavar wrote:
> I am using Solr 1.4.1 on Windows/MS SQL Server and am using DIH for importing data. Indexing and all was working perfectly fine. However today when I started full indexing again, Solr halts/stucks at the line "Creating a connection for entity........." There are no further messages after that. I can see that DIH is busy and on the DIH console I can see "A command is still running", I can also see total rows fetched = 0 and total request made to datasource = 1 and time is increasing however it is not doing anything. This is the exact configuration that worked for me. I am not really able to understand the problem here. Also in the index directory where I am storing the index there are just 3 files: 2 segment files + 1 lucene*****-write.lock file.
> ...
> data-config.xml:
> ....
> <dataSource type="JdbcDataSource" driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
> url="jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders" user="testUser" password="password"/>
> <document>
> .....
> .....
>
> Logs:
>
> INFO: Server startup in 2016 ms
> Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.DataImporter doFullImport
> INFO: Starting Full Import
> Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/dataimport params={command=full-import} status=0 QTime=11
> Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
> INFO: Read dataimport.properties
> Nov 23, 2011 4:11:27 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
> INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
> Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=C:\solrindexes\index,segFN=segments_6,version=1322041133719,generation=6,filenames=[segments_6]
> Nov 23, 2011 4:11:27 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
> INFO: newest commit = 1322041133719
> Nov 23, 2011 4:11:27 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Creating a connection for entity SampleText with URL: jdbc:sqlserver://127.0.0.1:1433;databaseName=SampleOrders
> </PRE>
> <BR>
> ******************************************************************************************<BR>This message may contain confidential or proprietary information intended only for the use of the<BR>addressee(s) named above or may contain information that is legally privileged. If you are<BR>not the intended addressee, or the person responsible for delivering it to the intended addressee,<BR>you are hereby notified that reading, disseminating, distributing or copying this message is strictly<BR>prohibited. If you have received this message by mistake, please immediately notify us by<BR>replying to the message and delete the original message and any copies immediately thereafter.<BR>
> <BR>
> Thank you.~<BR>
> ******************************************************************************************<BR>
> FAFLD<BR>
> <PRE>