You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Turner, Robbin J" <Ro...@boeing.com> on 2009/12/16 21:17:16 UTC
RE: Request Assistance with DIH, development console
Has there been any reported problems with -(hyphens) or _ (underscore)for the datasource connection and/or user/password information in the dataconfig.xml?
I've not been able to find out how to set the webapp property for the user.dir in tomcat. But my output from Catalina.out reports all htat on startup, I get nothing that indicates:
INFO: Starting Catalina
Dec 16, 2009 10:53:29 AM org.apache.Catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: Using JNDI solr.home: /opt/solr
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader <init>
INFO: Solr home set to '/opt/solr/'
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader replaceClassLoader
INFO: Adding 'file:/opt/solr/lib/ojdbc14.jar' to classloader
Dec 16, 2009 10:53:29 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: Using JNDI solr.home: /opt/solr
Dec 16, 2009 10:53:29 AM org.apache.solr.core.CoreContainer$Initializer initialize
INFO: looking for solr.xml: /opt/solr/solr.xml
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader <init>
INFO: Solr home set to '/opt/solr/'
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader replaceClassLoader
INFO: Adding 'file:/opt/solr/lib/ojdbc14.jar' to classloader
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrConfig initLibs
INFO: Adding lib dirs to ClassLoader
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrResourceLoader replaceClassLoader
INFO: Adding 'file:/opt/solr/lib/ojdbc14.jar' to classloader
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrConfig <init>
INFO: Loaded SolrConfig: solrconfig.xml
Dec 16, 2009 10:53:29 AM org.apache.solr.core.SolrCore <init>
INFO: Opening new SolrCore at /opt/solr, dataDir=/opt/indexes/solr/
Dec 16, 2009 10:53:29 AM org.apache.solr.schema.IndexeSchema readSchema
INFO: Reading Solr Schema
Dec 16, 2009 10:53:29 AM org.apache.solr.schema.IndexeSchema readSchema
INFO: Schema name=solr
Dec 16, 2009 10:53:29 AM org.apache.solr.util.plugin.AbstractPluginLoader load
.......
But then I get this when I engage the dataimport.jsp, and click on full import:
Dec 16, 2009 10:54:07 AM org.apache.solr.handler.dataimport.DataImportHandler processConfiguration
INFO: Processing configuration from solrconfig.xml: {config=dataconfig.xml}
Dec 16, 2009 10:54:07 AM org.apache.solr.handler.dataimport.DataImportHandler loadDataConfig
INFO: Data Configuratio loaded successfully
Dec 16, 2009 10:54:07 AM org.apache.solr.core.SolrCore execute
INFO: [] webapps=/solr path=/select params={clean=false&commit=true&command=reload-config&qt=/dataimport} status=0 Qtime=16
Dec 16, 2009 10:54:07 AM org.apache.solr.core.SolrCore execute
INFO: [] webapps=/solr path=/select params={clean=false&commit=true&command=full-import&qt=/dataimport} status=0 Qtime=2
Dec 16, 2009 10:54:07 AM org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Dec 16, 2009 10:54:07 AM org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
WARNING: Unable to read: dataimport.properties
Dec 16, 2009 10:54:07 AM org.apache.solr.handler.dataimport.DataImporter doFullImport
SERVERE: Full Import failed
prg.apache.solr.handler.dataimport.DataImportHandlerException: No dataSource :null available for entity: mydocs Processing Document #1
at org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:279)
at org.apache.solr.handler.dataimport.ContextImp.getDataSource(ContextImpl.java:93)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:52)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(SqlEntityProcessorWrapper.java:71)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:331)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389)
at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Dec 16, 2009 10:54:07 AM org.apache.solr.update.DirectupdateHandler2 rollback
.....
My dataconfig.xml is
<dataConfig>
<dataSource name="JdbcDataSource" driver="oracle.jdbc.driver.OracleDriver"
url="jdbc.oracle:thin:@xxx-xxx-x:1521:<sid>"
username="xxxx_xxxx_xxx"
password="xxxxx_xx" />
<document>
<entity name="mydocs"
query=select id, obj_clob from myschema.objects_view where obj_clob is not null"
transformer=ClobTransformer,LogTransformer"
logTemplate="The nameis ${mydocs.name}" logLevel="debug">
<field colunm="id" />
<field column="obj_clob" clob="true" />
</entity>
</document>
</dataConfig>
I'm at a loss as to what I should be checking. Any help would be very much appreciated.
-----Original Message-----
From: Ken Lane (kenlane) [mailto:kenlane@cisco.com]
Sent: Wednesday, December 16, 2009 7:48 AM
To: solr-user@lucene.apache.org
Cc: Turner, Robbin J
Subject: RE: Request Assistance with DIH
Regarding getting the working directory set,
I got the cwd set to mysolr home by using a tomcat startup system property. In windows, there is a UI for Tomcat where you can set these java options:
-Dsolr.solr.home=C:\Appsolar
-Dsolr.data.dir=C:\Appsolar\data
In that UI, there is also a "Working Path" you can set up under the "Startup" tab, which I set to C:\Appsolar
When I start Tomcat, in my Catalina log, I get this:
INFO: No /solr/home in JNDI
Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
locateSolrHome
INFO: using system property solr.solr.home: C:\Appsolar Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader <init>
INFO: Solr home set to 'C:\Appsolar\'
Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/jetty-6.1.3.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/jetty-util-6.1.3.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/jsp-2.1/' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/ojdbc14.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/servlet-api-2.5-6.1.3.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
locateSolrHome
INFO: No /solr/home in JNDI
Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
locateSolrHome
INFO: using system property solr.solr.home: C:\Appsolar Dec 16, 2009 7:34:22 AM org.apache.solr.core.CoreContainer$Initializer
initialize
INFO: looking for solr.xml: C:\Appsolar\solr.xml Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader <init>
INFO: Solr home set to 'C:\Appsolar\'
Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/jetty-6.1.3.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/jetty-util-6.1.3.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/jsp-2.1/' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/ojdbc14.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrResourceLoader
replaceClassLoader
INFO: Adding 'file:/C:/Appsolar/lib/servlet-api-2.5-6.1.3.jar' to classloader Dec 16, 2009 7:34:22 AM org.apache.solr.core.SolrConfig initLibs
INFO: Adding specified lib dirs to ClassLoader Dec 16, 2009 7:34:23 AM org.apache.solr.core.SolrConfig <init>
INFO: Loaded SolrConfig: solrconfig.xml
Dec 16, 2009 7:34:23 AM org.apache.solr.core.SolrCore <init>
INFO: Opening new SolrCore at C:\Appsolar\, dataDir=C:\Appsolar\data\
-----Original Message-----
From: Turner, Robbin J [mailto:Robbin.J.Turner@boeing.com]
Sent: Tuesday, December 15, 2009 4:29 PM
To: solr-user@lucene.apache.org
Subject: RE: Request Assistance with DIH
Thanks for the reply, just what I was looking for in answer. I am running under Tomcat 6 on Solaris 10, the person that replied before you looks like they running under jetty. I have configured jndi context. I stop and start tomcat using the Solaris SMF, equivalent to services in linux. But my cwd is point to root, I have solr home specified in Catalina/localhost/solr.xml. Is there anything else that I can do to force cwd to point to solr/home?
Thanks again
Robbin
-----Original Message-----
From: Ken Lane (kenlane) [mailto:kenlane@cisco.com]
Sent: Monday, December 14, 2009 11:04 AM
To: solr-user@lucene.apache.org
Subject: RE: Request Assistance with DIH
Hi Robbin,
I just went through this myself (I am a newbie).
The key things to look at are:
1. Your data_config.xml. I created a table called 'foo' and an ora_data_config.xml file with a simple example to get it working that looks like this:
<dataConfig>
<dataSource type="JdbcDataSource"
driver="oracle.jdbc.driver.OracleDriver"
url="jdbc:oracle:thin:@servername.cisco.com:1521:sidname"
user="scott"
password="tiger"/>
<document>
<entity name="user"
query="SELECT id, username from foo">
</entity>
</document>
</dataConfig>
Some gotcha's:
If your Oracle DB is configured with Service_name rather than SID (ie.
You may be running failover, RAC, etc), the url parameter of jdbc connection can read like this:
url="jdbc:oracle:thin:@(DESCRIPTION = (LOAD_BALANCE = on) (FAILOVER = on) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = <your_server_name_here>.cisco.com)(PORT = 1528))) (CONNECT_DATA = (SERVICE_NAME = <your_service_name>.<your domain>.COM)))"
2. In your solr_config.xml file, add something like this to reference the above listed file:
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">ora-data-config.xml</str>
</lst>
</requestHandler>
3. I have Solr1.4 running under Tomcat 6. It looks like you are trying the jetty example, but pay mind to getting the "cwd" pointing to your solr home by setting your JNDI path as described in the dataimporthandler wiki.
4. When it blows up, as it did numerous times for me until I got it right, check the logs. As I am running under Tomcat, I was able to check <tomcat_home>\logs\catalina.2009-12-14.log to view DIH errors both upon restart of Tomcat and after running the DIH.
5. There are some tools to check your JDBC connection you might try before pulling too much of your hair out. Try here:
http://otn.oracle.com/sample_code/tech/java/sqlj_jdbc/content.html
Good Luck!
Ken
-----Original Message-----
From: Turner, Robbin J [mailto:Robbin.J.Turner@boeing.com]
Sent: Monday, December 14, 2009 10:27 AM
To: solr-user@lucene.apache.org
Subject: RE: Request Assistance with DIH
How does this help answer my question? I am trying to use the DATAImportHandler Development console. The url you suggest assumes I had it working already.
Looking at my logs and the response to the Development console, it does not appear that the connection to Oracle is being made.
So if someone could offer some configuration/connection setup directions I would very much appreciate it.
Thanks
Robbin
-----Original Message-----
From: Joel Nylund [mailto:jnylund@yahoo.com]
Sent: Friday, December 11, 2009 8:26 PM
To: solr-user@lucene.apache.org
Subject: Re: Request Assistance with DIH
add ?command=full-import to your url
http://localhost:8983/solr/dataimport?command=full-import
thanks
Joel
On Dec 11, 2009, at 7:45 PM, Robbin wrote:
> I've been trying to use the DIH with oracle and would love it if
> someone could give me some pointers. I put the ojdbc14.jar in both
> the Tomcat lib and <solr home>/lib. I created a dataimport.xml and
> enabled it in the solrconfig.xml. I go to the http://<solr server>/
> solr/admin/dataimport.jsp. This all seems to be fine, but I get the
> default page response and doesn't look like the connection to the
> oracle server is even attempted.
>
> I'm using the Solr 1.4 release on Nov 10.
> Do I need an oracle client on the server? I thought having the ojdbc
> jar should be sufficient. Any help or configuration examples for
> setting this up would be much appreciated.
>
> Thanks
> Robbin
Re: Request Assistance with DIH, development console
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Dec 17, 2009 at 1:47 AM, Turner, Robbin J <
Robbin.J.Turner@boeing.com> wrote:
>
> My dataconfig.xml is
>
> <dataConfig>
> <dataSource name="JdbcDataSource"
> driver="oracle.jdbc.driver.OracleDriver"
> url="jdbc.oracle:thin:@xxx-xxx-x:1521:<sid>"
> username="xxxx_xxxx_xxx"
> password="xxxxx_xx" />
> <document>
> <entity name="mydocs"
> query=select id, obj_clob from
> myschema.objects_view where obj_clob is not null"
> transformer=ClobTransformer,LogTransformer"
> logTemplate="The nameis ${mydocs.name}"
> logLevel="debug">
>
> <field colunm="id" />
> <field column="obj_clob" clob="true" />
> </entity>
> </document>
> </dataConfig>
>
> I'm at a loss as to what I should be checking. Any help would be very much
> appreciated.
>
>
If you specify a name for your data source, then that name must also be
specified in your entity. I think you meant to use type="JdbcDataSource"
instead of name="JdbcDataSource". Do that and it will work.
--
Regards,
Shalin Shekhar Mangar.