You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Vince McMahon <si...@gmail.com> on 2023/11/27 07:32:36 UTC
Need help with solr 8.11 DataImport
Hi,
I am new to Solr dataimport and am trying to do a full-import from
Postgres. Both Solr 8.11.2 and Postgres 12 are installed in Win10.
Postgres is connecting using id solrpull and login is fine via Pgadmin.
Solr web and core are running.
I don't see any document imported and I don't find any error. How to fix
this?
Raw Status-Output
{
"responseHeader": {
"status": 0,
"QTime": 0
},
"initArgs": [
"defaults",
[
"config",
"db_configs.xml"
]
],
"command": "status",
"status": "idle",
"importResponse": "",
"statusMessages": {
"Time Elapsed": "0:3:21.15",
"Total Requests made to DataSource": "1",
"Total Rows Fetched": "0",
"Total Documents Processed": "0",
"Total Documents Skipped": "0",
"Full Dump Started": "2023-11-27 06:38:15",
"Full Import failed": "2023-11-27 06:38:35"
}
}
Configuration
<dataConfig>
<dataSource type="JdbcDataSource"
driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost/world"
user="solrpull"
password="NotTelling"/>
<document>
<entity name="city"
query="select id,name from city">
<field column="id" name="id"/>
<field column="name" name="name_str"/>
</entity>
</document></dataConfig>
log4j2
11/27/2023, 2:13:00 AM WARN false CoreContainer Not all security plugins
configured! authentication=disabled authorization=disabled. Solr is only as
secure as you make it. Consider configuring authentication/authorization
before exposing Solr to users internal or external. See
https://s.apache.org/solrsecurity for more info
11/27/2023, 2:13:00 AM WARN false x:cookbookCSV SolrConfig solrconfig.xml:
<jmx> is no longer supported,​ use solr.xml:/metrics/reporter section
instead
11/27/2023, 2:13:00 AM WARN false x:cookbookCSV WordDelimiterFilterFactory Solr
loaded a deprecated plugin/analysis class
[solr.WordDelimiterFilterFactory]. Please consult documentation how to
replace it accordingly.
11/27/2023, 2:13:00 AM WARN false x:d
11/27/2023, 2:13:00 AM WARN false x:mail SolrConfig solrconfig.xml: <jmx>
is no longer supported,​ use solr.xml:/metrics/reporter section
instead
11/27/2023, 2:13:00 AM WARN false x:db LatLonType Solr loaded a deprecated
plugin/analysis class [solr.LatLonType]. Please consult documentation how
to replace it accordingly.
11/27/2023, 2:13:01 AM WARN false x:atom DataImportHandler Data Import
Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
11/27/2023, 2:13:01 AM WARN false x:mail DataImportHandler Data Import
Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
11/27/2023, 2:13:01 AM WARN false x:db DataImportHandler Data Import
Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
11/27/2023, 2:13:01 AM WARN false x:solr SolrConfig solrconfig.xml: <jmx>
is no longer supported,​ use solr.xml:/metrics/reporter section
instead
11/27/2023, 2:13:01 AM WARN false x:p SolrConfig solrconfig.xml: <jmx> is
no longer supported,​ use solr.xml:/metrics/reporter section instead
11/27/2023, 2:13:01 AM WARN false x:solr DataImportHandler Data Import
Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
11/27/2023, 2:13:02 AM WARN false x:tika DataImportHandler Data Import
Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
11/27/2023, 2:13:02 AM WARN false x:sample_core SolrConfig solrconfig.xml:
<jmx> is no longer supported,​ use solr.xml:/metrics/reporter section
instead
11/27/2023, 2:13:02 AM WARN false x:f18maf SolrConfig solrconfig.xml: <jmx>
is no longer supported,​ use solr.xml:/metrics/reporter section
instead
11/27/2023, 2:13:02 AM WARN false x:demo SolrConfig solrconfig.xml: <jmx>
is no longer supported,​ use solr.xml:/metrics/reporter section
instead
11/27/2023, 2:13:02 AM WARN false x:p DataImportHandler Data Imp My
database has data
select id,name from city
"id" "name"
1 "Seattle"
2 "Japan"
Successfully run. Total query runtime: 37 msec.
2 rows affected.
Re: Need help with solr 8.11 DataImport
Posted by Mikhail Khludnev <mk...@apache.org>.
Good Job, Vince!
On Mon, Nov 27, 2023 at 7:45 PM Vince McMahon <si...@gmail.com>
wrote:
> >
> > I've figured it out. Thank you, Mikhail
> >>
> >>
>
--
Sincerely yours
Mikhail Khludnev
Re: Need help with solr 8.11 DataImport
Posted by Vince McMahon <si...@gmail.com>.
>
> I've figured it out. Thank you, Mikhail
>>
>>
Re: Need help with solr 8.11 DataImport
Posted by Vince McMahon <si...@gmail.com>.
I start solr by
C:\solr-8.11.2\bin\*solr start*
When I extract solr the dist, lib folders they are in
C:\solr-8.11.2\dist
C:\solr-8.11.2\server\*lib*
C:\solr-8.11.2\server\solr\configsets\*lib*
Where should be the jar files be located and the correct syntax in
C:\solr-8.11.2\*myNewCore*\conf\*solrconfig.xml*?
Are the following correct for Windows?
<lib path="..\dist\solr-dataimporthandler-8.11.2.jar" />
<lib path="..\dist\postgresql-42.2.6.jar" />
The jar files are postgresql-42.2.6.jar
solr-dataimporthandler-8.11.2.jar
>>
>>
Re: Need help with solr 8.11 DataImport
Posted by Vince McMahon <si...@gmail.com>.
Hi,
I think it points to the right schema. In the conf/managed-schema file I
added these lines
<field name="id" type="string" docValues="false" indexed="true" stored=
"true"/>
<field name="name" type="text_general" docValues="false" indexed="true"
stored="true"/>
The SQL INSERT are committed. select * from city returns all rows after
rebooting the computer.
There is a debug check box and
The Raw Debug Response is:
{
"responseHeader": {
"status": 0,
"QTime": 51
},
"initArgs": [
"defaults",
[
"config",
"db_configs.xml"
]
],
"command": "full-import",
"mode": "debug",
"documents": [],
"verbose-output": [],
"status": "idle",
"importResponse": "",
"statusMessages": {
"Time Elapsed": "0:0:0.44",
"Total Requests made to DataSource": "1",
"Total Rows Fetched": "0",
"Total Documents Processed": "0",
"Total Documents Skipped": "0",
"Full Dump Started": "2023-11-27 12:58:12",
"Full Import failed": "2023-11-27 12:58:12"
}
}
On Mon, Nov 27, 2023 at 5:15 AM Mikhail Khludnev <mk...@apache.org> wrote:
> Hello Vince.
>
> I think these numbers speak for themselves:
>
> "Total Requests made to DataSource": "1",
> > "Total Rows Fetched": "0",
> > "Total Documents Processed": "0",
> > "Total Documents Skipped": "0",
>
> Does it point to the right schema? Have you committed SQL INSERT
> transaction?
> I remember there's a debug parameter in DIH, you may use it.
>
> On Mon, Nov 27, 2023 at 10:33 AM Vince McMahon <
> sippingonesandzeros@gmail.com> wrote:
>
> > Hi,
> >
> > I am new to Solr dataimport and am trying to do a full-import from
> > Postgres. Both Solr 8.11.2 and Postgres 12 are installed in Win10.
> >
> > Postgres is connecting using id solrpull and login is fine via Pgadmin.
> > Solr web and core are running.
> >
> > I don't see any document imported and I don't find any error. How to fix
> > this?
> >
> > Raw Status-Output
> >
> > {
> > "responseHeader": {
> > "status": 0,
> > "QTime": 0
> > },
> > "initArgs": [
> > "defaults",
> > [
> > "config",
> > "db_configs.xml"
> > ]
> > ],
> > "command": "status",
> > "status": "idle",
> > "importResponse": "",
> > "statusMessages": {
> > "Time Elapsed": "0:3:21.15",
> > "Total Requests made to DataSource": "1",
> > "Total Rows Fetched": "0",
> > "Total Documents Processed": "0",
> > "Total Documents Skipped": "0",
> > "Full Dump Started": "2023-11-27 06:38:15",
> > "Full Import failed": "2023-11-27 06:38:35"
> > }
> > }
> >
> > Configuration
> >
> > <dataConfig>
> > <dataSource type="JdbcDataSource"
> > driver="org.postgresql.Driver"
> > url="jdbc:postgresql://localhost/world"
> > user="solrpull"
> > password="NotTelling"/>
> > <document>
> > <entity name="city"
> > query="select id,name from city">
> > <field column="id" name="id"/>
> > <field column="name" name="name_str"/>
> > </entity>
> > </document></dataConfig>
> >
> > log4j2
> > 11/27/2023, 2:13:00 AM WARN false CoreContainer Not all security plugins
> > configured! authentication=disabled authorization=disabled. Solr is only
> as
> > secure as you make it. Consider configuring authentication/authorization
> > before exposing Solr to users internal or external. See
> > https://s.apache.org/solrsecurity for more info
> > 11/27/2023, 2:13:00 AM WARN false x:cookbookCSV SolrConfig
> solrconfig.xml:
> > <jmx> is no longer supported,​ use solr.xml:/metrics/reporter
> section
> > instead
> > 11/27/2023, 2:13:00 AM WARN false x:cookbookCSV
> WordDelimiterFilterFactory
> > Solr
> > loaded a deprecated plugin/analysis class
> > [solr.WordDelimiterFilterFactory]. Please consult documentation how to
> > replace it accordingly.
> > 11/27/2023, 2:13:00 AM WARN false x:d
> > 11/27/2023, 2:13:00 AM WARN false x:mail SolrConfig solrconfig.xml: <jmx>
> > is no longer supported,​ use solr.xml:/metrics/reporter section
> > instead
> > 11/27/2023, 2:13:00 AM WARN false x:db LatLonType Solr loaded a
> deprecated
> > plugin/analysis class [solr.LatLonType]. Please consult documentation how
> > to replace it accordingly.
> > 11/27/2023, 2:13:01 AM WARN false x:atom DataImportHandler Data Import
> > Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> > 11/27/2023, 2:13:01 AM WARN false x:mail DataImportHandler Data Import
> > Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> > 11/27/2023, 2:13:01 AM WARN false x:db DataImportHandler Data Import
> > Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> > 11/27/2023, 2:13:01 AM WARN false x:solr SolrConfig solrconfig.xml: <jmx>
> > is no longer supported,​ use solr.xml:/metrics/reporter section
> > instead
> > 11/27/2023, 2:13:01 AM WARN false x:p SolrConfig solrconfig.xml: <jmx> is
> > no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> > 11/27/2023, 2:13:01 AM WARN false x:solr DataImportHandler Data Import
> > Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> > 11/27/2023, 2:13:02 AM WARN false x:tika DataImportHandler Data Import
> > Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> > 11/27/2023, 2:13:02 AM WARN false x:sample_core SolrConfig
> solrconfig.xml:
> > <jmx> is no longer supported,​ use solr.xml:/metrics/reporter
> section
> > instead
> > 11/27/2023, 2:13:02 AM WARN false x:f18maf SolrConfig solrconfig.xml:
> <jmx>
> > is no longer supported,​ use solr.xml:/metrics/reporter section
> > instead
> > 11/27/2023, 2:13:02 AM WARN false x:demo SolrConfig solrconfig.xml: <jmx>
> > is no longer supported,​ use solr.xml:/metrics/reporter section
> > instead
> > 11/27/2023, 2:13:02 AM WARN false x:p DataImportHandler Data Imp My
> > database has data
> >
> >
> > select id,name from city
> > "id" "name"
> > 1 "Seattle"
> > 2 "Japan"
> > Successfully run. Total query runtime: 37 msec.
> > 2 rows affected.
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: Need help with solr 8.11 DataImport
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello Vince.
I think these numbers speak for themselves:
"Total Requests made to DataSource": "1",
> "Total Rows Fetched": "0",
> "Total Documents Processed": "0",
> "Total Documents Skipped": "0",
Does it point to the right schema? Have you committed SQL INSERT
transaction?
I remember there's a debug parameter in DIH, you may use it.
On Mon, Nov 27, 2023 at 10:33 AM Vince McMahon <
sippingonesandzeros@gmail.com> wrote:
> Hi,
>
> I am new to Solr dataimport and am trying to do a full-import from
> Postgres. Both Solr 8.11.2 and Postgres 12 are installed in Win10.
>
> Postgres is connecting using id solrpull and login is fine via Pgadmin.
> Solr web and core are running.
>
> I don't see any document imported and I don't find any error. How to fix
> this?
>
> Raw Status-Output
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 0
> },
> "initArgs": [
> "defaults",
> [
> "config",
> "db_configs.xml"
> ]
> ],
> "command": "status",
> "status": "idle",
> "importResponse": "",
> "statusMessages": {
> "Time Elapsed": "0:3:21.15",
> "Total Requests made to DataSource": "1",
> "Total Rows Fetched": "0",
> "Total Documents Processed": "0",
> "Total Documents Skipped": "0",
> "Full Dump Started": "2023-11-27 06:38:15",
> "Full Import failed": "2023-11-27 06:38:35"
> }
> }
>
> Configuration
>
> <dataConfig>
> <dataSource type="JdbcDataSource"
> driver="org.postgresql.Driver"
> url="jdbc:postgresql://localhost/world"
> user="solrpull"
> password="NotTelling"/>
> <document>
> <entity name="city"
> query="select id,name from city">
> <field column="id" name="id"/>
> <field column="name" name="name_str"/>
> </entity>
> </document></dataConfig>
>
> log4j2
> 11/27/2023, 2:13:00 AM WARN false CoreContainer Not all security plugins
> configured! authentication=disabled authorization=disabled. Solr is only as
> secure as you make it. Consider configuring authentication/authorization
> before exposing Solr to users internal or external. See
> https://s.apache.org/solrsecurity for more info
> 11/27/2023, 2:13:00 AM WARN false x:cookbookCSV SolrConfig solrconfig.xml:
> <jmx> is no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> 11/27/2023, 2:13:00 AM WARN false x:cookbookCSV WordDelimiterFilterFactory
> Solr
> loaded a deprecated plugin/analysis class
> [solr.WordDelimiterFilterFactory]. Please consult documentation how to
> replace it accordingly.
> 11/27/2023, 2:13:00 AM WARN false x:d
> 11/27/2023, 2:13:00 AM WARN false x:mail SolrConfig solrconfig.xml: <jmx>
> is no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> 11/27/2023, 2:13:00 AM WARN false x:db LatLonType Solr loaded a deprecated
> plugin/analysis class [solr.LatLonType]. Please consult documentation how
> to replace it accordingly.
> 11/27/2023, 2:13:01 AM WARN false x:atom DataImportHandler Data Import
> Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> 11/27/2023, 2:13:01 AM WARN false x:mail DataImportHandler Data Import
> Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> 11/27/2023, 2:13:01 AM WARN false x:db DataImportHandler Data Import
> Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> 11/27/2023, 2:13:01 AM WARN false x:solr SolrConfig solrconfig.xml: <jmx>
> is no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> 11/27/2023, 2:13:01 AM WARN false x:p SolrConfig solrconfig.xml: <jmx> is
> no longer supported,​ use solr.xml:/metrics/reporter section instead
> 11/27/2023, 2:13:01 AM WARN false x:solr DataImportHandler Data Import
> Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> 11/27/2023, 2:13:02 AM WARN false x:tika DataImportHandler Data Import
> Handler is deprecated as of Solr 8.6. See SOLR-14066 for more details.
> 11/27/2023, 2:13:02 AM WARN false x:sample_core SolrConfig solrconfig.xml:
> <jmx> is no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> 11/27/2023, 2:13:02 AM WARN false x:f18maf SolrConfig solrconfig.xml: <jmx>
> is no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> 11/27/2023, 2:13:02 AM WARN false x:demo SolrConfig solrconfig.xml: <jmx>
> is no longer supported,​ use solr.xml:/metrics/reporter section
> instead
> 11/27/2023, 2:13:02 AM WARN false x:p DataImportHandler Data Imp My
> database has data
>
>
> select id,name from city
> "id" "name"
> 1 "Seattle"
> 2 "Japan"
> Successfully run. Total query runtime: 37 msec.
> 2 rows affected.
>
--
Sincerely yours
Mikhail Khludnev