You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Christian Dannemann <ch...@merus.eu> on 2019/11/26 21:17:37 UTC
Some newby questions ...
Hi Everyone,
I've managed to successfully install solr on my server, and it's running
and I have created a core and indexed some 1000 pdf files... Searches work,
and so far, so good.
There are two issues that I can't resolve even though I have tried for a
good 8 hours, so I feel I can hopefully reach out having tried myself
before asking stupid questions.
Environment:
- Ubuntu OpenJDK 64-Bit Server VM 11.0.4
11.0.4+11-post-Ubuntu-1ubuntu218.04.3
- Processors8
- Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
-Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
-Dsolr.data.home=
-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
-Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
-Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
-XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
-XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
10539 /var/solr/logs
-Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
-Xms512m-Xmx512m-Xss256k
and my core: CWD:
/opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
/var/solr/data/meeting_minutes/dataIndex:
/var/solr/data/meeting_minutes/data/indexImpl:
org.apache.solr.core.NRTCachingDirectoryFactory
Issue 1: I want to secure my server with basic authentication (that's why
I'm running on port 10539 at the moment, but that's not security ...
I've put a file security.json in
/opt/solr/server/solr/configsets/_default/conf, but that doesn't do
anything. No request for a password when I access the web interface or do
searches
Any help on that would be highly appreciated.
Issue 2: I would like to index a lot of emails that reside on a local imap
server (dovecot). I have managed to configure that, and my
configuration shows up in the "dataimport" menu on my core
(meeting_minutes) in the web interface: <dataConfig> <document> <entity
processor="MailEntityProcessor" user="christian" password="******" host=
"localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00" batchSize=
"200" folders="Secretary" processAttachement="false" name="mail_entity"/> </
document> </dataConfig>
(the password does of course appear in cleartext in my config file ....)
I can execute the command from the web interface, and get the message
Last Update: 21:08:18
Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
My dovecot server shows nothing in the logs, and I can change the password
and don't get any error message - solr always claims it executed, and that
there were 0 Fetched etc.
I manually have logged into the dovecot server with imap, and the folder
"Secretary" does indeed show up when I list the folders. It appears solr
doesn't even try to connect to my dovecot server.
I can't use the debug option, as apparently, I have to
enable.dih.dataConfigParam has to be set to true, but I don't have the
slightest clue where and how to set that. (It took me quite a while to
figure out that changing the default port had to be done in /etc/default/
solr.in.sh, rather than in solrconfig.xml...)
The performance is great, and I love the stuff that works, but I find it
very hard to get solr going, configure it, and get my imap dataConfig going.
Any help is greatly appreciated!
Best regards,
Christian
Re: Some newby questions ...
Posted by Christian Dannemann <ch...@merus.eu>.
... sorry, forgot to post solr.log:
2019-11-26 21:24:11.242 INFO (qtp1884155890-24) [ x:meeting_minutes]
o.a.s.h.d.DataImporter Loading DIH Configuration: mail-data-config.xml
2019-11-26 21:24:11.245 INFO (qtp1884155890-24) [ x:meeting_minutes]
o.a.s.h.d.c.DIHConfiguration id is a required field in SolrSchema . But not
found in DataConfig
2019-11-26 21:24:11.245 INFO (qtp1884155890-24) [ x:meeting_minutes]
o.a.s.h.d.DataImporter Data Configuration loaded successfully
2019-11-26 21:24:11.246 INFO (qtp1884155890-24) [ x:meeting_minutes]
o.a.s.c.S.Request [meeting_minutes] webapp=/solr path=/dataimport
params={core=meeting_minutes&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1574803444168&verbose=false}
status=0 QTime=4
2019-11-26 21:24:11.246 INFO (Thread-18) [ ] o.a.s.h.d.DataImporter
Starting Full Import
2019-11-26 21:24:11.252 WARN (Thread-18) [ ]
o.a.s.h.d.SimplePropertiesWriter Unable to read: dataimport.properties
2019-11-26 21:24:11.257 INFO (Thread-18) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [meeting_minutes] webapp=/solr
path=/dataimport
params={core=meeting_minutes&indent=on&commit=true&name=dataimport&clean=false&wt=json&command=full-import&_=1574803444168&verbose=false}
status=0 QTime=4{} 0 15
2019-11-26 21:24:11.294 INFO (qtp1884155890-42) [ x:meeting_minutes]
o.a.s.c.S.Request [meeting_minutes] webapp=/solr path=/dataimport
params={indent=on&wt=json&command=status&_=1574803444168} status=0 QTime=0
Best regards,
Christian
On Tue, 26 Nov 2019 at 21:17, Christian Dannemann <ch...@merus.eu>
wrote:
> Hi Everyone,
>
> I've managed to successfully install solr on my server, and it's running
> and I have created a core and indexed some 1000 pdf files... Searches work,
> and so far, so good.
>
> There are two issues that I can't resolve even though I have tried for a
> good 8 hours, so I feel I can hopefully reach out having tried myself
> before asking stupid questions.
>
> Environment:
>
> - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
> 11.0.4+11-post-Ubuntu-1ubuntu218.04.3
> - Processors8
> - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
> -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
> -Dsolr.data.home=
> -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
> -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
> -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
> -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
> -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
> -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> 10539 /var/solr/logs
> -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
> -Xms512m-Xmx512m-Xss256k
>
>
> and my core: CWD:
> /opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
> /var/solr/data/meeting_minutes/dataIndex:
> /var/solr/data/meeting_minutes/data/indexImpl:
> org.apache.solr.core.NRTCachingDirectoryFactory
> Issue 1: I want to secure my server with basic authentication (that's why
> I'm running on port 10539 at the moment, but that's not security ...
>
> I've put a file security.json in
> /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> anything. No request for a password when I access the web interface or do
> searches
> Any help on that would be highly appreciated.
> Issue 2: I would like to index a lot of emails that reside on a local imap
> server (dovecot). I have managed to configure that, and my
> configuration shows up in the "dataimport" menu on my core
> (meeting_minutes) in the web interface: <dataConfig> <document> <entity
> processor="MailEntityProcessor" user="christian" password="******" host=
> "localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00"
> batchSize="200" folders="Secretary" processAttachement="false" name=
> "mail_entity"/> </document> </dataConfig>
>
> (the password does of course appear in cleartext in my config file ....)
> I can execute the command from the web interface, and get the message
>
> Last Update: 21:08:18
> Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
>
> My dovecot server shows nothing in the logs, and I can change the password
> and don't get any error message - solr always claims it executed, and that
> there were 0 Fetched etc.
> I manually have logged into the dovecot server with imap, and the folder
> "Secretary" does indeed show up when I list the folders. It appears solr
> doesn't even try to connect to my dovecot server.
>
> I can't use the debug option, as apparently, I have to
> enable.dih.dataConfigParam has to be set to true, but I don't have the
> slightest clue where and how to set that. (It took me quite a while to
> figure out that changing the default port had to be done in /etc/default/
> solr.in.sh, rather than in solrconfig.xml...)
>
> The performance is great, and I love the stuff that works, but I find it
> very hard to get solr going, configure it, and get my imap dataConfig going.
>
> Any help is greatly appreciated!
>
> Best regards,
>
> Christian
>
>
>
>
>
>
>
>
--
Christian Dannemann
Managing Director
Merus Software Ltd
http://merus.eu
DDI:+44 1453 708610
Re: Some newby questions ...
Posted by Christian Dannemann <ch...@merus.eu>.
Hi Jan,
I'm afraid I don't run in cloud mode, and I get
Failed to parse command-line arguments due to: Missing argument for option:
solrIncludeFile
usage: org.apache.solr.util.SolrCLI
Best regards,
Christian
On Tue, 26 Nov 2019 at 21:30, Jan Høydahl <ja...@cominvent.com> wrote:
> Try
>
> bin/solr auth enable -blockUnknown true -credentials user:password -z
> local host:11539
>
> That should enable basic auth for you given you run in cloud mode (-c
> switch)
>
> Jan Høydahl
>
> > 26. nov. 2019 kl. 22:18 skrev Christian Dannemann <ch...@merus.eu>:
> >
> > Hi Everyone,
> >
> > I've managed to successfully install solr on my server, and it's running
> > and I have created a core and indexed some 1000 pdf files... Searches
> work,
> > and so far, so good.
> >
> > There are two issues that I can't resolve even though I have tried for a
> > good 8 hours, so I feel I can hopefully reach out having tried myself
> > before asking stupid questions.
> >
> > Environment:
> >
> > - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
> > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3
> > - Processors8
> > - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
> > -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
> > -Dsolr.data.home=
> > -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
> > -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
> > -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
> > -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
> > -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
> >
> -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> > 10539 /var/solr/logs
> >
> -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
> > -Xms512m-Xmx512m-Xss256k
> >
> >
> > and my core: CWD:
> > /opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
> > /var/solr/data/meeting_minutes/dataIndex:
> > /var/solr/data/meeting_minutes/data/indexImpl:
> > org.apache.solr.core.NRTCachingDirectoryFactory
> > Issue 1: I want to secure my server with basic authentication (that's why
> > I'm running on port 10539 at the moment, but that's not security ...
> >
> > I've put a file security.json in
> > /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> > anything. No request for a password when I access the web interface or do
> > searches
> > Any help on that would be highly appreciated.
> > Issue 2: I would like to index a lot of emails that reside on a local
> imap
> > server (dovecot). I have managed to configure that, and my
> > configuration shows up in the "dataimport" menu on my core
> > (meeting_minutes) in the web interface: <dataConfig> <document> <entity
> > processor="MailEntityProcessor" user="christian" password="******" host=
> > "localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00"
> batchSize=
> > "200" folders="Secretary" processAttachement="false"
> name="mail_entity"/> </
> > document> </dataConfig>
> >
> > (the password does of course appear in cleartext in my config file ....)
> > I can execute the command from the web interface, and get the message
> >
> > Last Update: 21:08:18
> > Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
> >
> > My dovecot server shows nothing in the logs, and I can change the
> password
> > and don't get any error message - solr always claims it executed, and
> that
> > there were 0 Fetched etc.
> > I manually have logged into the dovecot server with imap, and the folder
> > "Secretary" does indeed show up when I list the folders. It appears solr
> > doesn't even try to connect to my dovecot server.
> >
> > I can't use the debug option, as apparently, I have to
> > enable.dih.dataConfigParam has to be set to true, but I don't have the
> > slightest clue where and how to set that. (It took me quite a while to
> > figure out that changing the default port had to be done in /etc/default/
> > solr.in.sh, rather than in solrconfig.xml...)
> >
> > The performance is great, and I love the stuff that works, but I find it
> > very hard to get solr going, configure it, and get my imap dataConfig
> going.
> >
> > Any help is greatly appreciated!
> >
> > Best regards,
> >
> > Christian
>
--
Christian Dannemann
Managing Director
Merus Software Ltd
http://merus.eu
DDI:+44 1453 708610
Re: Some newby questions ...
Posted by Jan Høydahl <ja...@cominvent.com>.
Try
bin/solr auth enable -blockUnknown true -credentials user:password -z local host:11539
That should enable basic auth for you given you run in cloud mode (-c switch)
Jan Høydahl
> 26. nov. 2019 kl. 22:18 skrev Christian Dannemann <ch...@merus.eu>:
>
> Hi Everyone,
>
> I've managed to successfully install solr on my server, and it's running
> and I have created a core and indexed some 1000 pdf files... Searches work,
> and so far, so good.
>
> There are two issues that I can't resolve even though I have tried for a
> good 8 hours, so I feel I can hopefully reach out having tried myself
> before asking stupid questions.
>
> Environment:
>
> - Ubuntu OpenJDK 64-Bit Server VM 11.0.4
> 11.0.4+11-post-Ubuntu-1ubuntu218.04.3
> - Processors8
> - Args-DSTOP.KEY=solrrocks-DSTOP.PORT=9539-Djetty.home=/opt/solr/server
> -Djetty.port=10539-Dlog4j.configurationFile=file:/var/solr/log4j2.xml
> -Dsolr.data.home=
> -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
> -Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=10539
> -Dsolr.log.dir=/var/solr/logs-Dsolr.log.muteconsole
> -Dsolr.solr.home=/var/solr/data-Duser.timezone=UTC-XX:+AlwaysPreTouch
> -XX:+ParallelRefProcEnabled-XX:+PerfDisableSharedMem-XX:+UseG1GC
> -XX:+UseLargePages-XX:MaxGCPauseMillis=250-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh
> 10539 /var/solr/logs
> -Xlog:gc*:file=/var/solr/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
> -Xms512m-Xmx512m-Xss256k
>
>
> and my core: CWD:
> /opt/solr-8.3.0/serverInstance:/var/solr/data/meeting_minutesData:
> /var/solr/data/meeting_minutes/dataIndex:
> /var/solr/data/meeting_minutes/data/indexImpl:
> org.apache.solr.core.NRTCachingDirectoryFactory
> Issue 1: I want to secure my server with basic authentication (that's why
> I'm running on port 10539 at the moment, but that's not security ...
>
> I've put a file security.json in
> /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> anything. No request for a password when I access the web interface or do
> searches
> Any help on that would be highly appreciated.
> Issue 2: I would like to index a lot of emails that reside on a local imap
> server (dovecot). I have managed to configure that, and my
> configuration shows up in the "dataimport" menu on my core
> (meeting_minutes) in the web interface: <dataConfig> <document> <entity
> processor="MailEntityProcessor" user="christian" password="******" host=
> "localhost" protocol="imap" fetchMailsSince="2010-06-30 00:00:00" batchSize=
> "200" folders="Secretary" processAttachement="false" name="mail_entity"/> </
> document> </dataConfig>
>
> (the password does of course appear in cleartext in my config file ....)
> I can execute the command from the web interface, and get the message
>
> Last Update: 21:08:18
> Requests: 0 , Fetched: 0 , Skipped: 0 , Processed: 0
>
> My dovecot server shows nothing in the logs, and I can change the password
> and don't get any error message - solr always claims it executed, and that
> there were 0 Fetched etc.
> I manually have logged into the dovecot server with imap, and the folder
> "Secretary" does indeed show up when I list the folders. It appears solr
> doesn't even try to connect to my dovecot server.
>
> I can't use the debug option, as apparently, I have to
> enable.dih.dataConfigParam has to be set to true, but I don't have the
> slightest clue where and how to set that. (It took me quite a while to
> figure out that changing the default port had to be done in /etc/default/
> solr.in.sh, rather than in solrconfig.xml...)
>
> The performance is great, and I love the stuff that works, but I find it
> very hard to get solr going, configure it, and get my imap dataConfig going.
>
> Any help is greatly appreciated!
>
> Best regards,
>
> Christian
Re: Some newby questions ...
Posted by Christian Dannemann <ch...@merus.eu>.
Hmmm... I tried that as well, but it doesn't pick up the security.json
settings.
I run this instance on a computer that is on the internet, so just changing
the port is asking for trouble.
Looks like nobody knows how to import imap data ....
Best regards,
Christian
On Tue, 26 Nov 2019 at 23:55, Shawn Heisey <ap...@elyograg.org> wrote:
> On 11/26/2019 2:17 PM, Christian Dannemann wrote:
> > Issue 1: I want to secure my server with basic authentication (that's why
> > I'm running on port 10539 at the moment, but that's not security ...
> >
> > I've put a file security.json in
> > /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> > anything.
>
> Your solr home appears to be /var/solr/data ... that is where you need
> to place the security.json file for a setup that is not running SolrCloud.
>
>
> https://lucene.apache.org/solr/guide/8_3/authentication-and-authorization-plugins.html#in-standalone-mode
>
> If you install your Solr server in a network location where it cannot be
> reached by people you cannot trust, there is usually no need for
> security measures like authentication.
>
> > Issue 2: I would like to index a lot of emails that reside on a local
> imap
> > server (dovecot).
>
> I've got no idea how to use the dataimport imap capability.
>
> Thanks,
> Shawn
>
--
Christian Dannemann
Managing Director
Merus Software Ltd
http://merus.eu
DDI:+44 1453 708610
Re: Some newby questions ...
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/26/2019 2:17 PM, Christian Dannemann wrote:
> Issue 1: I want to secure my server with basic authentication (that's why
> I'm running on port 10539 at the moment, but that's not security ...
>
> I've put a file security.json in
> /opt/solr/server/solr/configsets/_default/conf, but that doesn't do
> anything.
Your solr home appears to be /var/solr/data ... that is where you need
to place the security.json file for a setup that is not running SolrCloud.
https://lucene.apache.org/solr/guide/8_3/authentication-and-authorization-plugins.html#in-standalone-mode
If you install your Solr server in a network location where it cannot be
reached by people you cannot trust, there is usually no need for
security measures like authentication.
> Issue 2: I would like to index a lot of emails that reside on a local imap
> server (dovecot).
I've got no idea how to use the dataimport imap capability.
Thanks,
Shawn