Remote streaming - posting a URL which is password protected

[12rad <pr...@gmail.com>]

I want to post index a http document that is password protected. 
It has a username name login. 
I tried doing this 

curl -u username:password
"http://localhost:8983/solr/update/extract?literal.id=doc900&commit=true" -F
stream.url=http://somewebsite.com/docs/DOC2609

but it just indexes the login page only.

--
View this message in context: http://lucene.472066.n3.nabble.com/Remote-streaming-posting-a-URL-which-is-password-protected-tp3985221.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Remote streaming - posting a URL which is password protected

[Jan Høydahl <ja...@cominvent.com>]

Hi,

Using curl -u will only attempt to log in to Jetty/Solr, which is not password protected I assume. What you really would like is for the HTTP call which Solr does based on stream.url to attempt a login. Such functionality is not implemented as far as I know. You may try the syntax stream.url=http://username:password@somewebsite.com/docs/DOC2609 but I have not tested it. Why can't you download the file locally first? If you're looking for a production grade HTTP crawler you could look at ManifoldCF.

--
Jan Høydahl, search solution architect
Cominvent AS - www.facebook.com/Cominvent
Solr Training - www.solrtraining.com

On 21. mai 2012, at 22:44, 12rad wrote:

> I want to post index a http document that is password protected. 
> It has a username name login. 
> I tried doing this 
> 
> curl -u username:password
> "http://localhost:8983/solr/update/extract?literal.id=doc900&commit=true" -F
> stream.url=http://somewebsite.com/docs/DOC2609
> 
> but it just indexes the login page only.
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Remote-streaming-posting-a-URL-which-is-password-protected-tp3985221.html
> Sent from the Solr - User mailing list archive at Nabble.com.