You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by msaunier <ms...@citya.com> on 2018/04/19 09:41:55 UTC
UpdateProcessor SolrCloud and ManifoldCF
Hello Karl,
I have a problem to call UpdateProcessor with ManifoldCF commit to
SolrCloud.
URL to use UpdateProcessor with curl :
curl
"http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdatePr
ocessorJar&commit=true" -H 'Content-type: application/json' -d '[{"content":
"test.test", "id":"test_2"}]'
So, in ManifoldCF, I have add in Output Connections, in SolrCloud Paths :
Update handler:
/update/extract?processor=CityaTestUpdateProcessorJar&commit=true
Remove handler: /update?processor=CityaTestUpdateProcessorJar&commit=true
But, ManifoldCF not call update processor when he push documents.
Do you have an idea ?
Thanks you.
RE: UpdateProcessor SolrCloud and ManifoldCF
Posted by msaunier <ms...@citya.com>.
https://issues.apache.org/jira/browse/CONNECTORS-1503
Thanks
Maxence,
De : msaunier [mailto:msaunier@citya.com]
Envoyé : jeudi 19 avril 2018 15:08
À : user@manifoldcf.apache.org
Objet : RE: UpdateProcessor SolrCloud and ManifoldCF
In SolrCloud we can’t modify processor chain, because, SolrCloud used just API to config. So, use an url parameter is the alone option.
https://lucene.apache.org/solr/guide/6_6/config-api.html#ConfigAPI-Whatabout_updateRequestProcessorChain_
I create a tiket on Jira and I respond with it’s url.
Thanks,
Maxence
De : Karl Wright [mailto:daddywri@gmail.com]
Envoyé : jeudi 19 avril 2018 14:59
À : user@manifoldcf.apache.org <ma...@manifoldcf.apache.org>
Objet : Re: UpdateProcessor SolrCloud and ManifoldCF
The Arguments tab is not supposed to add a field. It is for arguments to be sent only, not metadata. But since this is a POST request we are sending in, it's probable that the Solr Connector is sending it in as a form parameter.
It most certainly will *never* be sent as a GET request because that's not how we send documents to Solr (and never did -- they are limited in length that way). Sending query parameters on the URL for a POST is usually not a good idea, and this has never been necessary with Solr in the past.
This suggests that the behavior of Solr has changed recently. I will not have time to look at this today, so can you create a ticket for further research? Thanks!
Karl
On Thu, Apr 19, 2018 at 8:32 AM, msaunier <msaunier@citya.com <ma...@citya.com> > wrote:
Ok. I have add an argument « Znode path » with « processor » name and « CityaTestUpdateProcessorJar » value but ManifoldCF send a field processor, not an GET PARAM.
Solr response :
org.apache.solr.common.SolrException: ERROR: [doc=file://///srvics01/ways_holding/gestion_ged/gerance/3573/201102081135_ENVOIDEVISPP.doc] unknown field 'processor'
I think, arguments add field, not add param on the url.
Thanks.
Maxence,
De : Karl Wright [mailto:daddywri@gmail.com <ma...@gmail.com> ]
Envoyé : jeudi 19 avril 2018 13:44
À : user@manifoldcf.apache.org <ma...@manifoldcf.apache.org>
Objet : Re: UpdateProcessor SolrCloud and ManifoldCF
Hi Maurice,
You're not supposed to add arguments to the handler paths. They're just paths and not full URLs.
You can add arbitrary URL arguments to be sent to Solr elsewhere in the configuration. Look at the "Arguments" tab. For commits, look at the "Commits" tab.
Karl
On Thu, Apr 19, 2018 at 5:41 AM, msaunier <msaunier@citya.com <ma...@citya.com> > wrote:
Hello Karl,
I have a problem to call UpdateProcessor with ManifoldCF commit to SolrCloud.
URL to use UpdateProcessor with curl :
curl "http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdateProcessorJar <http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdateProcessorJar&commit=true> &commit=true" -H 'Content-type: application/json' -d '[{"content": "test.test", "id":"test_2"}]'
So, in ManifoldCF, I have add in Output Connections, in SolrCloud Paths :
Update handler: /update/extract?processor=CityaTestUpdateProcessorJar&commit=true
Remove handler: /update?processor=CityaTestUpdateProcessorJar&commit=true
But, ManifoldCF not call update processor when he push documents.
Do you have an idea ?
Thanks you.
RE: UpdateProcessor SolrCloud and ManifoldCF
Posted by msaunier <ms...@citya.com>.
In SolrCloud we can’t modify processor chain, because, SolrCloud used just API to config. So, use an url parameter is the alone option.
https://lucene.apache.org/solr/guide/6_6/config-api.html#ConfigAPI-Whatabout_updateRequestProcessorChain_
I create a tiket on Jira and I respond with it’s url.
Thanks,
Maxence
De : Karl Wright [mailto:daddywri@gmail.com]
Envoyé : jeudi 19 avril 2018 14:59
À : user@manifoldcf.apache.org
Objet : Re: UpdateProcessor SolrCloud and ManifoldCF
The Arguments tab is not supposed to add a field. It is for arguments to be sent only, not metadata. But since this is a POST request we are sending in, it's probable that the Solr Connector is sending it in as a form parameter.
It most certainly will *never* be sent as a GET request because that's not how we send documents to Solr (and never did -- they are limited in length that way). Sending query parameters on the URL for a POST is usually not a good idea, and this has never been necessary with Solr in the past.
This suggests that the behavior of Solr has changed recently. I will not have time to look at this today, so can you create a ticket for further research? Thanks!
Karl
On Thu, Apr 19, 2018 at 8:32 AM, msaunier <msaunier@citya.com <ma...@citya.com> > wrote:
Ok. I have add an argument « Znode path » with « processor » name and « CityaTestUpdateProcessorJar » value but ManifoldCF send a field processor, not an GET PARAM.
Solr response :
org.apache.solr.common.SolrException: ERROR: [doc=file://///srvics01/ways_holding/gestion_ged/gerance/3573/201102081135_ENVOIDEVISPP.doc] unknown field 'processor'
I think, arguments add field, not add param on the url.
Thanks.
Maxence,
De : Karl Wright [mailto:daddywri@gmail.com <ma...@gmail.com> ]
Envoyé : jeudi 19 avril 2018 13:44
À : user@manifoldcf.apache.org <ma...@manifoldcf.apache.org>
Objet : Re: UpdateProcessor SolrCloud and ManifoldCF
Hi Maurice,
You're not supposed to add arguments to the handler paths. They're just paths and not full URLs.
You can add arbitrary URL arguments to be sent to Solr elsewhere in the configuration. Look at the "Arguments" tab. For commits, look at the "Commits" tab.
Karl
On Thu, Apr 19, 2018 at 5:41 AM, msaunier <msaunier@citya.com <ma...@citya.com> > wrote:
Hello Karl,
I have a problem to call UpdateProcessor with ManifoldCF commit to SolrCloud.
URL to use UpdateProcessor with curl :
curl "http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdateProcessorJar <http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdateProcessorJar&commit=true> &commit=true" -H 'Content-type: application/json' -d '[{"content": "test.test", "id":"test_2"}]'
So, in ManifoldCF, I have add in Output Connections, in SolrCloud Paths :
Update handler: /update/extract?processor=CityaTestUpdateProcessorJar&commit=true
Remove handler: /update?processor=CityaTestUpdateProcessorJar&commit=true
But, ManifoldCF not call update processor when he push documents.
Do you have an idea ?
Thanks you.
Re: UpdateProcessor SolrCloud and ManifoldCF
Posted by Karl Wright <da...@gmail.com>.
The Arguments tab is not supposed to add a field. It is for arguments to
be sent only, not metadata. But since this is a POST request we are
sending in, it's probable that the Solr Connector is sending it in as a
form parameter.
It most certainly will *never* be sent as a GET request because that's not
how we send documents to Solr (and never did -- they are limited in length
that way). Sending query parameters on the URL for a POST is usually not a
good idea, and this has never been necessary with Solr in the past.
This suggests that the behavior of Solr has changed recently. I will not
have time to look at this today, so can you create a ticket for further
research? Thanks!
Karl
On Thu, Apr 19, 2018 at 8:32 AM, msaunier <ms...@citya.com> wrote:
> Ok. I have add an argument « Znode path » with « processor » name and
> « CityaTestUpdateProcessorJar » value but ManifoldCF send a field
> processor, not an GET PARAM.
>
>
>
> Solr response :
>
> org.apache.solr.common.SolrException: ERROR: [doc=file://///srvics01/ways_
> holding/gestion_ged/gerance/3573/201102081135_ENVOIDEVISPP.doc] unknown
> field 'processor'
>
>
>
> I think, arguments add field, not add param on the url.
>
>
>
> Thanks.
>
> Maxence,
>
>
>
> *De :* Karl Wright [mailto:daddywri@gmail.com]
> *Envoyé :* jeudi 19 avril 2018 13:44
> *À :* user@manifoldcf.apache.org
> *Objet :* Re: UpdateProcessor SolrCloud and ManifoldCF
>
>
>
> Hi Maurice,
>
>
>
> You're not supposed to add arguments to the handler paths. They're just
> paths and not full URLs.
>
>
>
> You can add arbitrary URL arguments to be sent to Solr elsewhere in the
> configuration. Look at the "Arguments" tab. For commits, look at the
> "Commits" tab.
>
>
>
> Karl
>
>
>
>
>
> On Thu, Apr 19, 2018 at 5:41 AM, msaunier <ms...@citya.com> wrote:
>
> Hello Karl,
>
>
>
> I have a problem to call UpdateProcessor with ManifoldCF commit to
> SolrCloud.
>
>
>
> URL to use UpdateProcessor with curl :
>
> curl "http://srv-formation-solr:8983/solr/dev1/update?processor=
> CityaTestUpdateProcessorJar&commit=true" -H 'Content-type:
> application/json' -d '[{"content": "test.test", "id":"test_2"}]'
>
>
>
> So, in ManifoldCF, I have add in Output Connections, in SolrCloud Paths :
>
> Update handler: /update/extract?processor=CityaTestUpdateProcessorJar&
> commit=true
>
> Remove handler: /update?processor=CityaTestUpdateProcessorJar&commit=true
>
>
>
>
>
> But, ManifoldCF not call update processor when he push documents.
>
>
>
> Do you have an idea ?
>
>
>
> Thanks you.
>
>
>
RE: UpdateProcessor SolrCloud and ManifoldCF
Posted by msaunier <ms...@citya.com>.
Ok. I have add an argument « Znode path » with « processor » name and « CityaTestUpdateProcessorJar » value but ManifoldCF send a field processor, not an GET PARAM.
Solr response :
org.apache.solr.common.SolrException: ERROR: [doc=file://///srvics01/ways_holding/gestion_ged/gerance/3573/201102081135_ENVOIDEVISPP.doc] unknown field 'processor'
I think, arguments add field, not add param on the url.
Thanks.
Maxence,
De : Karl Wright [mailto:daddywri@gmail.com]
Envoyé : jeudi 19 avril 2018 13:44
À : user@manifoldcf.apache.org
Objet : Re: UpdateProcessor SolrCloud and ManifoldCF
Hi Maurice,
You're not supposed to add arguments to the handler paths. They're just paths and not full URLs.
You can add arbitrary URL arguments to be sent to Solr elsewhere in the configuration. Look at the "Arguments" tab. For commits, look at the "Commits" tab.
Karl
On Thu, Apr 19, 2018 at 5:41 AM, msaunier <msaunier@citya.com <ma...@citya.com> > wrote:
Hello Karl,
I have a problem to call UpdateProcessor with ManifoldCF commit to SolrCloud.
URL to use UpdateProcessor with curl :
curl "http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdateProcessorJar <http://srv-formation-solr:8983/solr/dev1/update?processor=CityaTestUpdateProcessorJar&commit=true> &commit=true" -H 'Content-type: application/json' -d '[{"content": "test.test", "id":"test_2"}]'
So, in ManifoldCF, I have add in Output Connections, in SolrCloud Paths :
Update handler: /update/extract?processor=CityaTestUpdateProcessorJar&commit=true
Remove handler: /update?processor=CityaTestUpdateProcessorJar&commit=true
But, ManifoldCF not call update processor when he push documents.
Do you have an idea ?
Thanks you.
Re: UpdateProcessor SolrCloud and ManifoldCF
Posted by Karl Wright <da...@gmail.com>.
Hi Maurice,
You're not supposed to add arguments to the handler paths. They're just
paths and not full URLs.
You can add arbitrary URL arguments to be sent to Solr elsewhere in the
configuration. Look at the "Arguments" tab. For commits, look at the
"Commits" tab.
Karl
On Thu, Apr 19, 2018 at 5:41 AM, msaunier <ms...@citya.com> wrote:
> Hello Karl,
>
>
>
> I have a problem to call UpdateProcessor with ManifoldCF commit to
> SolrCloud.
>
>
>
> URL to use UpdateProcessor with curl :
>
> curl "http://srv-formation-solr:8983/solr/dev1/update?processor=
> CityaTestUpdateProcessorJar&commit=true" -H 'Content-type:
> application/json' -d '[{"content": "test.test", "id":"test_2"}]'
>
>
>
> So, in ManifoldCF, I have add in Output Connections, in SolrCloud Paths :
>
> Update handler: /update/extract?processor=CityaTestUpdateProcessorJar&
> commit=true
>
> Remove handler: /update?processor=CityaTestUpdateProcessorJar&commit=true
>
>
>
>
>
> But, ManifoldCF not call update processor when he push documents.
>
>
>
> Do you have an idea ?
>
>
>
> Thanks you.
>