You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by rabii lamriq <la...@gmail.com> on 2018/09/21 13:56:06 UTC
x-ndjson via Knox
Hi All,
From Grafana, I Post multisearch query to Elasticsearch using the bulk API
with the newline delimited JSON (NDJSON) format.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
like this:
curl -u myuser -i -H “Content-Type: application/json” -XPOST "
http://host_ES:900/index/_msearch" --data-binary ’
{ first json doc}
{second json doc}
’
when bypassing Knox, The query work well either we made json or x-ndjson,
However, through Knox gateway, if the content-type=application/json, the
query retun this error of 400:
HTTP/1.1 400 Bad Request
content-type: application/json
But when we replace json by x-ndjson like mentioned in the bulk API, it
work well.
My problem is that Grafana send their query with the
content-type=application/json and not x-ndjson, for this, it is possible to
make any thing on Knox to solve this problem?
Regards
Rab
Re: x-ndjson via Knox
Posted by Sandeep Moré <mo...@gmail.com>.
Ok, I think the problem here is that the request payload is getting
stripped for some reason.
Looking closely at your curl request it looks like you are using XPOST, try
using POST and see if it works.
I don't think Knox has support for XPOST.
Also, if POST works can you file a bug.
Best,
Sandeep
On Mon, Oct 8, 2018 at 7:22 AM rabii lamriq <la...@gmail.com> wrote:
> Hi Sandeep
>
> The error is :
> {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation
> Failed: 1: no requests
> added;"}],"type":"action_request_validation_exception","reason":"Validation
> Failed: 1: no requests added;"},"status":400}
>
> I don't find how to rewrite rules.
>
> Best
> Rabii
>
> Le ven. 5 oct. 2018 à 18:53, Sandeep Moré <mo...@gmail.com> a
> écrit :
>
>> When you say it does not work what error do you get ? do you see the
>> rewrite rules for header getting triggered in the debug logs ?
>> Also, what is the exception you get on the elasticsearch side.
>>
>> Best,
>> Sandeep
>>
>> On Fri, Oct 5, 2018 at 11:23 AM rabii lamriq <la...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I tried to change the content-type of the http header in the
>>> ElasticsearchDispatcher java class, but doesn't work :(
>>>
>>> Anyone can give help where we should change the content-type for make
>>> "application/x-ndjson" in class of ElasticsearchDispatcher (attached)
>>>
>>> Regards
>>> Rabii
>>>
>>> Le mar. 25 sept. 2018 à 16:20, rabii lamriq <la...@gmail.com> a écrit :
>>>
>>>> I am not understand how to apply this for content-type.
>>>>
>>>> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <mo...@gmail.com> a
>>>> écrit :
>>>>
>>>>> This is an example in Webhdfs service def.
>>>>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <la...@gmail.com> wrote:
>>>>>
>>>>>> Hi Sandeep,
>>>>>>
>>>>>> Yes, I know that is possible, but I don't know how to rewrite the
>>>>>> rule to access to the header and change the Content-type, I can't find any
>>>>>> thing related to this customization.
>>>>>>
>>>>>> Regards
>>>>>> Rabii
>>>>>>
>>>>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com>
>>>>>> a écrit :
>>>>>>
>>>>>>> Yes, you can use the rewrite rules to update Headers, I don't
>>>>>>> remember exactly which service uses it but there should be few services
>>>>>>> that rewrite headers.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Rewrite rules for knox (link
>>>>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>>>>>> can help me to change header? is it possible?
>>>>>>>>
>>>>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> Any help please??
>>>>>>>>>
>>>>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> thanks every One, for your feedbacks,
>>>>>>>>>>
>>>>>>>>>> Yes, ES accept both application/json and application/x-ndjson,
>>>>>>>>>> there is no problem in ES ni in Grafana, when i test my query with curl to
>>>>>>>>>> send POST to ES, it work well ether json or x-ndjson.
>>>>>>>>>>
>>>>>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't
>>>>>>>>>> know why. so, since Grafana send their POST with only json and not
>>>>>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search doesn't
>>>>>>>>>> work.
>>>>>>>>>>
>>>>>>>>>> I think we have two solution:
>>>>>>>>>>
>>>>>>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>>>>>>> x-ndjson instead of json, or
>>>>>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>>>>>>
>>>>>>>>>> If some one have any idea of how to make this or other solution.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Rabii
>>>>>>>>>>
>>>>>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org>
>>>>>>>>>> a écrit :
>>>>>>>>>>
>>>>>>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>>>>>>
>>>>>>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>>>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>>>>>>>> why I asked for the Elasticsearch logs.
>>>>>>>>>>>
>>>>>>>>>>> It might not be necessary to rewrite the content type.
>>>>>>>>>>>
>>>>>>>>>>> Kevin Risden
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> My problem is how to change the content-type=application/json
>>>>>>>>>>>> to application/x-ndjson by rewriting the header in the knox.
>>>>>>>>>>>>
>>>>>>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>>>>>>> and knox return status 400,
>>>>>>>>>>>>
>>>>>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed
>>>>>>>>>>>> that if i replace json by x-ndjson through knox it wirk well.
>>>>>>>>>>>>
>>>>>>>>>>>> Sence i can't change the content-type in grafana, is this
>>>>>>>>>>>> possible by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Rabii
>>>>>>>>>>>>
>>>>>>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>>>>>>> Kevin Risden
>>>>>>>>>>>> <kr...@apache.org> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Do you have the error from Elasticsearch logs? It might
>>>>>>>>>>>>> explain if Knox or Grafana is sending a bad request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin Risden
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <
>>>>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't see any issues from Knox side, what error do you see
>>>>>>>>>>>>>> in Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>>>>>>>> content-type.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <
>>>>>>>>>>>>>> lamri9@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Sandeep
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks for your feedback, below is the deboug log for this
>>>>>>>>>>>>>>> curl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>>>>>>>> POST http://
>>>>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>>>>>>>> status: 400
>>>>>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> Rabii
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <
>>>>>>>>>>>>>>> moresandeep@gmail.com> a écrit :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello Rab,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they
>>>>>>>>>>>>>>>> will tell us exactly what is happening.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <
>>>>>>>>>>>>>>>> lamri9@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch
>>>>>>>>>>>>>>>>> using the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>>>>>>>> like this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json”
>>>>>>>>>>>>>>>>> -XPOST "http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>>>>>>>> { first json doc}
>>>>>>>>>>>>>>>>> {second json doc}
>>>>>>>>>>>>>>>>> ’
>>>>>>>>>>>>>>>>> when bypassing Knox, The query work well either we made
>>>>>>>>>>>>>>>>> json or x-ndjson,
>>>>>>>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the
>>>>>>>>>>>>>>>>> bulk API, it work well.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>> Rab
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hi Sandeep
The error is :
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation
Failed: 1: no requests
added;"}],"type":"action_request_validation_exception","reason":"Validation
Failed: 1: no requests added;"},"status":400}
I don't find how to rewrite rules.
Best
Rabii
Le ven. 5 oct. 2018 à 18:53, Sandeep Moré <mo...@gmail.com> a écrit :
> When you say it does not work what error do you get ? do you see the
> rewrite rules for header getting triggered in the debug logs ?
> Also, what is the exception you get on the elasticsearch side.
>
> Best,
> Sandeep
>
> On Fri, Oct 5, 2018 at 11:23 AM rabii lamriq <la...@gmail.com> wrote:
>
>> Hi All,
>>
>> I tried to change the content-type of the http header in the
>> ElasticsearchDispatcher java class, but doesn't work :(
>>
>> Anyone can give help where we should change the content-type for make
>> "application/x-ndjson" in class of ElasticsearchDispatcher (attached)
>>
>> Regards
>> Rabii
>>
>> Le mar. 25 sept. 2018 à 16:20, rabii lamriq <la...@gmail.com> a écrit :
>>
>>> I am not understand how to apply this for content-type.
>>>
>>> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <mo...@gmail.com> a
>>> écrit :
>>>
>>>> This is an example in Webhdfs service def.
>>>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>>>>
>>>>
>>>>
>>>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <la...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep,
>>>>>
>>>>> Yes, I know that is possible, but I don't know how to rewrite the rule
>>>>> to access to the header and change the Content-type, I can't find any thing
>>>>> related to this customization.
>>>>>
>>>>> Regards
>>>>> Rabii
>>>>>
>>>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>>> Yes, you can use the rewrite rules to update Headers, I don't
>>>>>> remember exactly which service uses it but there should be few services
>>>>>> that rewrite headers.
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Rewrite rules for knox (link
>>>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>>>>> can help me to change header? is it possible?
>>>>>>>
>>>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hello
>>>>>>>>
>>>>>>>> Any help please??
>>>>>>>>
>>>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> thanks every One, for your feedbacks,
>>>>>>>>>
>>>>>>>>> Yes, ES accept both application/json and application/x-ndjson,
>>>>>>>>> there is no problem in ES ni in Grafana, when i test my query with curl to
>>>>>>>>> send POST to ES, it work well ether json or x-ndjson.
>>>>>>>>>
>>>>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't
>>>>>>>>> know why. so, since Grafana send their POST with only json and not
>>>>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search doesn't
>>>>>>>>> work.
>>>>>>>>>
>>>>>>>>> I think we have two solution:
>>>>>>>>>
>>>>>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>>>>>> x-ndjson instead of json, or
>>>>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>>>>>
>>>>>>>>> If some one have any idea of how to make this or other solution.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Rabii
>>>>>>>>>
>>>>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org>
>>>>>>>>> a écrit :
>>>>>>>>>
>>>>>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>>>>>
>>>>>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>>>>>>> why I asked for the Elasticsearch logs.
>>>>>>>>>>
>>>>>>>>>> It might not be necessary to rewrite the content type.
>>>>>>>>>>
>>>>>>>>>> Kevin Risden
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> My problem is how to change the content-type=application/json to
>>>>>>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>>>>>>
>>>>>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>>>>>> and knox return status 400,
>>>>>>>>>>>
>>>>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed
>>>>>>>>>>> that if i replace json by x-ndjson through knox it wirk well.
>>>>>>>>>>>
>>>>>>>>>>> Sence i can't change the content-type in grafana, is this
>>>>>>>>>>> possible by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Rabii
>>>>>>>>>>>
>>>>>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>>>>>> Kevin Risden
>>>>>>>>>>> <kr...@apache.org> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Do you have the error from Elasticsearch logs? It might explain
>>>>>>>>>>>> if Knox or Grafana is sending a bad request.
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin Risden
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <
>>>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't see any issues from Knox side, what error do you see
>>>>>>>>>>>>> in Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>>>>>>> content-type.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <
>>>>>>>>>>>>> lamri9@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Sandeep
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks for your feedback, below is the deboug log for this
>>>>>>>>>>>>>> curl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>>>>>>> POST http://
>>>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>>>>>>> status: 400
>>>>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Rabii
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <
>>>>>>>>>>>>>> moresandeep@gmail.com> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello Rab,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will
>>>>>>>>>>>>>>> tell us exactly what is happening.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <
>>>>>>>>>>>>>>> lamri9@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch
>>>>>>>>>>>>>>>> using the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>>>>>>> like this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json”
>>>>>>>>>>>>>>>> -XPOST "http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>>>>>>> { first json doc}
>>>>>>>>>>>>>>>> {second json doc}
>>>>>>>>>>>>>>>> ’
>>>>>>>>>>>>>>>> when bypassing Knox, The query work well either we made
>>>>>>>>>>>>>>>> json or x-ndjson,
>>>>>>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the
>>>>>>>>>>>>>>>> bulk API, it work well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>> Rab
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
Re: x-ndjson via Knox
Posted by Sandeep Moré <mo...@gmail.com>.
When you say it does not work what error do you get ? do you see the
rewrite rules for header getting triggered in the debug logs ?
Also, what is the exception you get on the elasticsearch side.
Best,
Sandeep
On Fri, Oct 5, 2018 at 11:23 AM rabii lamriq <la...@gmail.com> wrote:
> Hi All,
>
> I tried to change the content-type of the http header in the
> ElasticsearchDispatcher java class, but doesn't work :(
>
> Anyone can give help where we should change the content-type for make
> "application/x-ndjson" in class of ElasticsearchDispatcher (attached)
>
> Regards
> Rabii
>
> Le mar. 25 sept. 2018 à 16:20, rabii lamriq <la...@gmail.com> a écrit :
>
>> I am not understand how to apply this for content-type.
>>
>> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <mo...@gmail.com> a
>> écrit :
>>
>>> This is an example in Webhdfs service def.
>>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>>>
>>>
>>>
>>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <la...@gmail.com> wrote:
>>>
>>>> Hi Sandeep,
>>>>
>>>> Yes, I know that is possible, but I don't know how to rewrite the rule
>>>> to access to the header and change the Content-type, I can't find any thing
>>>> related to this customization.
>>>>
>>>> Regards
>>>> Rabii
>>>>
>>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Yes, you can use the rewrite rules to update Headers, I don't remember
>>>>> exactly which service uses it but there should be few services that rewrite
>>>>> headers.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Rewrite rules for knox (link
>>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>>>> can help me to change header? is it possible?
>>>>>>
>>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> Any help please??
>>>>>>>
>>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> thanks every One, for your feedbacks,
>>>>>>>>
>>>>>>>> Yes, ES accept both application/json and application/x-ndjson,
>>>>>>>> there is no problem in ES ni in Grafana, when i test my query with curl to
>>>>>>>> send POST to ES, it work well ether json or x-ndjson.
>>>>>>>>
>>>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't
>>>>>>>> know why. so, since Grafana send their POST with only json and not
>>>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search doesn't
>>>>>>>> work.
>>>>>>>>
>>>>>>>> I think we have two solution:
>>>>>>>>
>>>>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>>>>> x-ndjson instead of json, or
>>>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>>>>
>>>>>>>> If some one have any idea of how to make this or other solution.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Rabii
>>>>>>>>
>>>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>>>>
>>>>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>>>>>> why I asked for the Elasticsearch logs.
>>>>>>>>>
>>>>>>>>> It might not be necessary to rewrite the content type.
>>>>>>>>>
>>>>>>>>> Kevin Risden
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> My problem is how to change the content-type=application/json to
>>>>>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>>>>>
>>>>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>>>>> and knox return status 400,
>>>>>>>>>>
>>>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed
>>>>>>>>>> that if i replace json by x-ndjson through knox it wirk well.
>>>>>>>>>>
>>>>>>>>>> Sence i can't change the content-type in grafana, is this
>>>>>>>>>> possible by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Rabii
>>>>>>>>>>
>>>>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>>>>> Kevin Risden
>>>>>>>>>> <kr...@apache.org> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Do you have the error from Elasticsearch logs? It might explain
>>>>>>>>>>> if Knox or Grafana is sending a bad request.
>>>>>>>>>>>
>>>>>>>>>>> Kevin Risden
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <
>>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>>>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>>>>>> content-type.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Sandeep
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>>>>>> POST http://
>>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>>>>>> status: 400
>>>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Rabii
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <
>>>>>>>>>>>>> moresandeep@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello Rab,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will
>>>>>>>>>>>>>> tell us exactly what is happening.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <
>>>>>>>>>>>>>> lamri9@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch
>>>>>>>>>>>>>>> using the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>>>>>> like this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST
>>>>>>>>>>>>>>> "http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>>>>>> { first json doc}
>>>>>>>>>>>>>>> {second json doc}
>>>>>>>>>>>>>>> ’
>>>>>>>>>>>>>>> when bypassing Knox, The query work well either we made json
>>>>>>>>>>>>>>> or x-ndjson,
>>>>>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the
>>>>>>>>>>>>>>> bulk API, it work well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> Rab
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hi All,
I tried to change the content-type of the http header in the
ElasticsearchDispatcher java class, but doesn't work :(
Anyone can give help where we should change the content-type for make
"application/x-ndjson" in class of ElasticsearchDispatcher (attached)
Regards
Rabii
Le mar. 25 sept. 2018 à 16:20, rabii lamriq <la...@gmail.com> a écrit :
> I am not understand how to apply this for content-type.
>
> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <mo...@gmail.com> a
> écrit :
>
>> This is an example in Webhdfs service def.
>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>>
>>
>>
>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <la...@gmail.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> Yes, I know that is possible, but I don't know how to rewrite the rule
>>> to access to the header and change the Content-type, I can't find any thing
>>> related to this customization.
>>>
>>> Regards
>>> Rabii
>>>
>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com> a
>>> écrit :
>>>
>>>> Yes, you can use the rewrite rules to update Headers, I don't remember
>>>> exactly which service uses it but there should be few services that rewrite
>>>> headers.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com> wrote:
>>>>
>>>>>
>>>>> Rewrite rules for knox (link
>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>>> can help me to change header? is it possible?
>>>>>
>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>>> Hello
>>>>>>
>>>>>> Any help please??
>>>>>>
>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> thanks every One, for your feedbacks,
>>>>>>>
>>>>>>> Yes, ES accept both application/json and application/x-ndjson, there
>>>>>>> is no problem in ES ni in Grafana, when i test my query with curl to send
>>>>>>> POST to ES, it work well ether json or x-ndjson.
>>>>>>>
>>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't
>>>>>>> know why. so, since Grafana send their POST with only json and not
>>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search doesn't
>>>>>>> work.
>>>>>>>
>>>>>>> I think we have two solution:
>>>>>>>
>>>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>>>> x-ndjson instead of json, or
>>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>>>
>>>>>>> If some one have any idea of how to make this or other solution.
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>> Rabii
>>>>>>>
>>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>>>
>>>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>>>>> why I asked for the Elasticsearch logs.
>>>>>>>>
>>>>>>>> It might not be necessary to rewrite the content type.
>>>>>>>>
>>>>>>>> Kevin Risden
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> My problem is how to change the content-type=application/json to
>>>>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>>>>
>>>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>>>> and knox return status 400,
>>>>>>>>>
>>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed that
>>>>>>>>> if i replace json by x-ndjson through knox it wirk well.
>>>>>>>>>
>>>>>>>>> Sence i can't change the content-type in grafana, is this possible
>>>>>>>>> by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Rabii
>>>>>>>>>
>>>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>>>> Kevin Risden
>>>>>>>>> <kr...@apache.org> a écrit :
>>>>>>>>>
>>>>>>>>>> Do you have the error from Elasticsearch logs? It might explain
>>>>>>>>>> if Knox or Grafana is sending a bad request.
>>>>>>>>>>
>>>>>>>>>> Kevin Risden
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <
>>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>>>>> content-type.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Sandeep
>>>>>>>>>>>>
>>>>>>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>>>>> POST http://
>>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>>>>> status: 400
>>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Rabii
>>>>>>>>>>>>
>>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <
>>>>>>>>>>>> moresandeep@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Rab,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will
>>>>>>>>>>>>> tell us exactly what is happening.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using
>>>>>>>>>>>>>> the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>>>>> like this:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>>>>> { first json doc}
>>>>>>>>>>>>>> {second json doc}
>>>>>>>>>>>>>> ’
>>>>>>>>>>>>>> when bypassing Knox, The query work well either we made json
>>>>>>>>>>>>>> or x-ndjson,
>>>>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the
>>>>>>>>>>>>>> bulk API, it work well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Rab
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
I am not understand how to apply this for content-type.
Le mar. 25 sept. 2018 à 15:18, Sandeep Moré <mo...@gmail.com> a
écrit :
> This is an example in Webhdfs service def.
> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>
>
>
> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <la...@gmail.com> wrote:
>
>> Hi Sandeep,
>>
>> Yes, I know that is possible, but I don't know how to rewrite the rule to
>> access to the header and change the Content-type, I can't find any thing
>> related to this customization.
>>
>> Regards
>> Rabii
>>
>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com> a
>> écrit :
>>
>>> Yes, you can use the rewrite rules to update Headers, I don't remember
>>> exactly which service uses it but there should be few services that rewrite
>>> headers.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com> wrote:
>>>
>>>>
>>>> Rewrite rules for knox (link
>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>> can help me to change header? is it possible?
>>>>
>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Hello
>>>>>
>>>>> Any help please??
>>>>>
>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> thanks every One, for your feedbacks,
>>>>>>
>>>>>> Yes, ES accept both application/json and application/x-ndjson, there
>>>>>> is no problem in ES ni in Grafana, when i test my query with curl to send
>>>>>> POST to ES, it work well ether json or x-ndjson.
>>>>>>
>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't
>>>>>> know why. so, since Grafana send their POST with only json and not
>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search doesn't
>>>>>> work.
>>>>>>
>>>>>> I think we have two solution:
>>>>>>
>>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>>> x-ndjson instead of json, or
>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>>
>>>>>> If some one have any idea of how to make this or other solution.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>> Rabii
>>>>>>
>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>>>>>> écrit :
>>>>>>
>>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>>
>>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>>>> why I asked for the Elasticsearch logs.
>>>>>>>
>>>>>>> It might not be necessary to rewrite the content type.
>>>>>>>
>>>>>>> Kevin Risden
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> My problem is how to change the content-type=application/json to
>>>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>>>
>>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>>> and knox return status 400,
>>>>>>>>
>>>>>>>> Hower when i reproduce the problem by curl comande, I noticed that
>>>>>>>> if i replace json by x-ndjson through knox it wirk well.
>>>>>>>>
>>>>>>>> Sence i can't change the content-type in grafana, is this possible
>>>>>>>> by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Rabii
>>>>>>>>
>>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>>> Kevin Risden
>>>>>>>> <kr...@apache.org> a écrit :
>>>>>>>>
>>>>>>>>> Do you have the error from Elasticsearch logs? It might explain if
>>>>>>>>> Knox or Grafana is sending a bad request.
>>>>>>>>>
>>>>>>>>> Kevin Risden
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <
>>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>>>> content-type.
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Sandeep
>>>>>>>>>>>
>>>>>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>>>>>
>>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>>>> POST http://
>>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>>>> status: 400
>>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Rabii
>>>>>>>>>>>
>>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <
>>>>>>>>>>> moresandeep@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Hello Rab,
>>>>>>>>>>>>
>>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will
>>>>>>>>>>>> tell us exactly what is happening.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using
>>>>>>>>>>>>> the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>>>> like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>>>> { first json doc}
>>>>>>>>>>>>> {second json doc}
>>>>>>>>>>>>> ’
>>>>>>>>>>>>> when bypassing Knox, The query work well either we made json
>>>>>>>>>>>>> or x-ndjson,
>>>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>>>
>>>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>>>
>>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the
>>>>>>>>>>>>> bulk API, it work well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Rab
>>>>>>>>>>>>>
>>>>>>>>>>>>
Re: x-ndjson via Knox
Posted by Sandeep Moré <mo...@gmail.com>.
This is an example in Webhdfs service def.
https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq <la...@gmail.com> wrote:
> Hi Sandeep,
>
> Yes, I know that is possible, but I don't know how to rewrite the rule to
> access to the header and change the Content-type, I can't find any thing
> related to this customization.
>
> Regards
> Rabii
>
> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com> a
> écrit :
>
>> Yes, you can use the rewrite rules to update Headers, I don't remember
>> exactly which service uses it but there should be few services that rewrite
>> headers.
>>
>> Best,
>> Sandeep
>>
>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com> wrote:
>>
>>>
>>> Rewrite rules for knox (link
>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>> can help me to change header? is it possible?
>>>
>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a écrit :
>>>
>>>> Hello
>>>>
>>>> Any help please??
>>>>
>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Hi,
>>>>>
>>>>> thanks every One, for your feedbacks,
>>>>>
>>>>> Yes, ES accept both application/json and application/x-ndjson, there
>>>>> is no problem in ES ni in Grafana, when i test my query with curl to send
>>>>> POST to ES, it work well ether json or x-ndjson.
>>>>>
>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't know
>>>>> why. so, since Grafana send their POST with only json and not x-ndjson, the
>>>>> scenario of Grafana -> Knox -> ES with multi search doesn't work.
>>>>>
>>>>> I think we have two solution:
>>>>>
>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>> x-ndjson instead of json, or
>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>
>>>>> If some one have any idea of how to make this or other solution.
>>>>>
>>>>>
>>>>> Regards
>>>>> Rabii
>>>>>
>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>>>>> écrit :
>>>>>
>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>
>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>>> why I asked for the Elasticsearch logs.
>>>>>>
>>>>>> It might not be necessary to rewrite the content type.
>>>>>>
>>>>>> Kevin Risden
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> My problem is how to change the content-type=application/json to
>>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>>
>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>> and knox return status 400,
>>>>>>>
>>>>>>> Hower when i reproduce the problem by curl comande, I noticed that
>>>>>>> if i replace json by x-ndjson through knox it wirk well.
>>>>>>>
>>>>>>> Sence i can't change the content-type in grafana, is this possible
>>>>>>> by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>
>>>>>>> Regards
>>>>>>> Rabii
>>>>>>>
>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>> Kevin Risden
>>>>>>> <kr...@apache.org> a écrit :
>>>>>>>
>>>>>>>> Do you have the error from Elasticsearch logs? It might explain if
>>>>>>>> Knox or Grafana is sending a bad request.
>>>>>>>>
>>>>>>>> Kevin Risden
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <
>>>>>>>> moresandeep@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>>> content-type.
>>>>>>>>>
>>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Sandeep
>>>>>>>>>>
>>>>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>>>>
>>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>>> status: 400
>>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Rabii
>>>>>>>>>>
>>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <
>>>>>>>>>> moresandeep@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>>> Hello Rab,
>>>>>>>>>>>
>>>>>>>>>>> Interesting, do you have the debug logs for this ? they will
>>>>>>>>>>> tell us exactly what is happening.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Sandeep
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using
>>>>>>>>>>>> the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>>> like this:
>>>>>>>>>>>>
>>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>>> { first json doc}
>>>>>>>>>>>> {second json doc}
>>>>>>>>>>>> ’
>>>>>>>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>>>>>>>> x-ndjson,
>>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>>
>>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>>
>>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the bulk
>>>>>>>>>>>> API, it work well.
>>>>>>>>>>>>
>>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Rab
>>>>>>>>>>>>
>>>>>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hi Sandeep,
Yes, I know that is possible, but I don't know how to rewrite the rule to
access to the header and change the Content-type, I can't find any thing
related to this customization.
Regards
Rabii
Le lun. 24 sept. 2018 à 17:19, Sandeep Moré <mo...@gmail.com> a
écrit :
> Yes, you can use the rewrite rules to update Headers, I don't remember
> exactly which service uses it but there should be few services that rewrite
> headers.
>
> Best,
> Sandeep
>
> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com> wrote:
>
>>
>> Rewrite rules for knox (link
>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>> can help me to change header? is it possible?
>>
>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a écrit :
>>
>>> Hello
>>>
>>> Any help please??
>>>
>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a écrit :
>>>
>>>> Hi,
>>>>
>>>> thanks every One, for your feedbacks,
>>>>
>>>> Yes, ES accept both application/json and application/x-ndjson, there is
>>>> no problem in ES ni in Grafana, when i test my query with curl to send POST
>>>> to ES, it work well ether json or x-ndjson.
>>>>
>>>> However, through Knox Gateway, it work only for x-ndjson, I don't know
>>>> why. so, since Grafana send their POST with only json and not x-ndjson, the
>>>> scenario of Grafana -> Knox -> ES with multi search doesn't work.
>>>>
>>>> I think we have two solution:
>>>>
>>>> 1- make some thing to force Grafana change header of POST and put
>>>> x-ndjson instead of json, or
>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>
>>>> If some one have any idea of how to make this or other solution.
>>>>
>>>>
>>>> Regards
>>>> Rabii
>>>>
>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>>>> écrit :
>>>>
>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>
>>>>> Elasticsearch endpoints should take both application/json and
>>>>> application/x-ndjson. There might be something else going on here. That is
>>>>> why I asked for the Elasticsearch logs.
>>>>>
>>>>> It might not be necessary to rewrite the content type.
>>>>>
>>>>> Kevin Risden
>>>>>
>>>>>
>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> My problem is how to change the content-type=application/json to
>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>
>>>>>> Because grafana send their query with content-type=application/json
>>>>>> and knox return status 400,
>>>>>>
>>>>>> Hower when i reproduce the problem by curl comande, I noticed that if
>>>>>> i replace json by x-ndjson through knox it wirk well.
>>>>>>
>>>>>> Sence i can't change the content-type in grafana, is this possible by
>>>>>> rewriting the header in knox to replace json by x-ndjson.
>>>>>>
>>>>>> Regards
>>>>>> Rabii
>>>>>>
>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>> Kevin Risden
>>>>>> <kr...@apache.org> a écrit :
>>>>>>
>>>>>>> Do you have the error from Elasticsearch logs? It might explain if
>>>>>>> Knox or Grafana is sending a bad request.
>>>>>>>
>>>>>>> Kevin Risden
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>>>> content-type.
>>>>>>>>
>>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Sandeep
>>>>>>>>>
>>>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>>>
>>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>>> status: 400
>>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Rabii
>>>>>>>>>
>>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com>
>>>>>>>>> a écrit :
>>>>>>>>>
>>>>>>>>>> Hello Rab,
>>>>>>>>>>
>>>>>>>>>> Interesting, do you have the debug logs for this ? they will tell
>>>>>>>>>> us exactly what is happening.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Sandeep
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using
>>>>>>>>>>> the bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>>> like this:
>>>>>>>>>>>
>>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>>> { first json doc}
>>>>>>>>>>> {second json doc}
>>>>>>>>>>> ’
>>>>>>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>>>>>>> x-ndjson,
>>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>>
>>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>>> content-type: application/json
>>>>>>>>>>>
>>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the bulk
>>>>>>>>>>> API, it work well.
>>>>>>>>>>>
>>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Rab
>>>>>>>>>>>
>>>>>>>>>>
Re: x-ndjson via Knox
Posted by Sandeep Moré <mo...@gmail.com>.
Yes, you can use the rewrite rules to update Headers, I don't remember
exactly which service uses it but there should be few services that rewrite
headers.
Best,
Sandeep
On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq <la...@gmail.com> wrote:
>
> Rewrite rules for knox (link
> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
> can help me to change header? is it possible?
>
> Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a écrit :
>
>> Hello
>>
>> Any help please??
>>
>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a écrit :
>>
>>> Hi,
>>>
>>> thanks every One, for your feedbacks,
>>>
>>> Yes, ES accept both application/json and application/x-ndjson, there is
>>> no problem in ES ni in Grafana, when i test my query with curl to send POST
>>> to ES, it work well ether json or x-ndjson.
>>>
>>> However, through Knox Gateway, it work only for x-ndjson, I don't know
>>> why. so, since Grafana send their POST with only json and not x-ndjson, the
>>> scenario of Grafana -> Knox -> ES with multi search doesn't work.
>>>
>>> I think we have two solution:
>>>
>>> 1- make some thing to force Grafana change header of POST and put
>>> x-ndjson instead of json, or
>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>
>>> If some one have any idea of how to make this or other solution.
>>>
>>>
>>> Regards
>>> Rabii
>>>
>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>>> écrit :
>>>
>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>
>>>> Elasticsearch endpoints should take both application/json and
>>>> application/x-ndjson. There might be something else going on here. That is
>>>> why I asked for the Elasticsearch logs.
>>>>
>>>> It might not be necessary to rewrite the content type.
>>>>
>>>> Kevin Risden
>>>>
>>>>
>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> My problem is how to change the content-type=application/json to
>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>
>>>>> Because grafana send their query with content-type=application/json
>>>>> and knox return status 400,
>>>>>
>>>>> Hower when i reproduce the problem by curl comande, I noticed that if
>>>>> i replace json by x-ndjson through knox it wirk well.
>>>>>
>>>>> Sence i can't change the content-type in grafana, is this possible by
>>>>> rewriting the header in knox to replace json by x-ndjson.
>>>>>
>>>>> Regards
>>>>> Rabii
>>>>>
>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>> Kevin Risden
>>>>> <kr...@apache.org> a écrit :
>>>>>
>>>>>> Do you have the error from Elasticsearch logs? It might explain if
>>>>>> Knox or Grafana is sending a bad request.
>>>>>>
>>>>>> Kevin Risden
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>>> content-type.
>>>>>>>
>>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Sandeep
>>>>>>>>
>>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>>
>>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>>> status: 400
>>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>>> /es5/<index_name>/_msearch
>>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Rabii
>>>>>>>>
>>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com>
>>>>>>>> a écrit :
>>>>>>>>
>>>>>>>>> Hello Rab,
>>>>>>>>>
>>>>>>>>> Interesting, do you have the debug logs for this ? they will tell
>>>>>>>>> us exactly what is happening.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Sandeep
>>>>>>>>>
>>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using the
>>>>>>>>>> bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>>> like this:
>>>>>>>>>>
>>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>>> { first json doc}
>>>>>>>>>> {second json doc}
>>>>>>>>>> ’
>>>>>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>>>>>> x-ndjson,
>>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>>
>>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>>> content-type: application/json
>>>>>>>>>>
>>>>>>>>>> But when we replace json by x-ndjson like mentioned in the bulk
>>>>>>>>>> API, it work well.
>>>>>>>>>>
>>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Rab
>>>>>>>>>>
>>>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Rewrite rules for knox (link
<https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
can help me to change header? is it possible?
Le lun. 24 sept. 2018 à 16:43, rabii lamriq <la...@gmail.com> a écrit :
> Hello
>
> Any help please??
>
> Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a écrit :
>
>> Hi,
>>
>> thanks every One, for your feedbacks,
>>
>> Yes, ES accept both application/json and application/x-ndjson, there is
>> no problem in ES ni in Grafana, when i test my query with curl to send POST
>> to ES, it work well ether json or x-ndjson.
>>
>> However, through Knox Gateway, it work only for x-ndjson, I don't know
>> why. so, since Grafana send their POST with only json and not x-ndjson, the
>> scenario of Grafana -> Knox -> ES with multi search doesn't work.
>>
>> I think we have two solution:
>>
>> 1- make some thing to force Grafana change header of POST and put
>> x-ndjson instead of json, or
>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>
>> If some one have any idea of how to make this or other solution.
>>
>>
>> Regards
>> Rabii
>>
>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a
>> écrit :
>>
>>> https://github.com/elastic/elasticsearch/issues/25718
>>>
>>> Elasticsearch endpoints should take both application/json and
>>> application/x-ndjson. There might be something else going on here. That is
>>> why I asked for the Elasticsearch logs.
>>>
>>> It might not be necessary to rewrite the content type.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> My problem is how to change the content-type=application/json to
>>>> application/x-ndjson by rewriting the header in the knox.
>>>>
>>>> Because grafana send their query with content-type=application/json
>>>> and knox return status 400,
>>>>
>>>> Hower when i reproduce the problem by curl comande, I noticed that if i
>>>> replace json by x-ndjson through knox it wirk well.
>>>>
>>>> Sence i can't change the content-type in grafana, is this possible by
>>>> rewriting the header in knox to replace json by x-ndjson.
>>>>
>>>> Regards
>>>> Rabii
>>>>
>>>> Le ven. 21 sept. 2018 à 18:05,
>>>> Kevin Risden
>>>> <kr...@apache.org> a écrit :
>>>>
>>>>> Do you have the error from Elasticsearch logs? It might explain if
>>>>> Knox or Grafana is sending a bad request.
>>>>>
>>>>> Kevin Risden
>>>>>
>>>>>
>>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I don't see any issues from Knox side, what error do you see in
>>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>>> content-type.
>>>>>>
>>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Sandeep
>>>>>>>
>>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>>
>>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>>> status: 400
>>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>>> character set UTF-8 for entity of type application/json
>>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>>> /es5/<index_name>/_msearch
>>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>>
>>>>>>> Regards
>>>>>>> Rabii
>>>>>>>
>>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com>
>>>>>>> a écrit :
>>>>>>>
>>>>>>>> Hello Rab,
>>>>>>>>
>>>>>>>> Interesting, do you have the debug logs for this ? they will tell
>>>>>>>> us exactly what is happening.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Sandeep
>>>>>>>>
>>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using the
>>>>>>>>> bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>>> like this:
>>>>>>>>>
>>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>>> { first json doc}
>>>>>>>>> {second json doc}
>>>>>>>>> ’
>>>>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>>>>> x-ndjson,
>>>>>>>>> However, through Knox gateway, if the
>>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>>
>>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>>> content-type: application/json
>>>>>>>>>
>>>>>>>>> But when we replace json by x-ndjson like mentioned in the bulk
>>>>>>>>> API, it work well.
>>>>>>>>>
>>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Rab
>>>>>>>>>
>>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hello
Any help please??
Le lun. 24 sept. 2018 à 09:31, rabii lamriq <la...@gmail.com> a écrit :
> Hi,
>
> thanks every One, for your feedbacks,
>
> Yes, ES accept both application/json and application/x-ndjson, there is no
> problem in ES ni in Grafana, when i test my query with curl to send POST to
> ES, it work well ether json or x-ndjson.
>
> However, through Knox Gateway, it work only for x-ndjson, I don't know
> why. so, since Grafana send their POST with only json and not x-ndjson, the
> scenario of Grafana -> Knox -> ES with multi search doesn't work.
>
> I think we have two solution:
>
> 1- make some thing to force Grafana change header of POST and put x-ndjson
> instead of json, or
> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>
> If some one have any idea of how to make this or other solution.
>
>
> Regards
> Rabii
>
> Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a écrit :
>
>> https://github.com/elastic/elasticsearch/issues/25718
>>
>> Elasticsearch endpoints should take both application/json and
>> application/x-ndjson. There might be something else going on here. That is
>> why I asked for the Elasticsearch logs.
>>
>> It might not be necessary to rewrite the content type.
>>
>> Kevin Risden
>>
>>
>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> My problem is how to change the content-type=application/json to
>>> application/x-ndjson by rewriting the header in the knox.
>>>
>>> Because grafana send their query with content-type=application/json and
>>> knox return status 400,
>>>
>>> Hower when i reproduce the problem by curl comande, I noticed that if i
>>> replace json by x-ndjson through knox it wirk well.
>>>
>>> Sence i can't change the content-type in grafana, is this possible by
>>> rewriting the header in knox to replace json by x-ndjson.
>>>
>>> Regards
>>> Rabii
>>>
>>> Le ven. 21 sept. 2018 à 18:05,
>>> Kevin Risden
>>> <kr...@apache.org> a écrit :
>>>
>>>> Do you have the error from Elasticsearch logs? It might explain if Knox
>>>> or Grafana is sending a bad request.
>>>>
>>>> Kevin Risden
>>>>
>>>>
>>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> I don't see any issues from Knox side, what error do you see in
>>>>> Grafana ? I think this could be because Grafana is expecting certain
>>>>> content-type.
>>>>>
>>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Sandeep
>>>>>>
>>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>>
>>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>>> status: 400
>>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>>> character set UTF-8 for entity of type application/json
>>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>>> response entity content type: application/json; charset=UTF-8
>>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>>> /es5/<index_name>/_msearch
>>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>>
>>>>>> Regards
>>>>>> Rabii
>>>>>>
>>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com>
>>>>>> a écrit :
>>>>>>
>>>>>>> Hello Rab,
>>>>>>>
>>>>>>> Interesting, do you have the debug logs for this ? they will tell us
>>>>>>> exactly what is happening.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using the
>>>>>>>> bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>>> like this:
>>>>>>>>
>>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>>> { first json doc}
>>>>>>>> {second json doc}
>>>>>>>> ’
>>>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>>>> x-ndjson,
>>>>>>>> However, through Knox gateway, if the
>>>>>>>> content-type=application/json, the query retun this error of 400:
>>>>>>>>
>>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>>> content-type: application/json
>>>>>>>>
>>>>>>>> But when we replace json by x-ndjson like mentioned in the bulk
>>>>>>>> API, it work well.
>>>>>>>>
>>>>>>>> My problem is that Grafana send their query with the
>>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Rab
>>>>>>>>
>>>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hi,
thanks every One, for your feedbacks,
Yes, ES accept both application/json and application/x-ndjson, there is no
problem in ES ni in Grafana, when i test my query with curl to send POST to
ES, it work well ether json or x-ndjson.
However, through Knox Gateway, it work only for x-ndjson, I don't know why.
so, since Grafana send their POST with only json and not x-ndjson, the
scenario of Grafana -> Knox -> ES with multi search doesn't work.
I think we have two solution:
1- make some thing to force Grafana change header of POST and put x-ndjson
instead of json, or
2- rewrite Header by Knox to forwarded with x-ndjson to ES.
If some one have any idea of how to make this or other solution.
Regards
Rabii
Le ven. 21 sept. 2018 à 19:21, Kevin Risden <kr...@apache.org> a écrit :
> https://github.com/elastic/elasticsearch/issues/25718
>
> Elasticsearch endpoints should take both application/json and
> application/x-ndjson. There might be something else going on here. That is
> why I asked for the Elasticsearch logs.
>
> It might not be necessary to rewrite the content type.
>
> Kevin Risden
>
>
> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com> wrote:
>
>> Hi all,
>>
>> My problem is how to change the content-type=application/json to
>> application/x-ndjson by rewriting the header in the knox.
>>
>> Because grafana send their query with content-type=application/json and
>> knox return status 400,
>>
>> Hower when i reproduce the problem by curl comande, I noticed that if i
>> replace json by x-ndjson through knox it wirk well.
>>
>> Sence i can't change the content-type in grafana, is this possible by
>> rewriting the header in knox to replace json by x-ndjson.
>>
>> Regards
>> Rabii
>>
>> Le ven. 21 sept. 2018 à 18:05,
>> Kevin Risden
>> <kr...@apache.org> a écrit :
>>
>>> Do you have the error from Elasticsearch logs? It might explain if Knox
>>> or Grafana is sending a bad request.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
>>> wrote:
>>>
>>>> I don't see any issues from Knox side, what error do you see in Grafana
>>>> ? I think this could be because Grafana is expecting certain content-type.
>>>>
>>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com> wrote:
>>>>
>>>>> Hi Sandeep
>>>>>
>>>>> thanks for your feedback, below is the deboug log for this curl
>>>>>
>>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>>> <host-elast>:9200/<index_name>/_msearch
>>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>>> status: 400
>>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>>> character set UTF-8 for entity of type application/json
>>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>>> response entity content type: application/json; charset=UTF-8
>>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>>> /es5/<index_name>/_msearch
>>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>>
>>>>> Regards
>>>>> Rabii
>>>>>
>>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com> a
>>>>> écrit :
>>>>>
>>>>>> Hello Rab,
>>>>>>
>>>>>> Interesting, do you have the debug logs for this ? they will tell us
>>>>>> exactly what is happening.
>>>>>>
>>>>>> Best,
>>>>>> Sandeep
>>>>>>
>>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> From Grafana, I Post multisearch query to Elasticsearch using the
>>>>>>> bulk API with the newline delimited JSON (NDJSON) format.
>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>>> like this:
>>>>>>>
>>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>>> { first json doc}
>>>>>>> {second json doc}
>>>>>>> ’
>>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>>> x-ndjson,
>>>>>>> However, through Knox gateway, if the content-type=application/json,
>>>>>>> the query retun this error of 400:
>>>>>>>
>>>>>>> HTTP/1.1 400 Bad Request
>>>>>>> content-type: application/json
>>>>>>>
>>>>>>> But when we replace json by x-ndjson like mentioned in the bulk API,
>>>>>>> it work well.
>>>>>>>
>>>>>>> My problem is that Grafana send their query with the
>>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>>> make any thing on Knox to solve this problem?
>>>>>>>
>>>>>>> Regards
>>>>>>> Rab
>>>>>>>
>>>>>>
Re: x-ndjson via Knox
Posted by Kevin Risden <kr...@apache.org>.
https://github.com/elastic/elasticsearch/issues/25718
Elasticsearch endpoints should take both application/json and
application/x-ndjson. There might be something else going on here. That is
why I asked for the Elasticsearch logs.
It might not be necessary to rewrite the content type.
Kevin Risden
On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq <la...@gmail.com> wrote:
> Hi all,
>
> My problem is how to change the content-type=application/json to
> application/x-ndjson by rewriting the header in the knox.
>
> Because grafana send their query with content-type=application/json and
> knox return status 400,
>
> Hower when i reproduce the problem by curl comande, I noticed that if i
> replace json by x-ndjson through knox it wirk well.
>
> Sence i can't change the content-type in grafana, is this possible by
> rewriting the header in knox to replace json by x-ndjson.
>
> Regards
> Rabii
>
> Le ven. 21 sept. 2018 à 18:05,
> Kevin Risden
> <kr...@apache.org> a écrit :
>
>> Do you have the error from Elasticsearch logs? It might explain if Knox
>> or Grafana is sending a bad request.
>>
>> Kevin Risden
>>
>>
>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
>> wrote:
>>
>>> I don't see any issues from Knox side, what error do you see in Grafana
>>> ? I think this could be because Grafana is expecting certain content-type.
>>>
>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com> wrote:
>>>
>>>> Hi Sandeep
>>>>
>>>> thanks for your feedback, below is the deboug log for this curl
>>>>
>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>> <host-elast>:9200/<index_name>/_msearch
>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>> status: 400
>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>> character set UTF-8 for entity of type application/json
>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>> response entity content type: application/json; charset=UTF-8
>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>> /es5/<index_name>/_msearch
>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>
>>>> Regards
>>>> Rabii
>>>>
>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com> a
>>>> écrit :
>>>>
>>>>> Hello Rab,
>>>>>
>>>>> Interesting, do you have the debug logs for this ? they will tell us
>>>>> exactly what is happening.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> From Grafana, I Post multisearch query to Elasticsearch using the
>>>>>> bulk API with the newline delimited JSON (NDJSON) format.
>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>> like this:
>>>>>>
>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>>> { first json doc}
>>>>>> {second json doc}
>>>>>> ’
>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>> x-ndjson,
>>>>>> However, through Knox gateway, if the content-type=application/json,
>>>>>> the query retun this error of 400:
>>>>>>
>>>>>> HTTP/1.1 400 Bad Request
>>>>>> content-type: application/json
>>>>>>
>>>>>> But when we replace json by x-ndjson like mentioned in the bulk API,
>>>>>> it work well.
>>>>>>
>>>>>> My problem is that Grafana send their query with the
>>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>>> make any thing on Knox to solve this problem?
>>>>>>
>>>>>> Regards
>>>>>> Rab
>>>>>>
>>>>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hi all,
My problem is how to change the content-type=application/json to
application/x-ndjson by rewriting the header in the knox.
Because grafana send their query with content-type=application/json and
knox return status 400,
Hower when i reproduce the problem by curl comande, I noticed that if i
replace json by x-ndjson through knox it wirk well.
Sence i can't change the content-type in grafana, is this possible by
rewriting the header in knox to replace json by x-ndjson.
Regards
Rabii
Le ven. 21 sept. 2018 à 18:05, Kevin Risden <kr...@apache.org> a écrit :
> Do you have the error from Elasticsearch logs? It might explain if Knox or
> Grafana is sending a bad request.
>
> Kevin Risden
>
>
> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com>
> wrote:
>
>> I don't see any issues from Knox side, what error do you see in Grafana
>> ? I think this could be because Grafana is expecting certain content-type.
>>
>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com> wrote:
>>
>>> Hi Sandeep
>>>
>>> thanks for your feedback, below is the deboug log for this curl
>>>
>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>> <host-elast>:9200/<index_name>/_msearch
>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>> status: 400
>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>> character set UTF-8 for entity of type application/json
>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>> response entity content type: application/json; charset=UTF-8
>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>> /es5/<index_name>/_msearch
>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>
>>> Regards
>>> Rabii
>>>
>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com> a
>>> écrit :
>>>
>>>> Hello Rab,
>>>>
>>>> Interesting, do you have the debug logs for this ? they will tell us
>>>> exactly what is happening.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> From Grafana, I Post multisearch query to Elasticsearch using the bulk
>>>>> API with the newline delimited JSON (NDJSON) format.
>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>> like this:
>>>>>
>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>>> { first json doc}
>>>>> {second json doc}
>>>>> ’
>>>>> when bypassing Knox, The query work well either we made json or
>>>>> x-ndjson,
>>>>> However, through Knox gateway, if the content-type=application/json,
>>>>> the query retun this error of 400:
>>>>>
>>>>> HTTP/1.1 400 Bad Request
>>>>> content-type: application/json
>>>>>
>>>>> But when we replace json by x-ndjson like mentioned in the bulk API,
>>>>> it work well.
>>>>>
>>>>> My problem is that Grafana send their query with the
>>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>>> make any thing on Knox to solve this problem?
>>>>>
>>>>> Regards
>>>>> Rab
>>>>>
>>>>
Re: x-ndjson via Knox
Posted by Kevin Risden <kr...@apache.org>.
Do you have the error from Elasticsearch logs? It might explain if Knox or
Grafana is sending a bad request.
Kevin Risden
On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré <mo...@gmail.com> wrote:
> I don't see any issues from Knox side, what error do you see in Grafana ?
> I think this could be because Grafana is expecting certain content-type.
>
> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com> wrote:
>
>> Hi Sandeep
>>
>> thanks for your feedback, below is the deboug log for this curl
>>
>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>> <host-elast>:9200/<index_name>/_msearch
>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>> status: 400
>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>> character set UTF-8 for entity of type application/json
>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>> response entity content type: application/json; charset=UTF-8
>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>> /es5/<index_name>/_msearch
>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>
>> Regards
>> Rabii
>>
>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com> a
>> écrit :
>>
>>> Hello Rab,
>>>
>>> Interesting, do you have the debug logs for this ? they will tell us
>>> exactly what is happening.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> From Grafana, I Post multisearch query to Elasticsearch using the bulk
>>>> API with the newline delimited JSON (NDJSON) format.
>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>> like this:
>>>>
>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>> http://host_ES:900/index/_msearch" --data-binary ’
>>>> { first json doc}
>>>> {second json doc}
>>>> ’
>>>> when bypassing Knox, The query work well either we made json or
>>>> x-ndjson,
>>>> However, through Knox gateway, if the content-type=application/json,
>>>> the query retun this error of 400:
>>>>
>>>> HTTP/1.1 400 Bad Request
>>>> content-type: application/json
>>>>
>>>> But when we replace json by x-ndjson like mentioned in the bulk API, it
>>>> work well.
>>>>
>>>> My problem is that Grafana send their query with the
>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>> make any thing on Knox to solve this problem?
>>>>
>>>> Regards
>>>> Rab
>>>>
>>>
Re: x-ndjson via Knox
Posted by Sandeep Moré <mo...@gmail.com>.
I don't see any issues from Knox side, what error do you see in Grafana ?
I think this could be because Grafana is expecting certain content-type.
On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq <la...@gmail.com> wrote:
> Hi Sandeep
>
> thanks for your feedback, below is the deboug log for this curl
>
> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
> direction: IN via explicit rule: ES5/inbound/query to URL: http://
> <host-elast>:9200/<index_name>/_msearch
> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
> POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
> status: 400
> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
> character set UTF-8 for entity of type application/json
> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
> response entity content type: application/json; charset=UTF-8
> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
> (GatewayFilter.java:doFilter(116)) - Received request: POST
> /es5/<index_name>/_msearch
> 2018-09-18 17:23:54,521 INFO hadoop.gateway
> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>
> Regards
> Rabii
>
> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com> a
> écrit :
>
>> Hello Rab,
>>
>> Interesting, do you have the debug logs for this ? they will tell us
>> exactly what is happening.
>>
>> Best,
>> Sandeep
>>
>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> From Grafana, I Post multisearch query to Elasticsearch using the bulk
>>> API with the newline delimited JSON (NDJSON) format.
>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>> like this:
>>>
>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>> http://host_ES:900/index/_msearch" --data-binary ’
>>> { first json doc}
>>> {second json doc}
>>> ’
>>> when bypassing Knox, The query work well either we made json or x-ndjson,
>>> However, through Knox gateway, if the content-type=application/json, the
>>> query retun this error of 400:
>>>
>>> HTTP/1.1 400 Bad Request
>>> content-type: application/json
>>>
>>> But when we replace json by x-ndjson like mentioned in the bulk API, it
>>> work well.
>>>
>>> My problem is that Grafana send their query with the
>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>> make any thing on Knox to solve this problem?
>>>
>>> Regards
>>> Rab
>>>
>>
Re: x-ndjson via Knox
Posted by rabii lamriq <la...@gmail.com>.
Hi Sandeep
thanks for your feedback, below is the deboug log for this curl
2018-09-18 17:23:23,154 DEBUG hadoop.gateway
(UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
https://<host-knox>:8443/gateway/default/es5/<index_name>/_msearch,
direction: IN via explicit rule: ES5/inbound/query to URL: http://
<host-elast>:9200/<index_name>/_msearch
2018-09-18 17:23:23,156 DEBUG hadoop.gateway
(ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
POST http://<host-elast>:9200/<index_name>/_msearch?doAs=<u_user>
2018-09-18 17:23:23,193 DEBUG hadoop.gateway
(ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
status: 400
2018-09-18 17:23:23,194 DEBUG hadoop.gateway
(DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
character set UTF-8 for entity of type application/json
2018-09-18 17:23:23,195 DEBUG hadoop.gateway
(DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
response entity content type: application/json; charset=UTF-8
2018-09-18 17:23:54,519 DEBUG hadoop.gateway
(GatewayFilter.java:doFilter(116)) - Received request: POST
/es5/<index_name>/_msearch
2018-09-18 17:23:54,521 INFO hadoop.gateway
(AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
Regards
Rabii
Le ven. 21 sept. 2018 à 17:17, Sandeep Moré <mo...@gmail.com> a
écrit :
> Hello Rab,
>
> Interesting, do you have the debug logs for this ? they will tell us
> exactly what is happening.
>
> Best,
> Sandeep
>
> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com> wrote:
>
>> Hi All,
>>
>> From Grafana, I Post multisearch query to Elasticsearch using the bulk
>> API with the newline delimited JSON (NDJSON) format.
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>> like this:
>>
>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>> http://host_ES:900/index/_msearch" --data-binary ’
>> { first json doc}
>> {second json doc}
>> ’
>> when bypassing Knox, The query work well either we made json or x-ndjson,
>> However, through Knox gateway, if the content-type=application/json, the
>> query retun this error of 400:
>>
>> HTTP/1.1 400 Bad Request
>> content-type: application/json
>>
>> But when we replace json by x-ndjson like mentioned in the bulk API, it
>> work well.
>>
>> My problem is that Grafana send their query with the
>> content-type=application/json and not x-ndjson, for this, it is possible to
>> make any thing on Knox to solve this problem?
>>
>> Regards
>> Rab
>>
>
Re: x-ndjson via Knox
Posted by Sandeep Moré <mo...@gmail.com>.
Hello Rab,
Interesting, do you have the debug logs for this ? they will tell us
exactly what is happening.
Best,
Sandeep
On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq <la...@gmail.com> wrote:
> Hi All,
>
> From Grafana, I Post multisearch query to Elasticsearch using the bulk API
> with the newline delimited JSON (NDJSON) format.
> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
> like this:
>
> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
> http://host_ES:900/index/_msearch" --data-binary ’
> { first json doc}
> {second json doc}
> ’
> when bypassing Knox, The query work well either we made json or x-ndjson,
> However, through Knox gateway, if the content-type=application/json, the
> query retun this error of 400:
>
> HTTP/1.1 400 Bad Request
> content-type: application/json
>
> But when we replace json by x-ndjson like mentioned in the bulk API, it
> work well.
>
> My problem is that Grafana send their query with the
> content-type=application/json and not x-ndjson, for this, it is possible to
> make any thing on Knox to solve this problem?
>
> Regards
> Rab
>