You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by Theyaa Matti <th...@gmail.com> on 2018/10/09 16:01:27 UTC
Spark History UI Error WARN HttpParser: Header is too large >8192
Hi,
I am getting this error message "WARN HttpParser: Header is too large
>8192" when trying to access the spark history ui through knox. Any idea
please?
Also when trying to load the executors page, I get : GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))]
when knox is requesting executorspage-template.html
appreciate any help here.
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Sandeep Moré <mo...@gmail.com>.
You can try creating a new rewrite rule that drops that problematic header
if you don't need all those groups.
Best,
Sandeep
On Wed, Oct 10, 2018 at 7:01 PM Theyaa Matti <th...@gmail.com> wrote:
> Hi David, Kevin,
> I have reached out to the spark users community to see if their are
> alternatives or any work going on to fix this issue. One other thing I am
> trying to test out is the limiting the request header sent from Knox to the
> backend services, like spark history ui.
>
> The main reason this issue appear on our end is the large number of
> groups the user belongs to that is pushing the request header to go beyond
> 8192. Those groups are not needed in the backend services and I am looking
> to see if there is a way for Knox to exclude them when forwarding the
> request to the backend?
>
> Best,
>
> Theyaa.
>
>
> On Wed, Oct 10, 2018 at 2:26 PM Kevin Risden <kr...@apache.org> wrote:
>
>> Moving this back to the user list since it was sent without the list:
>>
>> Question: Is this error coming from the spark history server or the knox
>> server?
>>
>> Answer: It is coming from the spark history UI. Here is what I see in the
>> log for spark history ui.
>>
>> 18/10/10 12:30:29 WARN HttpParser: Header is too large >8192
>>
>> 18/10/10 12:30:29 WARN HttpParser: badMessage: 413 for
>> HttpChannelOverHttp@2756e900{r=1,c=false,a=IDLE,uri=/?doAs=<user>}
>>
>>
>> With the above information there is probably nothing that can be done
>> from Knox. The bad news is that there is no configuration in Spark for this
>> yet. See the following:
>> * https://issues.apache.org/jira/browse/SPARK-15090
>> *
>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L337
>> * Should see something about:
>> * connector.setRequestHeaderSize(requestHeaderSize);
>> * connector.setResponseHeaderSize(responseHeaderSize);
>>
>> Without a configuration to change, the Spark history UI is going to
>> default to the 8192 header size. A related change was done for most
>> projects to make that header size configurable.
>>
>> Kevin Risden
>>
>>
>> On Tue, Oct 9, 2018 at 8:45 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> Hi David,
>>> Thank you for the quick response. I do have that property set
>>> with the recommended value in Knox, but the issue still persists only with
>>> the Spark History UI. The issue appears only when I enable ssl for knox. If
>>> I turn ssl off in Knox the issue disappears, and if I enable it, it shows
>>> up right away.
>>>
>>> Are there any equivalent properties for Spark History UI? since
>>> I suspect the issue is with the Spark History UI and not with Knox since
>>> all the other UIs work fine with Knox.
>>>
>>> Best,
>>>
>>> Theyaa.
>>>
>>>
>>>
>>> On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <
>>> dvillarreal@hortonworks.com> wrote:
>>>
>>>> Hi Theyaa,
>>>>
>>>> Change the size of gateway.httpserver.requestHeaderBuffer property. I
>>>> think the default is 8912 (8k) change to 16384. See if that helps.
>>>>
>>>> For the second problem Request is a replay (34))] this message is often
>>>> seen when the timing of one of the servers is off. Make sure you use NTPD
>>>> on all servers and they are all in sync. If everything is in sync you can
>>>> work around this issue by turning off krb5 replay cache. With the following
>>>> parameter
>>>> -Dsun.security.krb5.rcache=none
>>>>
>>>> dav
>>>>
>>>>
>>>> On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> I am getting this error message "WARN HttpParser: Header is too
>>>> large
>>>> >8192" when trying to access the spark history ui through knox. Any
>>>> idea
>>>> please?
>>>>
>>>> Also when trying to load the executors page, I get : GSS initiate
>>>> failed
>>>> [Caused by GSSException: Failure unspecified at GSS-API level
>>>> (Mechanism level:
>>>> Request is a replay (34))]
>>>>
>>>> when knox is requesting executorspage-template.html
>>>>
>>>> appreciate any help here.
>>>>
>>>>
>>>>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Theyaa Matti <th...@gmail.com>.
Hi David, Kevin,
I have reached out to the spark users community to see if their are
alternatives or any work going on to fix this issue. One other thing I am
trying to test out is the limiting the request header sent from Knox to the
backend services, like spark history ui.
The main reason this issue appear on our end is the large number of
groups the user belongs to that is pushing the request header to go beyond
8192. Those groups are not needed in the backend services and I am looking
to see if there is a way for Knox to exclude them when forwarding the
request to the backend?
Best,
Theyaa.
On Wed, Oct 10, 2018 at 2:26 PM Kevin Risden <kr...@apache.org> wrote:
> Moving this back to the user list since it was sent without the list:
>
> Question: Is this error coming from the spark history server or the knox
> server?
>
> Answer: It is coming from the spark history UI. Here is what I see in the
> log for spark history ui.
>
> 18/10/10 12:30:29 WARN HttpParser: Header is too large >8192
>
> 18/10/10 12:30:29 WARN HttpParser: badMessage: 413 for
> HttpChannelOverHttp@2756e900{r=1,c=false,a=IDLE,uri=/?doAs=<user>}
>
>
> With the above information there is probably nothing that can be done from
> Knox. The bad news is that there is no configuration in Spark for this yet.
> See the following:
> * https://issues.apache.org/jira/browse/SPARK-15090
> *
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L337
> * Should see something about:
> * connector.setRequestHeaderSize(requestHeaderSize);
> * connector.setResponseHeaderSize(responseHeaderSize);
>
> Without a configuration to change, the Spark history UI is going to
> default to the 8192 header size. A related change was done for most
> projects to make that header size configurable.
>
> Kevin Risden
>
>
> On Tue, Oct 9, 2018 at 8:45 PM Theyaa Matti <th...@gmail.com> wrote:
>
>> Hi David,
>> Thank you for the quick response. I do have that property set
>> with the recommended value in Knox, but the issue still persists only with
>> the Spark History UI. The issue appears only when I enable ssl for knox. If
>> I turn ssl off in Knox the issue disappears, and if I enable it, it shows
>> up right away.
>>
>> Are there any equivalent properties for Spark History UI? since
>> I suspect the issue is with the Spark History UI and not with Knox since
>> all the other UIs work fine with Knox.
>>
>> Best,
>>
>> Theyaa.
>>
>>
>>
>> On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>>> Hi Theyaa,
>>>
>>> Change the size of gateway.httpserver.requestHeaderBuffer property. I
>>> think the default is 8912 (8k) change to 16384. See if that helps.
>>>
>>> For the second problem Request is a replay (34))] this message is often
>>> seen when the timing of one of the servers is off. Make sure you use NTPD
>>> on all servers and they are all in sync. If everything is in sync you can
>>> work around this issue by turning off krb5 replay cache. With the following
>>> parameter
>>> -Dsun.security.krb5.rcache=none
>>>
>>> dav
>>>
>>>
>>> On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
>>>
>>> Hi,
>>> I am getting this error message "WARN HttpParser: Header is too
>>> large
>>> >8192" when trying to access the spark history ui through knox. Any
>>> idea
>>> please?
>>>
>>> Also when trying to load the executors page, I get : GSS initiate
>>> failed
>>> [Caused by GSSException: Failure unspecified at GSS-API level
>>> (Mechanism level:
>>> Request is a replay (34))]
>>>
>>> when knox is requesting executorspage-template.html
>>>
>>> appreciate any help here.
>>>
>>>
>>>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Kevin Risden <kr...@apache.org>.
Moving this back to the user list since it was sent without the list:
Question: Is this error coming from the spark history server or the knox
server?
Answer: It is coming from the spark history UI. Here is what I see in the
log for spark history ui.
18/10/10 12:30:29 WARN HttpParser: Header is too large >8192
18/10/10 12:30:29 WARN HttpParser: badMessage: 413 for
HttpChannelOverHttp@2756e900{r=1,c=false,a=IDLE,uri=/?doAs=<user>}
With the above information there is probably nothing that can be done from
Knox. The bad news is that there is no configuration in Spark for this yet.
See the following:
* https://issues.apache.org/jira/browse/SPARK-15090
*
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L337
* Should see something about:
* connector.setRequestHeaderSize(requestHeaderSize);
* connector.setResponseHeaderSize(responseHeaderSize);
Without a configuration to change, the Spark history UI is going to default
to the 8192 header size. A related change was done for most projects to
make that header size configurable.
Kevin Risden
On Tue, Oct 9, 2018 at 8:45 PM Theyaa Matti <th...@gmail.com> wrote:
> Hi David,
> Thank you for the quick response. I do have that property set
> with the recommended value in Knox, but the issue still persists only with
> the Spark History UI. The issue appears only when I enable ssl for knox. If
> I turn ssl off in Knox the issue disappears, and if I enable it, it shows
> up right away.
>
> Are there any equivalent properties for Spark History UI? since I
> suspect the issue is with the Spark History UI and not with Knox since all
> the other UIs work fine with Knox.
>
> Best,
>
> Theyaa.
>
>
>
> On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
>> Hi Theyaa,
>>
>> Change the size of gateway.httpserver.requestHeaderBuffer property. I
>> think the default is 8912 (8k) change to 16384. See if that helps.
>>
>> For the second problem Request is a replay (34))] this message is often
>> seen when the timing of one of the servers is off. Make sure you use NTPD
>> on all servers and they are all in sync. If everything is in sync you can
>> work around this issue by turning off krb5 replay cache. With the following
>> parameter
>> -Dsun.security.krb5.rcache=none
>>
>> dav
>>
>>
>> On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
>>
>> Hi,
>> I am getting this error message "WARN HttpParser: Header is too
>> large
>> >8192" when trying to access the spark history ui through knox. Any
>> idea
>> please?
>>
>> Also when trying to load the executors page, I get : GSS initiate
>> failed
>> [Caused by GSSException: Failure unspecified at GSS-API level
>> (Mechanism level:
>> Request is a replay (34))]
>>
>> when knox is requesting executorspage-template.html
>>
>> appreciate any help here.
>>
>>
>>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Theyaa Matti <th...@gmail.com>.
Hi David,
Thank you for the quick response. I do have that property set with
the recommended value in Knox, but the issue still persists only with the
Spark History UI. The issue appears only when I enable ssl for knox. If I
turn ssl off in Knox the issue disappears, and if I enable it, it shows up
right away.
Are there any equivalent properties for Spark History UI? since I
suspect the issue is with the Spark History UI and not with Knox since all
the other UIs work fine with Knox.
Best,
Theyaa.
On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <dv...@hortonworks.com>
wrote:
> Hi Theyaa,
>
> Change the size of gateway.httpserver.requestHeaderBuffer property. I
> think the default is 8912 (8k) change to 16384. See if that helps.
>
> For the second problem Request is a replay (34))] this message is often
> seen when the timing of one of the servers is off. Make sure you use NTPD
> on all servers and they are all in sync. If everything is in sync you can
> work around this issue by turning off krb5 replay cache. With the following
> parameter
> -Dsun.security.krb5.rcache=none
>
> dav
>
>
> On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
>
> Hi,
> I am getting this error message "WARN HttpParser: Header is too
> large
> >8192" when trying to access the spark history ui through knox. Any
> idea
> please?
>
> Also when trying to load the executors page, I get : GSS initiate
> failed
> [Caused by GSSException: Failure unspecified at GSS-API level
> (Mechanism level:
> Request is a replay (34))]
>
> when knox is requesting executorspage-template.html
>
> appreciate any help here.
>
>
>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Theyaa Matti <th...@gmail.com>.
Hi David,
Thank you for the quick response. I do have that property set with
the recommended value in Knox, but the issue still persists only with the
Spark History UI. The issue appears only when I enable ssl for knox. If I
turn ssl off in Knox the issue disappears, and if I enable it, it shows up
right away.
Are there any equivalent properties for Spark History UI? since I
suspect the issue is with the Spark History UI and not with Knox since all
the other UIs work fine with Knox.
Best,
Theyaa.
On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <dv...@hortonworks.com>
wrote:
> Hi Theyaa,
>
> Change the size of gateway.httpserver.requestHeaderBuffer property. I
> think the default is 8912 (8k) change to 16384. See if that helps.
>
> For the second problem Request is a replay (34))] this message is often
> seen when the timing of one of the servers is off. Make sure you use NTPD
> on all servers and they are all in sync. If everything is in sync you can
> work around this issue by turning off krb5 replay cache. With the following
> parameter
> -Dsun.security.krb5.rcache=none
>
> dav
>
>
> On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
>
> Hi,
> I am getting this error message "WARN HttpParser: Header is too
> large
> >8192" when trying to access the spark history ui through knox. Any
> idea
> please?
>
> Also when trying to load the executors page, I get : GSS initiate
> failed
> [Caused by GSSException: Failure unspecified at GSS-API level
> (Mechanism level:
> Request is a replay (34))]
>
> when knox is requesting executorspage-template.html
>
> appreciate any help here.
>
>
>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by David Villarreal <dv...@hortonworks.com>.
Hi Theyaa,
Change the size of gateway.httpserver.requestHeaderBuffer property. I think the default is 8912 (8k) change to 16384. See if that helps.
For the second problem Request is a replay (34))] this message is often seen when the timing of one of the servers is off. Make sure you use NTPD on all servers and they are all in sync. If everything is in sync you can work around this issue by turning off krb5 replay cache. With the following parameter
-Dsun.security.krb5.rcache=none
dav
On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
Hi,
I am getting this error message "WARN HttpParser: Header is too large
>8192" when trying to access the spark history ui through knox. Any idea
please?
Also when trying to load the executors page, I get : GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))]
when knox is requesting executorspage-template.html
appreciate any help here.
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Theyaa Matti <th...@gmail.com>.
Apologies Kevin, I will make sure to send it only to the user group and
thank you for the quick response. I know about the hive configuration but I
do not see similar ones for the spark side. I did apply the knox ones but
did not work since I believe the issue is on the spark side. Any pointers
would be appreciated.
Best
On Tue, Oct 9, 2018 at 12:35 PM Kevin Risden <kr...@apache.org> wrote:
> The issue for the header could be on the Spark side as well.
>
> Some more background details:
> https://risdenk.github.io/2018/03/04/http-413-full-head-kerberos-spnego-authentication.html
>
> Kevin Risden
>
>
> On Tue, Oct 9, 2018 at 12:32 PM Kevin Risden <kr...@apache.org> wrote:
>
>> Please don't send to two different mailing lists at the same time.
>> Removed the dev list.
>>
>> It sounds like you are describing the same issue as KNOX-624 [1]. The two
>> configurations are:
>>
>> gateway.httpserver.requestHeaderBuffer
>> gateway.httpserver.responseHeaderBuffer
>>
>> 1. https://issues.apache.org/jira/browse/KNOX-624
>>
>> Kevin Risden
>>
>>
>> On Tue, Oct 9, 2018 at 12:01 PM Theyaa Matti <th...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am getting this error message "WARN HttpParser: Header is too
>>> large >8192" when trying to access the spark history ui through knox. Any
>>> idea please?
>>>
>>> Also when trying to load the executors page, I get : GSS initiate
>>> failed [Caused by GSSException: Failure unspecified at GSS-API level
>>> (Mechanism level: Request is a replay (34))]
>>>
>>> when knox is requesting executorspage-template.html
>>>
>>> appreciate any help here.
>>>
>>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Kevin Risden <kr...@apache.org>.
The issue for the header could be on the Spark side as well.
Some more background details:
https://risdenk.github.io/2018/03/04/http-413-full-head-kerberos-spnego-authentication.html
Kevin Risden
On Tue, Oct 9, 2018 at 12:32 PM Kevin Risden <kr...@apache.org> wrote:
> Please don't send to two different mailing lists at the same time. Removed
> the dev list.
>
> It sounds like you are describing the same issue as KNOX-624 [1]. The two
> configurations are:
>
> gateway.httpserver.requestHeaderBuffer
> gateway.httpserver.responseHeaderBuffer
>
> 1. https://issues.apache.org/jira/browse/KNOX-624
>
> Kevin Risden
>
>
> On Tue, Oct 9, 2018 at 12:01 PM Theyaa Matti <th...@gmail.com>
> wrote:
>
>> Hi,
>> I am getting this error message "WARN HttpParser: Header is too large
>> >8192" when trying to access the spark history ui through knox. Any idea
>> please?
>>
>> Also when trying to load the executors page, I get : GSS initiate failed
>> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level:
>> Request is a replay (34))]
>>
>> when knox is requesting executorspage-template.html
>>
>> appreciate any help here.
>>
>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by Kevin Risden <kr...@apache.org>.
Please don't send to two different mailing lists at the same time. Removed
the dev list.
It sounds like you are describing the same issue as KNOX-624 [1]. The two
configurations are:
gateway.httpserver.requestHeaderBuffer
gateway.httpserver.responseHeaderBuffer
1. https://issues.apache.org/jira/browse/KNOX-624
Kevin Risden
On Tue, Oct 9, 2018 at 12:01 PM Theyaa Matti <th...@gmail.com> wrote:
> Hi,
> I am getting this error message "WARN HttpParser: Header is too large
> >8192" when trying to access the spark history ui through knox. Any idea
> please?
>
> Also when trying to load the executors page, I get : GSS initiate failed
> [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level:
> Request is a replay (34))]
>
> when knox is requesting executorspage-template.html
>
> appreciate any help here.
>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Posted by David Villarreal <dv...@hortonworks.com>.
Hi Theyaa,
Change the size of gateway.httpserver.requestHeaderBuffer property. I think the default is 8912 (8k) change to 16384. See if that helps.
For the second problem Request is a replay (34))] this message is often seen when the timing of one of the servers is off. Make sure you use NTPD on all servers and they are all in sync. If everything is in sync you can work around this issue by turning off krb5 replay cache. With the following parameter
-Dsun.security.krb5.rcache=none
dav
On 10/9/18, 9:01 AM, "Theyaa Matti" <th...@gmail.com> wrote:
Hi,
I am getting this error message "WARN HttpParser: Header is too large
>8192" when trying to access the spark history ui through knox. Any idea
please?
Also when trying to load the executors page, I get : GSS initiate failed
[Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))]
when knox is requesting executorspage-template.html
appreciate any help here.