You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Nicolae Popa (JIRA)" <ji...@apache.org> on 2016/06/07 07:54:21 UTC
[jira] [Commented] (HDFS-10423) Increase default value of httpfs
maxHttpHeaderSize
[ https://issues.apache.org/jira/browse/HDFS-10423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318079#comment-15318079 ]
Nicolae Popa commented on HDFS-10423:
-------------------------------------
I have attached the tested patch. The test is using the below script to generate a bloat header for http using the below ad hoc script:
It will generate an http request with various sizes to stress the tomcat settings.
for size in 1024 2048 4096 8192 8193 16384 32768 44443 44444 44445 65536 65537 ; do
echo "################################################################################################################################################################"
echo "### Trying with request size: $size"
echo "################################################################################################################################################################"
echo
http_request_size=$size
cat <<-EOF>http_request
GET /webhdfs/v1/user/npopa?op=liststatus&user.name=npopa HTTP/1.1
Host: localhost:14000
EOF
http_request_size_tmp=`wc -c http_request|awk '{print $1}'`
export to_fill=`expr $http_request_size - $http_request_size_tmp - 5` # -5 comes from "h: " and new lines before and after the header
export RNDSTR=`cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w $to_fill| head -n 1`
echo "h: $RNDSTR">>http_request
echo "">>http_request
echo "Request is: "
cat http_request
echo
echo "Request size is: "
ls -ltr http_request
echo
echo "Result is: "
nc localhost 14000 < http_request
echo
done
The scenarios and results are:
1) before the patch
2) after the patch
3) with the export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh
In before-HTTP-10423.txt we can see it starts failing after a HTTP 8192 request
In after-HTTP-10423.txt we can see it starts failing after a HTTP 65536 request (as expected)
I also run a test setting export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh and the behavior was as expected. It worked with an HTTP request of 44444 bytes and failed with a request of 44445 bytes (as expected). (see after-HDFS-10423_withCustomHeader44444.txt)
For each size below are the expected results:
successful result is:
Result is:
HTTP/1.1 200 OK
failed result is:
Result is:
HTTP/1.1 400 Bad Request
> Increase default value of httpfs maxHttpHeaderSize
> --------------------------------------------------
>
> Key: HDFS-10423
> URL: https://issues.apache.org/jira/browse/HDFS-10423
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.6.4
> Reporter: Nicolae Popa
> Priority: Minor
> Attachments: HDFS-10423.01.patch, after-HDFS-10423.txt, after-HDFS-10423_withCustomHeader44444.txt, before-HDFS-10423.txt
>
>
> The Tomcat default value of maxHttpHeaderSize is 8k, which is too low for certain Hadoop workloads in kerberos enabled environments. This JIRA will to change it to 65536 in server.xml
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org