You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jan Kaluža <jk...@redhat.com> on 2016/01/08 08:49:37 UTC
mod_mime_magic, gzipped tarballs and Docker
Hi,
it seems Docker client has a problem handling httpd responses [1] when
you run Docker server behind httpd working as a reverse proxy. It is
caused by "mod_mime_magic" adding following Content-Type and
Content-Encoding to gzipped tarballs sent as a response by Docker server:
Content-Type: application/x-tar
Content-Encoding: x-gzip
Docker client expects gzipped tarballs to be received, but because it
receives Content-Encoding: x-gzip, it decompresses the response and
receives just plain tar (at least that's how I understand it).
Strictly speaking, httpd is probably right here according to RFC:
"""
The Content-Encoding entity-header field is used as a modifier to the
media-type. When present, its value indicates what additional content
codings have been applied to the entity-body, and thus what decoding
mechanisms must be applied in order to obtain the media-type referenced
by the Content-Type header field.
"""
So, the mod_mime_magic is saying here that the body is tarball encoded
by gzip.
But I think even we are right here, the Content-Encoding should be used
only when httpd itself encodes the original content, otherwise it
confuses clients (I've asked few people and it confuses some web
browsers too - I can get more info if needed.).
Maybe we could stop setting Content-Encoding in mod_mime_magic and just
use Content-Type?
[1] https://github.com/docker/docker/issues/17595
Regards,
Jan Kaluza
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Ronald Masaya <ro...@gmail.com>.
Accept
On Jan 19, 2016 4:22 AM, "William A Rowe Jr" <wr...@rowe-clan.net> wrote:
> On Mon, Jan 18, 2016 at 5:13 AM, Jan Kaluža <jk...@redhat.com> wrote:
>
>> On 01/08/2016 07:44 PM, William A Rowe Jr wrote:
>>
>>> Do we have to repeat the softmagic call if checkzmagic resolves to
>>> x-gzip/x-deflate and the internal content type needs to be deciphered?
>>>
>>
>> That's true.
>>
>> I think that Yann's patch moving the zmagic call after the softmagic call
>> would just mean that zmagic won't be called at all if the softmagic
>> recognizes the file format.
>>
>> We would have to check the softmagic result by something similar to
>> "magic_rsl_to_request" and if it's type we want to decompress, we would
>> have to run zmagic.
>>
>> Before really trying to do so, I want to ask if I understand the
>> reasoning right. Do we consider this way because users can then remove
>> x-gzip from mime magic and will be able to use it to disable the
>> mod_mime_magic behaviour discussed in this thread?
>>
>
> Yes... and in a more flexible manner that allows us to override any
> compression mode, not only deflate, by simply tweaking the magic entries.
>
> Putting magic file entry overrides into the mod_mime_magic directives is a
> lot more interesting than simply toggling compression recognition.
>
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jan Kaluža <jk...@redhat.com>.
On 01/18/2016 09:22 PM, William A Rowe Jr wrote:
> On Mon, Jan 18, 2016 at 5:13 AM, Jan Kaluža <jkaluza@redhat.com
> <ma...@redhat.com>> wrote:
>
> On 01/08/2016 07:44 PM, William A Rowe Jr wrote:
>
> Do we have to repeat the softmagic call if checkzmagic resolves to
> x-gzip/x-deflate and the internal content type needs to be
> deciphered?
>
>
> That's true.
>
> I think that Yann's patch moving the zmagic call after the softmagic
> call would just mean that zmagic won't be called at all if the
> softmagic recognizes the file format.
>
> We would have to check the softmagic result by something similar to
> "magic_rsl_to_request" and if it's type we want to decompress, we
> would have to run zmagic.
>
> Before really trying to do so, I want to ask if I understand the
> reasoning right. Do we consider this way because users can then
> remove x-gzip from mime magic and will be able to use it to disable
> the mod_mime_magic behaviour discussed in this thread?
>
>
> Yes... and in a more flexible manner that allows us to override any
> compression mode, not only deflate, by simply tweaking the magic entries.
>
> Putting magic file entry overrides into the mod_mime_magic directives is
> a lot more interesting than simply toggling compression recognition.
Okay... I have something half-way right now.
Attached patch changes the order of softmagic and zmagic. When softmagic
recognizes a file, the zmagic is called. zmagic method checks if the
file has been recognized as "x-gzip" and it tries to uncompress it and
call "tryit" again with the uncompressed data as before.
I'm now playing with an idea to extend the magic file so we wouldn't
have to keep the "x-gzip" -> "gzip -dcq" relation hardcoded, but it
could be defined in the magic file instead.
William, is the patch close to the behaviour you were describing?
Regards,
Jan Kaluza
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Mon, Jan 18, 2016 at 5:13 AM, Jan Kaluža <jk...@redhat.com> wrote:
> On 01/08/2016 07:44 PM, William A Rowe Jr wrote:
>
>> Do we have to repeat the softmagic call if checkzmagic resolves to
>> x-gzip/x-deflate and the internal content type needs to be deciphered?
>>
>
> That's true.
>
> I think that Yann's patch moving the zmagic call after the softmagic call
> would just mean that zmagic won't be called at all if the softmagic
> recognizes the file format.
>
> We would have to check the softmagic result by something similar to
> "magic_rsl_to_request" and if it's type we want to decompress, we would
> have to run zmagic.
>
> Before really trying to do so, I want to ask if I understand the reasoning
> right. Do we consider this way because users can then remove x-gzip from
> mime magic and will be able to use it to disable the mod_mime_magic
> behaviour discussed in this thread?
>
Yes... and in a more flexible manner that allows us to override any
compression mode, not only deflate, by simply tweaking the magic entries.
Putting magic file entry overrides into the mod_mime_magic directives is a
lot more interesting than simply toggling compression recognition.
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jan Kaluža <jk...@redhat.com>.
On 01/08/2016 07:44 PM, William A Rowe Jr wrote:
> Do we have to repeat the softmagic call if checkzmagic resolves to
> x-gzip/x-deflate and the internal content type needs to be deciphered?
That's true.
I think that Yann's patch moving the zmagic call after the softmagic
call would just mean that zmagic won't be called at all if the softmagic
recognizes the file format.
We would have to check the softmagic result by something similar to
"magic_rsl_to_request" and if it's type we want to decompress, we would
have to run zmagic.
Before really trying to do so, I want to ask if I understand the
reasoning right. Do we consider this way because users can then remove
x-gzip from mime magic and will be able to use it to disable the
mod_mime_magic behaviour discussed in this thread?
Regards,
Jan Kaluza
> If so, a tweak to this patch sounds like a winner, and could selectively
> recognize different inflation streams including bzip2 etc.
>
> On Fri, Jan 8, 2016 at 10:57 AM, Yann Ylavic <ylavic.dev@gmail.com
> <ma...@gmail.com>> wrote:
>
> On Fri, Jan 8, 2016 at 5:30 PM, Yann Ylavic <ylavic.dev@gmail.com
> <ma...@gmail.com>> wrote:
> > On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr <wrowe@rowe-clan.net <ma...@rowe-clan.net>> wrote:
> >>
> >> Agreed it is configuration, but cant we simply tweak our recommended
> >> conf/magic
> >> file???
> >>
> >> # standard unix compress
> >> # Enable the alternate line below to present gzip content as a transfer
> >> encoded
> >> # stream of the underlying content;
> >> #0 string \037\235 application/octet-stream
> >> x-compress
> >> 0 string \037\235 application/octet-stream
> >>
> >> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
> >> # Enable the alternate line below to present gzip content as a transfer
> >> encoded
> >> # stream of the underlying content;
> >> #0 string \037\213 application/octet-stream
> >> x-gzip
> >> 0 string \037\213 application/octet-stream
> >>
> >> WDYT?
> >
> > I wasn't aware of conf/magic file, but it seems not used by zmagic()
> > which hardcodes its types.
> > I may be missing something though...
>
> Looks like it is already like this in conf/magic of 2.4.x and 2.2.x.
> My bet is that zmagic() is run before softmagic(), so the magic file
> is of no help.
>
> Maybe a simpler patch could be:
>
> Index: modules/metadata/mod_mime_magic.c
> ===================================================================
> --- modules/metadata/mod_mime_magic.c (revision 1723283)
> +++ modules/metadata/mod_mime_magic.c (working copy)
> @@ -880,14 +885,6 @@ static int tryit(request_rec *r, unsigned char *bu
> int checkzmagic)
> {
> /*
> - * Try compression stuff
> - */
> - if (checkzmagic == 1) {
> - if (zmagic(r, buf, nb) == 1)
> - return OK;
> - }
> -
> - /*
> * try tests in /etc/magic (or surrogate magic file)
> */
> if (softmagic(r, buf, nb) == 1)
> @@ -900,6 +897,14 @@ static int tryit(request_rec *r, unsigned char *bu
> return OK;
>
> /*
> + * Try compression stuff
> + */
> + if (checkzmagic == 1) {
> + if (zmagic(r, buf, nb) == 1)
> + return OK;
> + }
> +
> + /*
> * abandon hope, all ye who remain here
> */
> return DECLINED;
> --
>
>
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
Do we have to repeat the softmagic call if checkzmagic resolves to
x-gzip/x-deflate and the internal content type needs to be deciphered?
If so, a tweak to this patch sounds like a winner, and could selectively
recognize different inflation streams including bzip2 etc.
On Fri, Jan 8, 2016 at 10:57 AM, Yann Ylavic <yl...@gmail.com> wrote:
> On Fri, Jan 8, 2016 at 5:30 PM, Yann Ylavic <yl...@gmail.com> wrote:
> > On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr <wr...@rowe-clan.net>
> wrote:
> >>
> >> Agreed it is configuration, but cant we simply tweak our recommended
> >> conf/magic
> >> file???
> >>
> >> # standard unix compress
> >> # Enable the alternate line below to present gzip content as a transfer
> >> encoded
> >> # stream of the underlying content;
> >> #0 string \037\235 application/octet-stream
> >> x-compress
> >> 0 string \037\235 application/octet-stream
> >>
> >> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
> >> # Enable the alternate line below to present gzip content as a transfer
> >> encoded
> >> # stream of the underlying content;
> >> #0 string \037\213 application/octet-stream
> >> x-gzip
> >> 0 string \037\213 application/octet-stream
> >>
> >> WDYT?
> >
> > I wasn't aware of conf/magic file, but it seems not used by zmagic()
> > which hardcodes its types.
> > I may be missing something though...
>
> Looks like it is already like this in conf/magic of 2.4.x and 2.2.x.
> My bet is that zmagic() is run before softmagic(), so the magic file
> is of no help.
>
> Maybe a simpler patch could be:
>
> Index: modules/metadata/mod_mime_magic.c
> ===================================================================
> --- modules/metadata/mod_mime_magic.c (revision 1723283)
> +++ modules/metadata/mod_mime_magic.c (working copy)
> @@ -880,14 +885,6 @@ static int tryit(request_rec *r, unsigned char *bu
> int checkzmagic)
> {
> /*
> - * Try compression stuff
> - */
> - if (checkzmagic == 1) {
> - if (zmagic(r, buf, nb) == 1)
> - return OK;
> - }
> -
> - /*
> * try tests in /etc/magic (or surrogate magic file)
> */
> if (softmagic(r, buf, nb) == 1)
> @@ -900,6 +897,14 @@ static int tryit(request_rec *r, unsigned char *bu
> return OK;
>
> /*
> + * Try compression stuff
> + */
> + if (checkzmagic == 1) {
> + if (zmagic(r, buf, nb) == 1)
> + return OK;
> + }
> +
> + /*
> * abandon hope, all ye who remain here
> */
> return DECLINED;
> --
>
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Jan 8, 2016 at 5:30 PM, Yann Ylavic <yl...@gmail.com> wrote:
> On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>>
>> Agreed it is configuration, but cant we simply tweak our recommended
>> conf/magic
>> file???
>>
>> # standard unix compress
>> # Enable the alternate line below to present gzip content as a transfer
>> encoded
>> # stream of the underlying content;
>> #0 string \037\235 application/octet-stream
>> x-compress
>> 0 string \037\235 application/octet-stream
>>
>> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
>> # Enable the alternate line below to present gzip content as a transfer
>> encoded
>> # stream of the underlying content;
>> #0 string \037\213 application/octet-stream
>> x-gzip
>> 0 string \037\213 application/octet-stream
>>
>> WDYT?
>
> I wasn't aware of conf/magic file, but it seems not used by zmagic()
> which hardcodes its types.
> I may be missing something though...
Looks like it is already like this in conf/magic of 2.4.x and 2.2.x.
My bet is that zmagic() is run before softmagic(), so the magic file
is of no help.
Maybe a simpler patch could be:
Index: modules/metadata/mod_mime_magic.c
===================================================================
--- modules/metadata/mod_mime_magic.c (revision 1723283)
+++ modules/metadata/mod_mime_magic.c (working copy)
@@ -880,14 +885,6 @@ static int tryit(request_rec *r, unsigned char *bu
int checkzmagic)
{
/*
- * Try compression stuff
- */
- if (checkzmagic == 1) {
- if (zmagic(r, buf, nb) == 1)
- return OK;
- }
-
- /*
* try tests in /etc/magic (or surrogate magic file)
*/
if (softmagic(r, buf, nb) == 1)
@@ -900,6 +897,14 @@ static int tryit(request_rec *r, unsigned char *bu
return OK;
/*
+ * Try compression stuff
+ */
+ if (checkzmagic == 1) {
+ if (zmagic(r, buf, nb) == 1)
+ return OK;
+ }
+
+ /*
* abandon hope, all ye who remain here
*/
return DECLINED;
--
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Jan 8, 2016 at 3:17 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
> On Fri, Jan 8, 2016 at 3:27 AM, Yann Ylavic <yl...@gmail.com> wrote:
>>
>>
>> On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža <jk...@redhat.com> wrote:
>> >
>> > Content-Type: application/x-tar
>> > Content-Encoding: x-gzip
>> >
>> > So, the mod_mime_magic is saying here that the body is tarball encoded
>> > by
>> > gzip.
>>
>> AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
>> the above headers.
>
>
> Well, not strictly true, mod_deflate uncompresses it ;-)
Hmm, not sure mod_deflate is in the place, at least it doesn't need to
AFAICT from the code.
But sure mod_deflate could be configured to uncompress once more :)
>
>> > Maybe we could stop setting Content-Encoding in mod_mime_magic and just
>> > use
>> > Content-Type?
>>
>> Since it's always been there, we probably should add a new
>> mod_mime_magic directive to control the behaviour and avoid breaking
>> cases.
>
>
> Agreed it is configuration, but cant we simply tweak our recommended
> conf/magic
> file???
>
> # standard unix compress
> # Enable the alternate line below to present gzip content as a transfer
> encoded
> # stream of the underlying content;
> #0 string \037\235 application/octet-stream
> x-compress
> 0 string \037\235 application/octet-stream
>
> # gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
> # Enable the alternate line below to present gzip content as a transfer
> encoded
> # stream of the underlying content;
> #0 string \037\213 application/octet-stream
> x-gzip
> 0 string \037\213 application/octet-stream
>
> WDYT?
I wasn't aware of conf/magic file, but it seems not used by zmagic()
which hardcodes its types.
I may be missing something though...
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jim Jagielski <ji...@jaguNET.com>.
Alternatively, it could use a global extern var and avoid the dup.
> On Jan 8, 2016, at 3:00 PM, Jim Jagielski <ji...@jaguNET.com> wrote:
>
> Looks right to me... using the address of a local var is nasty
> and broken and for sure will cause grief.
>
>> On Jan 8, 2016, at 2:30 PM, Christophe JAILLET <ch...@wanadoo.fr> wrote:
>>
>> Hi,
>>
>> not directly related to this discussion, but while looking at mod_mime_magic, could someone have a look at:
>> http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1491700
>>
>>
>> I also tried to get some feedback some time ago (http://marc.info/?l=apache-httpd-dev&m=137106479314698) but at that time, got no success.
>>
>> The fix looks obvious to me but I never took enough time to test it.
>> Any one interested?
>>
>> Best regards,
>> CJ
>>
>>
>
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jim Jagielski <ji...@jaguNET.com>.
Looks right to me... using the address of a local var is nasty
and broken and for sure will cause grief.
> On Jan 8, 2016, at 2:30 PM, Christophe JAILLET <ch...@wanadoo.fr> wrote:
>
> Hi,
>
> not directly related to this discussion, but while looking at mod_mime_magic, could someone have a look at:
> http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1491700
>
>
> I also tried to get some feedback some time ago (http://marc.info/?l=apache-httpd-dev&m=137106479314698) but at that time, got no success.
>
> The fix looks obvious to me but I never took enough time to test it.
> Any one interested?
>
> Best regards,
> CJ
>
>
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Christophe JAILLET <ch...@wanadoo.fr>.
Hi,
not directly related to this discussion, but while looking at
mod_mime_magic, could someone have a look at:
http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1491700
I also tried to get some feedback some time ago
(http://marc.info/?l=apache-httpd-dev&m=137106479314698) but at that
time, got no success.
The fix looks obvious to me but I never took enough time to test it.
Any one interested?
Best regards,
CJ
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Fri, Jan 8, 2016 at 3:27 AM, Yann Ylavic <yl...@gmail.com> wrote:
>
> On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža <jk...@redhat.com> wrote:
> >
> > Content-Type: application/x-tar
> > Content-Encoding: x-gzip
> >
> > So, the mod_mime_magic is saying here that the body is tarball encoded by
> > gzip.
>
> AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
> the above headers.
>
Well, not strictly true, mod_deflate uncompresses it ;-)
> But I think even we are right here, the Content-Encoding should be used
> only
> > when httpd itself encodes the original content, otherwise it confuses
> > clients (I've asked few people and it confuses some web browsers too - I
> can
> > get more info if needed.).
>
> Agreed, this looks wrong, the browsers will likely use the .tar file
> instead (this adds unnecessary cycles to both the server and client).
>
I've observed this on many clients downloading .tar.gz and storing the file
as the
decompressed .tar when I had configured .gz to report Content-Encoding:
x-gzip,
and this is simply using mod_mime configuration directives. But that was
shooting
myself in the foot, had not stopped to consider what mod_mime_magic might
do.
> Maybe we could stop setting Content-Encoding in mod_mime_magic and just
> use
> > Content-Type?
>
> Since it's always been there, we probably should add a new
> mod_mime_magic directive to control the behaviour and avoid breaking
> cases.
>
Agreed it is configuration, but cant we simply tweak our recommended
conf/magic
file???
# standard unix compress
# Enable the alternate line below to present gzip content as a transfer
encoded
# stream of the underlying content;
#0 string \037\235 application/octet-stream
x-compress
0 string \037\235 application/octet-stream
# gzip (GNU zip, not to be confused with [Info-ZIP/PKWARE] zip archiver)
# Enable the alternate line below to present gzip content as a transfer
encoded
# stream of the underlying content;
#0 string \037\213 application/octet-stream
x-gzip
0 string \037\213 application/octet-stream
WDYT?
> Something along:
>
> Index: modules/metadata/mod_mime_magic.c
> ===================================================================
> --- modules/metadata/mod_mime_magic.c (revision 1723283)
> +++ modules/metadata/mod_mime_magic.c (working copy)
> @@ -456,6 +456,7 @@ typedef struct {
> const char *magicfile; /* where magic be found */
> struct magic *magic; /* head of magic config list */
> struct magic *last;
> + int uncompress;
> } magic_server_config_rec;
>
> /* per-request info */
> @@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
> {
> AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
> "Path to MIME Magic file (in file(1) format)"),
> + AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
> + (void *)APR_OFFSETOF(magic_server_config_rec, uncompress), RSRC_CONF,
> + "Whether MIME should try to render uncompressed content by
> recognizing "
> + "the underlying type, or not (default)"),
> {NULL}
> };
>
> @@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
>
> static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
> {
> + magic_server_config_rec *conf = (magic_server_config_rec *)
> + ap_get_module_config(r->server->module_config,
> + &mime_magic_module);
> unsigned char *newbuf;
> int newsize;
> int i;
> @@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char *b
> if (i == ncompr)
> return 0;
>
> - if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> - /* set encoding type in the request record */
> - r->content_encoding = compr[i].encoding;
> -
> + if (!conf->uncompress) {
> + magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> + }
> + else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
> /* Try to detect the content type of the uncompressed data */
> if (tryit(r, newbuf, newsize, 0) != OK) {
> + magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> return 0;
> }
> + /* set encoding type in the request record */
> + r->content_encoding = compr[i].encoding;
> }
> return 1;
> }
>
Just trying to understand why conf/magic can't be the lever for this
behavior.
Are you trying to trip this behavior on a per-file/per-directory basis?
That would
be one case where keeping two entire conf/magic[12] files loaded would seem
excessive.
What about adding a MimeMagicItem directive to add/modify just a few lines
not in the master mime magic file table?
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jim Jagielski <ji...@jaguNET.com>.
It just seems like feature-creep and YetAnotherDirective for
something that is better handled some other way. As OtherBill
notes, a small edit to the magic file "fixes" this.
> On Jan 8, 2016, at 11:16 AM, Yann Ylavic <yl...@gmail.com> wrote:
>
> On Fri, Jan 8, 2016 at 1:43 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>> -0.9...
>>
>> This seems a very heavy solution to a specific one-off problem.
>
> Not sure this is the patch which is heavy here...
> Anytime mod_mime_magic recognizes the type of a gzip file (by the
> magic bytes), it unconditionally starts a new piped-process to
> uncompress the file so to determine the type of the gzipped content.
> Then, it forces "Content-Encoding: x-gzip" and (eg.) "Content-Type:
> x-tar", which causes the browser to think this was an original tarball
> compressed by the protocol, and then uncompress it for the user to get
> the (supposedly) original file.
> (Btw the "Content-Encoding" encoding is set whether or not the
> underlying type has been determined successfully, that looks buggy to
> me since the browser could get a C-E w/o C-T, so I fixed it in the
> proposed patch too).
>
> My patch simply disables this by default (and provides an opt-out to
> restore legacy behaviour, but it could easily take the other way too
> to preserve existing), such that now mod_mime_magic can also set only
> "Content-Type: x-gzip" and let (real) original go out with no special
> treatment.
>
> The patch is actually quite small to handle this IMHO...
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Fri, Jan 8, 2016 at 1:43 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> -0.9...
>
> This seems a very heavy solution to a specific one-off problem.
Not sure this is the patch which is heavy here...
Anytime mod_mime_magic recognizes the type of a gzip file (by the
magic bytes), it unconditionally starts a new piped-process to
uncompress the file so to determine the type of the gzipped content.
Then, it forces "Content-Encoding: x-gzip" and (eg.) "Content-Type:
x-tar", which causes the browser to think this was an original tarball
compressed by the protocol, and then uncompress it for the user to get
the (supposedly) original file.
(Btw the "Content-Encoding" encoding is set whether or not the
underlying type has been determined successfully, that looks buggy to
me since the browser could get a C-E w/o C-T, so I fixed it in the
proposed patch too).
My patch simply disables this by default (and provides an opt-out to
restore legacy behaviour, but it could easily take the other way too
to preserve existing), such that now mod_mime_magic can also set only
"Content-Type: x-gzip" and let (real) original go out with no special
treatment.
The patch is actually quite small to handle this IMHO...
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Ronald Masaya <ro...@gmail.com>.
Clash of clans
On Jan 8, 2016 8:43 PM, "Jim Jagielski" <ji...@jagunet.com> wrote:
> -0.9...
>
> This seems a very heavy solution to a specific one-off problem.
>
> > On Jan 8, 2016, at 4:27 AM, Yann Ylavic <yl...@gmail.com> wrote:
> >
> > Hi,
> >
> > On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža <jk...@redhat.com> wrote:
> >>
> >> Content-Type: application/x-tar
> >> Content-Encoding: x-gzip
> > []
> >>
> >> So, the mod_mime_magic is saying here that the body is tarball encoded
> by
> >> gzip.
> >
> > AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
> > the above headers.
> >
> >>
> >> But I think even we are right here, the Content-Encoding should be used
> only
> >> when httpd itself encodes the original content, otherwise it confuses
> >> clients (I've asked few people and it confuses some web browsers too -
> I can
> >> get more info if needed.).
> >
> > Agreed, this looks wrong, the browsers will likely use the .tar file
> > instead (this adds unnecessary cycles to both the server and client).
> >
> >>
> >> Maybe we could stop setting Content-Encoding in mod_mime_magic and just
> use
> >> Content-Type?
> >
> > Since it's always been there, we probably should add a new
> > mod_mime_magic directive to control the behaviour and avoid breaking
> > cases.
> >
> > Something along:
> >
> > Index: modules/metadata/mod_mime_magic.c
> > ===================================================================
> > --- modules/metadata/mod_mime_magic.c (revision 1723283)
> > +++ modules/metadata/mod_mime_magic.c (working copy)
> > @@ -456,6 +456,7 @@ typedef struct {
> > const char *magicfile; /* where magic be found */
> > struct magic *magic; /* head of magic config list */
> > struct magic *last;
> > + int uncompress;
> > } magic_server_config_rec;
> >
> > /* per-request info */
> > @@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
> > {
> > AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
> > "Path to MIME Magic file (in file(1) format)"),
> > + AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
> > + (void *)APR_OFFSETOF(magic_server_config_rec, uncompress),
> RSRC_CONF,
> > + "Whether MIME should try to render uncompressed content by
> recognizing "
> > + "the underlying type, or not (default)"),
> > {NULL}
> > };
> >
> > @@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
> >
> > static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
> > {
> > + magic_server_config_rec *conf = (magic_server_config_rec *)
> > + ap_get_module_config(r->server->module_config,
> > + &mime_magic_module);
> > unsigned char *newbuf;
> > int newsize;
> > int i;
> > @@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char
> *b
> > if (i == ncompr)
> > return 0;
> >
> > - if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> > - /* set encoding type in the request record */
> > - r->content_encoding = compr[i].encoding;
> > -
> > + if (!conf->uncompress) {
> > + magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> > + compr[i].encoding, NULL));
> > + }
> > + else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> > newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
> > /* Try to detect the content type of the uncompressed data */
> > if (tryit(r, newbuf, newsize, 0) != OK) {
> > + magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> > + compr[i].encoding, NULL));
> > return 0;
> > }
> > + /* set encoding type in the request record */
> > + r->content_encoding = compr[i].encoding;
> > }
> > return 1;
> > }
> > --
> >
> > Regards,
> > Yann.
>
>
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jim Jagielski <ji...@jaguNET.com>.
-0.9...
This seems a very heavy solution to a specific one-off problem.
> On Jan 8, 2016, at 4:27 AM, Yann Ylavic <yl...@gmail.com> wrote:
>
> Hi,
>
> On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža <jk...@redhat.com> wrote:
>>
>> Content-Type: application/x-tar
>> Content-Encoding: x-gzip
> []
>>
>> So, the mod_mime_magic is saying here that the body is tarball encoded by
>> gzip.
>
> AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
> the above headers.
>
>>
>> But I think even we are right here, the Content-Encoding should be used only
>> when httpd itself encodes the original content, otherwise it confuses
>> clients (I've asked few people and it confuses some web browsers too - I can
>> get more info if needed.).
>
> Agreed, this looks wrong, the browsers will likely use the .tar file
> instead (this adds unnecessary cycles to both the server and client).
>
>>
>> Maybe we could stop setting Content-Encoding in mod_mime_magic and just use
>> Content-Type?
>
> Since it's always been there, we probably should add a new
> mod_mime_magic directive to control the behaviour and avoid breaking
> cases.
>
> Something along:
>
> Index: modules/metadata/mod_mime_magic.c
> ===================================================================
> --- modules/metadata/mod_mime_magic.c (revision 1723283)
> +++ modules/metadata/mod_mime_magic.c (working copy)
> @@ -456,6 +456,7 @@ typedef struct {
> const char *magicfile; /* where magic be found */
> struct magic *magic; /* head of magic config list */
> struct magic *last;
> + int uncompress;
> } magic_server_config_rec;
>
> /* per-request info */
> @@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
> {
> AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
> "Path to MIME Magic file (in file(1) format)"),
> + AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
> + (void *)APR_OFFSETOF(magic_server_config_rec, uncompress), RSRC_CONF,
> + "Whether MIME should try to render uncompressed content by recognizing "
> + "the underlying type, or not (default)"),
> {NULL}
> };
>
> @@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
>
> static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
> {
> + magic_server_config_rec *conf = (magic_server_config_rec *)
> + ap_get_module_config(r->server->module_config,
> + &mime_magic_module);
> unsigned char *newbuf;
> int newsize;
> int i;
> @@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char *b
> if (i == ncompr)
> return 0;
>
> - if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> - /* set encoding type in the request record */
> - r->content_encoding = compr[i].encoding;
> -
> + if (!conf->uncompress) {
> + magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> + }
> + else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
> newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
> /* Try to detect the content type of the uncompressed data */
> if (tryit(r, newbuf, newsize, 0) != OK) {
> + magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
> + compr[i].encoding, NULL));
> return 0;
> }
> + /* set encoding type in the request record */
> + r->content_encoding = compr[i].encoding;
> }
> return 1;
> }
> --
>
> Regards,
> Yann.
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Yann Ylavic <yl...@gmail.com>.
Hi,
On Fri, Jan 8, 2016 at 8:49 AM, Jan Kaluža <jk...@redhat.com> wrote:
>
> Content-Type: application/x-tar
> Content-Encoding: x-gzip
[]
>
> So, the mod_mime_magic is saying here that the body is tarball encoded by
> gzip.
AIUI, mod_mime_magic does indeed try to uncompress the tar.gz and adds
the above headers.
>
> But I think even we are right here, the Content-Encoding should be used only
> when httpd itself encodes the original content, otherwise it confuses
> clients (I've asked few people and it confuses some web browsers too - I can
> get more info if needed.).
Agreed, this looks wrong, the browsers will likely use the .tar file
instead (this adds unnecessary cycles to both the server and client).
>
> Maybe we could stop setting Content-Encoding in mod_mime_magic and just use
> Content-Type?
Since it's always been there, we probably should add a new
mod_mime_magic directive to control the behaviour and avoid breaking
cases.
Something along:
Index: modules/metadata/mod_mime_magic.c
===================================================================
--- modules/metadata/mod_mime_magic.c (revision 1723283)
+++ modules/metadata/mod_mime_magic.c (working copy)
@@ -456,6 +456,7 @@ typedef struct {
const char *magicfile; /* where magic be found */
struct magic *magic; /* head of magic config list */
struct magic *last;
+ int uncompress;
} magic_server_config_rec;
/* per-request info */
@@ -511,6 +512,10 @@ static const command_rec mime_magic_cmds[] =
{
AP_INIT_TAKE1("MimeMagicFile", set_magicfile, NULL, RSRC_CONF,
"Path to MIME Magic file (in file(1) format)"),
+ AP_INIT_FLAG("MimeMagicUncompress", ap_set_flag_slot,
+ (void *)APR_OFFSETOF(magic_server_config_rec, uncompress), RSRC_CONF,
+ "Whether MIME should try to render uncompressed content by recognizing "
+ "the underlying type, or not (default)"),
{NULL}
};
@@ -2081,6 +2086,9 @@ static int ncompr = sizeof(compr) / sizeof(compr[0
static int zmagic(request_rec *r, unsigned char *buf, apr_size_t nbytes)
{
+ magic_server_config_rec *conf = (magic_server_config_rec *)
+ ap_get_module_config(r->server->module_config,
+ &mime_magic_module);
unsigned char *newbuf;
int newsize;
int i;
@@ -2095,15 +2103,20 @@ static int zmagic(request_rec *r, unsigned char *b
if (i == ncompr)
return 0;
- if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
- /* set encoding type in the request record */
- r->content_encoding = compr[i].encoding;
-
+ if (!conf->uncompress) {
+ magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
+ compr[i].encoding, NULL));
+ }
+ else if ((newsize = uncompress(r, i, &newbuf, HOWMANY)) > 0) {
newbuf[newsize-1] = '\0'; /* null-terminate uncompressed data */
/* Try to detect the content type of the uncompressed data */
if (tryit(r, newbuf, newsize, 0) != OK) {
+ magic_rsl_puts(r, apr_pstrcat(r->pool, "application/",
+ compr[i].encoding, NULL));
return 0;
}
+ /* set encoding type in the request record */
+ r->content_encoding = compr[i].encoding;
}
return 1;
}
--
Regards,
Yann.
Re: mod_mime_magic, gzipped tarballs and Docker
Posted by Jan Kaluža <jk...@redhat.com>.
On 01/08/2016 08:49 AM, Jan Kaluža wrote:
> Hi,
>
> it seems Docker client has a problem handling httpd responses [1] when
> you run Docker server behind httpd working as a reverse proxy. It is
> caused by "mod_mime_magic" adding following Content-Type and
> Content-Encoding to gzipped tarballs sent as a response by Docker server:
Just to correct myself a bit. I have found out it's not a reverse-proxy
case - in that case, mod_mime_magic would not be in the loop. They are
just serving Docker data using following config:
https://github.com/rbarlow/pulp_docker/blob/master/plugins/etc/httpd/conf.d/pulp_docker.conf
> Content-Type: application/x-tar
> Content-Encoding: x-gzip
>
> Docker client expects gzipped tarballs to be received, but because it
> receives Content-Encoding: x-gzip, it decompresses the response and
> receives just plain tar (at least that's how I understand it).
>
> Strictly speaking, httpd is probably right here according to RFC:
>
> """
> The Content-Encoding entity-header field is used as a modifier to the
> media-type. When present, its value indicates what additional content
> codings have been applied to the entity-body, and thus what decoding
> mechanisms must be applied in order to obtain the media-type referenced
> by the Content-Type header field.
> """
>
> So, the mod_mime_magic is saying here that the body is tarball encoded
> by gzip.
>
> But I think even we are right here, the Content-Encoding should be used
> only when httpd itself encodes the original content, otherwise it
> confuses clients (I've asked few people and it confuses some web
> browsers too - I can get more info if needed.).
>
> Maybe we could stop setting Content-Encoding in mod_mime_magic and just
> use Content-Type?
>
> [1] https://github.com/docker/docker/issues/17595
>
> Regards,
> Jan Kaluza
Regards,
Jan Kaluza