You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Dan Poirier <po...@pobox.com> on 2009/02/24 15:34:23 UTC
Serving filenames with wildcards using mod_proxy_ftp
Index: docs/manual/mod/mod_proxy.xml
===================================================================
--- docs/manual/mod/mod_proxy.xml (revision 747377)
+++ docs/manual/mod/mod_proxy.xml (working copy)
@@ -389,6 +389,47 @@
</usage>
</directivesynopsis>
+<directivesynopsis>
+<name>ProxyFtpListOnWildcard</name>
+<description>Whether wildcards in requested filenames trigger a file listing</description>
+<syntax>ProxyFtpListOnWildcard [on|off]</syntax>
+<default>on</default>
+<contextlist><context>server config</context><context>virtual host</context>
+ <context>directory</context></contextlist>
+<compatibility>Available in IHS 7.0.0.5 and later</compatibility> ????
+
+<usage>
+ <p>The <directive>ProxyFtpListOnWildcard</directive> directive
+ controls whether wildcard characters ("*?[{~") in requested
+ filenames cause <module>mod_proxy_ftp</module> to return a listing
+ of files instead of downloading a file. By default (value on),
+ they do. Set to "off" to allow downloading files even if they
+ have wildcard characters in their names.</p>
+</usage>
+</directivesynopsis>
+
+<directivesynopsis>
+<name>ProxyFtpEscapeWildcards</name>
+<description>Whether wildcards in requested filenames are escaped when sent to the FTP server</description>
+<syntax>ProxyFtpEscapeWildcards [on|off]</syntax>
+<default>on</default>
+<contextlist><context>server config</context><context>virtual host</context>
+ <context>directory</context></contextlist>
+<compatibility>Available in IHS 7.0.0.5 and later</compatibility> ????
+
+<usage>
+ <p>The <directive>ProxyFtpEscapeWildcards</directive> directive
+ controls whether wildcard characters ("*?[{~") in requested
+ filenames are escaped with backslash before sending them to the
+ FTP server. That is the default behavior, but many FTP servers
+ don't know about the escaping and try to serve the literal filenames
+ they were sent, including the backslashes in the names. </p>
+ <p>Set to "off" to allow downloading files with wildcards
+ in their names from FTP servers that don't understand wildcard
+ escaping.</p>
+</usage>
+</directivesynopsis>
+
<directivesynopsis type="section">
<name>ProxyMatch</name>
<description>Container for directives applied to regular-expression-matched
Index: docs/manual/mod/mod_proxy_ftp.xml
===================================================================
--- docs/manual/mod/mod_proxy_ftp.xml (revision 747377)
+++ docs/manual/mod/mod_proxy_ftp.xml (working copy)
@@ -144,5 +144,16 @@
</note>
</section> <!-- /ftppass -->
+ <section id="wildcard"><title>Why do I get a file listing when I expected
+ a file to be downloaded?</title>
+ <p>In order to allow both browsing the directories on an FTP server and
+ downloading files, Apache looks at the request URL. If it looks like
+ a directory, or contains wildcard characters ("*?[{~"), then it
+ guesses that a listing is wanted instead of a download.</p>
+ <p>You can disable the special handling of names with wildcard characters.
+ See the <directive>ProxyFtpListOnWildcard</directive> directive.
+ </p>
+ </section> <!-- /wildcard -->
+
</modulesynopsis>
Index: modules/proxy/mod_proxy.c
===================================================================
--- modules/proxy/mod_proxy.c (revision 747377)
+++ modules/proxy/mod_proxy.c (working copy)
@@ -1163,6 +1163,8 @@
new->cookie_path_str = apr_strmatch_precompile(p, "path=", 0);
new->cookie_domain_str = apr_strmatch_precompile(p, "domain=", 0);
new->interpolate_env = -1; /* unset */
+ new->ftp_list_on_wildcard = 1;
+ new->ftp_escape_wildcards = 1;
return (void *) new;
}
@@ -1190,6 +1192,19 @@
new->ftp_directory_charset = add->ftp_directory_charset ?
add->ftp_directory_charset :
base->ftp_directory_charset;
+ new->ftp_list_on_wildcard = add->ftp_list_on_wildcard_set ?
+ add->ftp_list_on_wildcard :
+ base->ftp_list_on_wildcard;
+ new->ftp_list_on_wildcard_set = add->ftp_list_on_wildcard_set ?
+ 1 :
+ base->ftp_list_on_wildcard_set;
+ new->ftp_escape_wildcards = add->ftp_escape_wildcards_set ?
+ add->ftp_escape_wildcards :
+ base->ftp_escape_wildcards;
+ new->ftp_escape_wildcards_set = add->ftp_escape_wildcards_set ?
+ 1 :
+ base->ftp_escape_wildcards_set;
+
return new;
}
@@ -1915,6 +1930,26 @@
return NULL;
}
+static const char *set_ftp_list_on_wildcard(cmd_parms *cmd, void *dconf,
+ int flag)
+{
+ proxy_dir_conf *conf = dconf;
+
+ conf->ftp_list_on_wildcard = flag;
+ conf->ftp_list_on_wildcard_set = 1;
+ return NULL;
+}
+
+static const char *set_ftp_escape_wildcards(cmd_parms *cmd, void *dconf,
+ int flag)
+{
+ proxy_dir_conf *conf = dconf;
+
+ conf->ftp_escape_wildcards = flag;
+ conf->ftp_escape_wildcards_set = 1;
+ return NULL;
+}
+
static void ap_add_per_proxy_conf(server_rec *s, ap_conf_vector_t *dir_config)
{
proxy_server_conf *sconf = ap_get_module_config(s->module_config,
@@ -2126,6 +2161,10 @@
"A balancer or worker name with list of params"),
AP_INIT_TAKE1("ProxyFtpDirCharset", set_ftp_directory_charset, NULL,
RSRC_CONF|ACCESS_CONF, "Define the character set for proxied FTP listings"),
+ AP_INIT_FLAG("ProxyFtpListOnWildcard", set_ftp_list_on_wildcard, NULL,
+ RSRC_CONF|ACCESS_CONF, "Whether wildcard characters in a path cause mod_proxy_ftp to list the files instead of trying to get them."),
+ AP_INIT_FLAG("ProxyFtpEscapeWildcards", set_ftp_escape_wildcards, NULL,
+ RSRC_CONF|ACCESS_CONF, "Whether the proxy should escape wildcards in paths before sending them to the FTP server. Defaults to on, but most FTP servers will need it turned off if you need to manage paths that contain wildcard characters."),
{NULL}
};
Index: modules/proxy/mod_proxy.h
===================================================================
--- modules/proxy/mod_proxy.h (revision 747377)
+++ modules/proxy/mod_proxy.h (working copy)
@@ -213,6 +213,10 @@
const apr_strmatch_pattern* cookie_domain_str;
const char *ftp_directory_charset;
int interpolate_env;
+ int ftp_list_on_wildcard;
+ int ftp_list_on_wildcard_set;
+ int ftp_escape_wildcards;
+ int ftp_escape_wildcards_set;
} proxy_dir_conf;
/* if we interpolate env vars per-request, we'll need a per-request
Index: modules/proxy/mod_proxy_ftp.c
===================================================================
--- modules/proxy/mod_proxy_ftp.c (revision 747377)
+++ modules/proxy/mod_proxy_ftp.c (working copy)
@@ -63,13 +63,21 @@
* Escape the globbing characters in a path used as argument to
* the FTP commands (SIZE, CWD, RETR, MDTM, ...).
* ftpd assumes '\\' as a quoting character to escape special characters.
+ * Just returns the original string if ProxyFtpEscapeWildcards has been
+ * configured "off".
* Returns: escaped string
*/
#define FTP_GLOBBING_CHARS "*?[{~"
-static char *ftp_escape_globbingchars(apr_pool_t *p, const char *path)
+static const char *ftp_escape_globbingchars(apr_pool_t *p, const char *path, proxy_dir_conf *dconf)
{
- char *ret = apr_palloc(p, 2*strlen(path)+sizeof(""));
+ char *ret;
char *d;
+
+ if (!dconf->ftp_escape_wildcards) {
+ return path;
+ }
+
+ ret = apr_palloc(p, 2*strlen(path)+sizeof(""));
for (d = ret; *path; ++path) {
if (strchr(FTP_GLOBBING_CHARS, *path) != NULL)
*d++ = '\\';
@@ -809,6 +817,8 @@
#if defined(USE_MDTM) && (defined(HAVE_TIMEGM) || defined(HAVE_GMTOFF))
apr_time_t mtime = 0L;
#endif
+ proxy_dir_conf *dconf = ap_get_module_config(r->per_dir_config,
+ &proxy_module);
/* stuff for PASV mode */
int connect = 0, use_port = 0;
@@ -1157,7 +1167,7 @@
* We could also have extended gen_test_char.c with a special T_ESCAPE_FTP_PATH
*/
rc = proxy_ftp_command(apr_pstrcat(p, "CWD ",
- ftp_escape_globbingchars(p, path), CRLF, NULL),
+ ftp_escape_globbingchars(p, path, dconf), CRLF, NULL),
r, origin, bb, &ftpmessage);
*strp = '/';
/* responses: 250, 421, 500, 501, 502, 530, 550 */
@@ -1480,9 +1490,10 @@
}
/* If len == 0 then it must be a directory (you can't RETR nothing)
- * Also, don't allow to RETR by wildcard. Instead, create a dirlisting
+ * Also, don't allow to RETR by wildcard. Instead, create a dirlisting,
+ * unless ProxyFtpListOnWildcard is off.
*/
- if (len == 0 || ftp_check_globbingchars(path)) {
+ if (len == 0 || (ftp_check_globbingchars(path) && dconf->ftp_list_on_wildcard)) {
dirlisting = 1;
}
else {
@@ -1503,7 +1514,7 @@
/* Therefore: switch to binary if the user did not specify ";type=a" */
ftp_set_TYPE(xfer_type, r, origin, bb, &ftpmessage);
rc = proxy_ftp_command(apr_pstrcat(p, "SIZE ",
- ftp_escape_globbingchars(p, path), CRLF, NULL),
+ ftp_escape_globbingchars(p, path, dconf), CRLF, NULL),
r, origin, bb, &ftpmessage);
if (rc == -1 || rc == 421) {
return ftp_proxyerror(r, backend, HTTP_BAD_GATEWAY,
@@ -1522,7 +1533,7 @@
"proxy: FTP: SIZE shows this is a directory");
dirlisting = 1;
rc = proxy_ftp_command(apr_pstrcat(p, "CWD ",
- ftp_escape_globbingchars(p, path), CRLF, NULL),
+ ftp_escape_globbingchars(p, path, dconf), CRLF, NULL),
r, origin, bb, &ftpmessage);
/* possible results: 250, 421, 500, 501, 502, 530, 550 */
/* 250 Requested file action okay, completed. */
@@ -1583,7 +1594,7 @@
* The "." and subsequent digits ("sss") are optional. <..>
* Time values are always represented in UTC (GMT)
*/
- rc = proxy_ftp_command(apr_pstrcat(p, "MDTM ", ftp_escape_globbingchars(p, path), CRLF, NULL),
+ rc = proxy_ftp_command(apr_pstrcat(p, "MDTM ", ftp_escape_globbingchars(p, path, dconf), CRLF, NULL),
r, origin, bb, &ftpmessage);
/* then extract the Last-Modified time from it (YYYYMMDDhhmmss or YYYYMMDDhhmmss.xxx GMT). */
if (rc == 213) {
@@ -1622,7 +1633,7 @@
}
#endif /* USE_MDTM */
/* FIXME: Handle range requests - send REST */
- buf = apr_pstrcat(p, "RETR ", ftp_escape_globbingchars(p, path), CRLF, NULL);
+ buf = apr_pstrcat(p, "RETR ", ftp_escape_globbingchars(p, path, dconf), CRLF, NULL);
}
rc = proxy_ftp_command(buf, r, origin, bb, &ftpmessage);
/* rc is an intermediate response for the LIST or RETR commands */
@@ -1659,7 +1670,7 @@
ftp_set_TYPE('A', r, origin, bb, NULL);
rc = proxy_ftp_command(apr_pstrcat(p, "CWD ",
- ftp_escape_globbingchars(p, path), CRLF, NULL),
+ ftp_escape_globbingchars(p, path, dconf), CRLF, NULL),
r, origin, bb, &ftpmessage);
/* possible results: 250, 421, 500, 501, 502, 530, 550 */
/* 250 Requested file action okay, completed. */
@@ -1709,9 +1720,6 @@
/* set content-type */
if (dirlisting) {
- proxy_dir_conf *dconf = ap_get_module_config(r->per_dir_config,
- &proxy_module);
-
ap_set_content_type(r, apr_pstrcat(p, "text/html;charset=",
dconf->ftp_directory_charset ?
dconf->ftp_directory_charset :
Re: Serving filenames with wildcards using mod_proxy_ftp
Posted by Dan Poirier <po...@pobox.com>.
Eric Covener <co...@gmail.com> writes:
> On Tue, Feb 24, 2009 at 10:29 AM, Dan Poirier <po...@pobox.com> wrote:
>> Also, in testing that change, I found that mod_proxy_ftp escapes
>> wildcards in filenames using backslashes when sending them to the FTP
>> server, which none of the FTP servers I was testing with understood.
>> To continue with my testing, I added another directive to turn that
>> behavior off, but I'd like to better understand why that behavior is
>> there, since it appears to assume a behavior that my FTP servers don't
>> have, and doesn't seem to be mentioned in RFC 959.
>>
>
> What backends/results did you try? Does anyone have any anecdotal
> experience about what backends might require escaping of the glob
> characters on a retrieve?
As I recall, I tried vsftpd, proftpd, and Apache FTP server.
> I'm wondering if we add the feature, should we flip the default in trunk?
I would say yes. I don't think this escaping could ever have been used
before, since any paths with globbing characters in them would have
triggered a file listing, which doesn't do the escaping. So until now,
apache would never have tried to retrieve any file with globbing
characters in its name.
There's not really any reason to make this an option, now that I think
about it, unless someone comes up with an example of an ftp server that
needs it. We might just want to remove the escaping code.
--
Dan Poirier <po...@pobox.com>
Re: Serving filenames with wildcards using mod_proxy_ftp
Posted by Eric Covener <co...@gmail.com>.
On Tue, Feb 24, 2009 at 10:29 AM, Dan Poirier <po...@pobox.com> wrote:
> The ProxyFtpListOnWildcard directive controls whether wildcard
> characters ("*?[{~") in requested filenames cause mod_proxy_ftp to
> return a listing of files instead of downloading a file. By default
> (value on), they do. Set to "off" to allow downloading files even
> if they have wildcard characters in their names. (directory
> context)
Looks reasonable
>
> I'd appreciate comments on this approach, and my proposed
> implementation (patch against trunk is attached).
>
> Also, in testing that change, I found that mod_proxy_ftp escapes
> wildcards in filenames using backslashes when sending them to the FTP
> server, which none of the FTP servers I was testing with understood.
> To continue with my testing, I added another directive to turn that
> behavior off, but I'd like to better understand why that behavior is
> there, since it appears to assume a behavior that my FTP servers don't
> have, and doesn't seem to be mentioned in RFC 959.
>
What backends/results did you try? Does anyone have any anecdotal
experience about what backends might require escaping of the glob
characters on a retrieve?
I'm wondering if we add the feature, should we flip the default in trunk?
--
Eric Covener
covener@gmail.com
Re: Serving filenames with wildcards using mod_proxy_ftp
Posted by Dan Poirier <po...@pobox.com>.
Sorry, I didn't expect the patch to end up before the explanation.
Here's the message that should have come first.
We need to be able to serve files whose names contain wildcards using
mod_proxy_ftp. Right now, mod_proxy_ftp provides a file listing when
it sees wildcards in the filename requested, which is reasonable for
interactive browsing of an FTP site. In this case, though, we need to
actually get files with those names, and we don't need the browsing
function (we already know the filenames). Unfortunately, we don't
control the filenames, or we'd just use filenames without wildcards
and avoid all this.
The solution I'm thinking about is to add a directive to allow
selectively disabling the behavior of doing file listings when
wildcards are seen in the requested filename. E.g.:
The ProxyFtpListOnWildcard directive controls whether wildcard
characters ("*?[{~") in requested filenames cause mod_proxy_ftp to
return a listing of files instead of downloading a file. By default
(value on), they do. Set to "off" to allow downloading files even
if they have wildcard characters in their names. (directory
context)
I'd appreciate comments on this approach, and my proposed
implementation (patch against trunk is attached).
Also, in testing that change, I found that mod_proxy_ftp escapes
wildcards in filenames using backslashes when sending them to the FTP
server, which none of the FTP servers I was testing with understood.
To continue with my testing, I added another directive to turn that
behavior off, but I'd like to better understand why that behavior is
there, since it appears to assume a behavior that my FTP servers don't
have, and doesn't seem to be mentioned in RFC 959.
--
Dan Poirier <po...@pobox.com>