You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by br...@apache.org on 2013/01/05 09:36:14 UTC
svn commit: r1429235 - in /subversion/trunk/tools/hook-scripts:
validate-files.conf.example validate-files.py
Author: breser
Date: Sat Jan 5 08:36:14 2013
New Revision: 1429235
URL: http://svn.apache.org/viewvc?rev=1429235&view=rev
Log:
Add validate-files.py, a python pre-commit hook-script that runs commands and
rejects commits if the command does not exit with a 0 exit code.
* tools/hook-scripts/validate-files.py: The hook-script itself.
* tools/hook-scripts/validate-files.conf.example: Example conf file.
Added:
subversion/trunk/tools/hook-scripts/validate-files.conf.example
subversion/trunk/tools/hook-scripts/validate-files.py (with props)
Added: subversion/trunk/tools/hook-scripts/validate-files.conf.example
URL: http://svn.apache.org/viewvc/subversion/trunk/tools/hook-scripts/validate-files.conf.example?rev=1429235&view=auto
==============================================================================
--- subversion/trunk/tools/hook-scripts/validate-files.conf.example (added)
+++ subversion/trunk/tools/hook-scripts/validate-files.conf.example Sat Jan 5 08:36:14 2013
@@ -0,0 +1,67 @@
+# DEFAULT section can be used to place options that can be referenced in
+# other section values with the %(option)s syntax. Note that the svnlook
+# value below is required as it is used by the script to determine the path
+# to the svnlook command in order to determine the changes. Feel free
+# to create additional values here that you can reuse in other options,
+# especially the command options to make it easier to maintain.
+[DEFAULT]
+svnlook = /usr/local/bin/svnlook
+#svnauthz = /usr/local/bin/svn-tools/svnauthz
+#xmllint = /usr/bin/xmllint
+
+# The repositories section has key value pairs where the key is a pattern
+# to match on the repository path and the value is a space separated list of
+# rules to apply to that repository. Multiple patterns can match and all
+# unique rules will be applied. The pattern is a Unix shell-style wildcard.
+# As seen below all repositories will have the svnauthz-validate and xmllint
+# rules applied and repositories in /repos or below will have admin-rw-authz
+# applied.
+[repositories]
+#* = svnauthz-validate xmllint
+#/repos/* = admin-rw-authz
+
+# Rules allow you define a pattern to match against which files in the
+# repository to run a command against. Rules are defined by creating a
+# section name starting with 'rule:' as seen below.
+#
+# The pattern option is a Unix shell-style wildcard match against the
+# files in the repo that the rule will be run for. A leading / in your
+# pattern will be ignored. Paths segments are / separated regardless of
+# platform.
+#
+# The command option is the command to run, this command will be run via
+# the shell of your platform. Your command will have variable replacement
+# made on it prior to execution as follows:
+# $REPO or ${REPO} expands to the path of the repository for the commit.
+# $TXN or ${TXN} expands to the transaction id of the commit.
+# $FILE or ${FILE} expands to the name of the file that matched the pattern.
+#
+# $ characters that are not followed by one of the above variable names will
+# be untouched.
+#
+# IMPORTANT: AS A CONSEQUENCE OF THE USE OF THE SHELL IT IS IMPORTANT TO
+# QUOTE THE ARGUMENTS OF YOUR COMMANDS. THE $FILE VARIABLE DOES CONTAIN
+# USER GENERATED DATA AND SHELL METACHARACTERS ARE NOT ESCAPED FOR YOU!
+
+# The following rule runs the svnauthz command's validate subcommand
+# for file named authz in the conf subdir if it is present in the commit.
+# This is a simple way to ensure that invalid authz files are not allowed
+# to be committed.
+#[rule:svnauthz-validate]
+#pattern = conf/authz
+#command = '%(svnauthz)s' validate -t '$TXN' '$REPO' '$FILE'
+
+# The following rule runs the svnauthz command's accessof subcommand
+# for any file ending in .authz for config subdir and checks that the admin
+# user has rw rights to the same file. This can be used to prevent an
+# authz file being committed that would remove access for the admin user.
+# Note that accessof also validates the validity of the file as well as
+# checking the permissions, so it's unecessary to run validate and accessof.
+#[rule:admin-rw-authz]
+#pattern = /conf/*.authz
+#command = '%(svnauthz)s' accessof --username admin --path '${FILE}' --is rw -t '${TXN}' '${REPO}' '${FILE}'
+
+# Use the xmllint command to validate all files ending in .xml
+#[rule:xmllint]
+#pattern = *.xml
+#command = '%(svnlook)s' cat -t '${TXN}' '${REPO}' '${FILE}' | '%(xmllint)s' --noout -
Added: subversion/trunk/tools/hook-scripts/validate-files.py
URL: http://svn.apache.org/viewvc/subversion/trunk/tools/hook-scripts/validate-files.py?rev=1429235&view=auto
==============================================================================
--- subversion/trunk/tools/hook-scripts/validate-files.py (added)
+++ subversion/trunk/tools/hook-scripts/validate-files.py Sat Jan 5 08:36:14 2013
@@ -0,0 +1,158 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Subversion pre-commit hook script that runs user configured commands
+to validate files in the commit and reject the commit if the commands
+exit with a non-zero exit code. The script expects a validate-files.conf
+file placed in the conf dir under the repo the commit is for."""
+
+import sys
+import os
+import subprocess
+import fnmatch
+from string import Template
+
+# Deal with the rename of ConfigParser to configparser in Python3
+try:
+ # Python >= 3.0
+ import configparser
+except ImportError:
+ # Python < 3.0
+ import ConfigParser as configparser
+
+class Config(configparser.SafeConfigParser):
+ """Superclass of SafeConfigParser with some customizations
+ for this script"""
+ def optionxform(self, option):
+ """Redefine optionxform so option names are case sensitive"""
+ return option
+
+ def getlist(self, section, option):
+ """Returns value of option as a list using whitespace to
+ split entries"""
+ value = self.get(section, option)
+ if value:
+ return value.split()
+ else:
+ return None
+
+ def get_matching_rules(self, repo):
+ """Return list of unique rules names that apply to a given repo"""
+ rules = {}
+ for option in self.options('repositories'):
+ if fnmatch.fnmatch(repo, option):
+ for rule in self.getlist('repositories', option):
+ rules[rule] = True
+ return rules.keys()
+
+ def get_rule_section_name(self, rule):
+ """Given a rule name provide the section name it is defined in."""
+ return 'rule:%s' % (rule)
+
+class Commands:
+ """Class to handle logic of running commands"""
+ def __init__(self, config):
+ self.config = config
+
+ def svnlook_changed(self, repo, txn):
+ """Provide list of files changed in txn of repo"""
+ svnlook = self.config.get('DEFAULT', 'svnlook')
+ cmd = "'%s' changed -t '%s' '%s'" % (svnlook, txn, repo)
+ p = subprocess.Popen(cmd, shell=True,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ changed = []
+ while True:
+ line = p.stdout.readline()
+ if not line:
+ break
+ line = line.strip()
+ text_mod = line[0:1]
+ # Only if the contents of the file changed (by addition or update)
+ # directories always end in / in the svnlook changed output
+ if line[-1] != "/" and (text_mod == "A" or text_mod == "U"):
+ changed.append(line[4:])
+
+ # wait on the command to finish so we can get the
+ # returncode/stderr output
+ data = p.communicate()
+ if p.returncode != 0:
+ sys.stderr.write(data[1])
+ sys.exit(2)
+
+ return changed
+
+ def user_command(self, section, repo, txn, fn):
+ """ Run the command defined for a given section.
+ Replaces $REPO, $TXN and $FILE with the repo, txn and fn arguments
+ in the defined command.
+
+ Returns a tuple of the exit code and the stderr output of the command"""
+ cmd_template = self.config.get(section, 'command')
+ cmd = Template(cmd_template).safe_substitute(REPO=repo,
+ TXN=txn, FILE=fn)
+ p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
+ data = p.communicate()
+ return (p.returncode, data[1])
+
+def main(repo, txn):
+ exitcode = 0
+ config = Config()
+ config.read(os.path.join(repo, 'conf', 'validate-files.conf'))
+ commands = Commands(config)
+
+ rules = config.get_matching_rules(repo)
+
+ # no matching rules so nothing to do
+ if len(rules) == 0:
+ sys.exit(0)
+
+ changed = commands.svnlook_changed(repo, txn)
+ # this shouldn't ever happen
+ if len(changed) == 0:
+ sys.exit(0)
+
+ for rule in rules:
+ section = config.get_rule_section_name(rule)
+ pattern = config.get(section, 'pattern')
+
+ # skip leading slashes if present in the pattern
+ if pattern[0] == '/': pattern = pattern[1:]
+
+ for fn in fnmatch.filter(changed, pattern):
+ (returncode, err_mesg) = commands.user_command(section, repo,
+ txn, fn)
+ if returncode != 0:
+ sys.stderr.write(
+ "\nError validating file '%s' with rule '%s' " \
+ "(exit code %d):\n" % (fn, rule, returncode))
+ sys.stderr.write(err_mesg)
+ exitcode = 1
+
+ return exitcode
+
+if __name__ == "__main__":
+ if len(sys.argv) != 3:
+ print "invalid args"
+ sys.exit(0)
+
+ try:
+ sys.exit(main(sys.argv[1], sys.argv[2]))
+ except configparser.Error as e:
+ sys.stderr.write("Error with the validate-files.conf: %s\n" % e)
+ sys.exit(2)
Propchange: subversion/trunk/tools/hook-scripts/validate-files.py
------------------------------------------------------------------------------
svn:executable = *
Re: svn commit: r1429235 - in /subversion/trunk/tools/hook-scripts:
validate-files.conf.example validate-files.py
Posted by Ben Reser <br...@apache.org>.
On Sat, Jan 5, 2013 at 1:00 AM, Ben Reser <br...@apache.org> wrote:
> Before someone else points it out there are further Python 3
> incompatibilities in the script. I'll clean these up soon.
Fixed in r1429453. I've tested succesfully with Python 2.7.1 and 3.3.0.
Re: svn commit: r1429235 - in /subversion/trunk/tools/hook-scripts:
validate-files.conf.example validate-files.py
Posted by Ben Reser <br...@apache.org>.
On Sat, Jan 5, 2013 at 12:36 AM, <br...@apache.org> wrote:
> Author: breser
> Date: Sat Jan 5 08:36:14 2013
> New Revision: 1429235
>
> URL: http://svn.apache.org/viewvc?rev=1429235&view=rev
> Log:
> Add validate-files.py, a python pre-commit hook-script that runs commands and
> rejects commits if the command does not exit with a 0 exit code.
>
> * tools/hook-scripts/validate-files.py: The hook-script itself.
>
> * tools/hook-scripts/validate-files.conf.example: Example conf file.
>
> Added:
> subversion/trunk/tools/hook-scripts/validate-files.conf.example
> subversion/trunk/tools/hook-scripts/validate-files.py (with props)
Before someone else points it out there are further Python 3
incompatibilities in the script. I'll clean these up soon.
Re: svn commit: r1429235 - in /subversion/trunk/tools/hook-scripts:
validate-files.conf.example validate-files.py
Posted by Ben Reser <be...@reser.org>.
On Sat, Jan 5, 2013 at 11:17 AM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> This quoting is insufficient, it's still prone to SQL injections. Since
> this is a problem every user of this script would have to solve, how
> about having the script ensure that $FILE doesn't contain "'"?
>
> Perhaps make this configurable via a "upon-single-quote = {continue|raise}"
> knob in the config file.
Thanks for the feedback. Switching to environment variables and
letting the shell expand the variables should resolve that.
Done in r1429444
Re: svn commit: r1429235 - in /subversion/trunk/tools/hook-scripts:
validate-files.conf.example validate-files.py
Posted by Ben Reser <be...@reser.org>.
On Sat, Jan 5, 2013 at 11:17 AM, Daniel Shahaf <d....@daniel.shahaf.name> wrote:
> This quoting is insufficient, it's still prone to SQL injections. Since
> this is a problem every user of this script would have to solve, how
> about having the script ensure that $FILE doesn't contain "'"?
>
> Perhaps make this configurable via a "upon-single-quote = {continue|raise}"
> knob in the config file.
Thanks for the feedback. Switching to environment variables and
letting the shell expand the variables should resolve that.
Done in r1429444
Re: svn commit: r1429235 - in
/subversion/trunk/tools/hook-scripts: validate-files.conf.example
validate-files.py
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Wow. I'm probably going to use that on svn.a.o. However..
breser@apache.org wrote on Sat, Jan 05, 2013 at 08:36:14 -0000:
> +# The command option is the command to run, this command will be run via
> +# the shell of your platform. Your command will have variable replacement
> +# made on it prior to execution as follows:
> +# $REPO or ${REPO} expands to the path of the repository for the commit.
> +# $TXN or ${TXN} expands to the transaction id of the commit.
> +# $FILE or ${FILE} expands to the name of the file that matched the pattern.
> +#
> +# $ characters that are not followed by one of the above variable names will
> +# be untouched.
> +#
> +# IMPORTANT: AS A CONSEQUENCE OF THE USE OF THE SHELL IT IS IMPORTANT TO
> +# QUOTE THE ARGUMENTS OF YOUR COMMANDS. THE $FILE VARIABLE DOES CONTAIN
> +# USER GENERATED DATA AND SHELL METACHARACTERS ARE NOT ESCAPED FOR YOU!
> +
> +# The following rule runs the svnauthz command's validate subcommand
> +# for file named authz in the conf subdir if it is present in the commit.
> +# This is a simple way to ensure that invalid authz files are not allowed
> +# to be committed.
> +#[rule:svnauthz-validate]
> +#pattern = conf/authz
> +#command = '%(svnauthz)s' validate -t '$TXN' '$REPO' '$FILE'
This quoting is insufficient, it's still prone to SQL injections. Since
this is a problem every user of this script would have to solve, how
about having the script ensure that $FILE doesn't contain "'"?
Perhaps make this configurable via a "upon-single-quote = {continue|raise}"
knob in the config file.
Re: svn commit: r1429235 - in
/subversion/trunk/tools/hook-scripts: validate-files.conf.example
validate-files.py
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Wow. I'm probably going to use that on svn.a.o. However..
breser@apache.org wrote on Sat, Jan 05, 2013 at 08:36:14 -0000:
> +# The command option is the command to run, this command will be run via
> +# the shell of your platform. Your command will have variable replacement
> +# made on it prior to execution as follows:
> +# $REPO or ${REPO} expands to the path of the repository for the commit.
> +# $TXN or ${TXN} expands to the transaction id of the commit.
> +# $FILE or ${FILE} expands to the name of the file that matched the pattern.
> +#
> +# $ characters that are not followed by one of the above variable names will
> +# be untouched.
> +#
> +# IMPORTANT: AS A CONSEQUENCE OF THE USE OF THE SHELL IT IS IMPORTANT TO
> +# QUOTE THE ARGUMENTS OF YOUR COMMANDS. THE $FILE VARIABLE DOES CONTAIN
> +# USER GENERATED DATA AND SHELL METACHARACTERS ARE NOT ESCAPED FOR YOU!
> +
> +# The following rule runs the svnauthz command's validate subcommand
> +# for file named authz in the conf subdir if it is present in the commit.
> +# This is a simple way to ensure that invalid authz files are not allowed
> +# to be committed.
> +#[rule:svnauthz-validate]
> +#pattern = conf/authz
> +#command = '%(svnauthz)s' validate -t '$TXN' '$REPO' '$FILE'
This quoting is insufficient, it's still prone to SQL injections. Since
this is a problem every user of this script would have to solve, how
about having the script ensure that $FILE doesn't contain "'"?
Perhaps make this configurable via a "upon-single-quote = {continue|raise}"
knob in the config file.