You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by br...@apache.org on 2013/05/26 22:06:47 UTC
svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub:
revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Author: brane
Date: Sun May 26 20:06:46 2013
New Revision: 1486463
URL: http://svn.apache.org/r1486463
Log:
Add support for unversioned metadata change notifications
(i.e., for now, revprop changes) to SvnPubSub.
* tools/server-side/svnpubsub/svnpubsub/server.py:
Add a second set of URLs that clients can subscribe to and hooks can post to.
(Notification): New; base class for notification objects.
(Commit): Change to subclass of Notification.
(Metadata): New; another subclass of Notification.
(Client.__init__): Accept another parameter 'kind' that define which
notifications (commits or metadata) this client object is interested in.
(Client.interested_in): Check notification types, too.
(SvnPubSub.__notification_uri_map): Maps URL paths to notification types.
(SvnPubSub.__init__): Remember the notification class for this instance.
(SvnPubSub.render_GET): Check notification kind and pass it on to Clients.
(SvnPubSub.notifyAll, SvnPubSub.renderPut):
Parametrize implementation based on notification kind.
(svnpubsub_server): Register two endpoints, for commits and notifications.
* tools/server-side/svnpubsub/revprop-change-hook.py: New hook script.
Based on commit-hook.py but generates revprop change notifications.
* tools/server-side/svnpubsub/svnpubsub/client.py
(Client.__init__): Accept optional metadata_callback.
(Notification): New; base class for notification objects.
(Commit): Make subclass of Notification.
(Metadata): New; subclass of Notification.
(JSONRecordHandler.__init__): Accept metadata_callback (can be None).
(JSONRecordHandler.feed): Fix bug in unexpected version exception text.
Emit metadata events if a handler is available.
(MultiClient.__init__): Accept optional metadata_callback.
(MultiCLient._add_channel): Optionally pass metadata_callback to Client.
* tools/server-side/svnpubsub/watcher.py (_metadata): New callback handler.
(main): Pass _metadata handler to MultiClient constructor.
Added:
subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py (with props)
Modified:
subversion/trunk/tools/server-side/svnpubsub/svnpubsub/client.py
subversion/trunk/tools/server-side/svnpubsub/svnpubsub/server.py
subversion/trunk/tools/server-side/svnpubsub/watcher.py
Added: subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py
URL: http://svn.apache.org/viewvc/subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py?rev=1486463&view=auto
==============================================================================
--- subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py (added)
+++ subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py Sun May 26 20:06:46 2013
@@ -0,0 +1,89 @@
+#!/usr/local/bin/python
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+SVNLOOK="/usr/local/svn-install/current/bin/svnlook"
+#SVNLOOK="/usr/local/bin/svnlook"
+
+HOST="127.0.0.1"
+PORT=2069
+
+import sys
+import subprocess
+try:
+ import simplejson as json
+except ImportError:
+ import json
+
+import urllib2
+
+def svncmd(cmd):
+ return subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
+
+def svncmd_uuid(repo):
+ cmd = "%s uuid %s" % (SVNLOOK, repo)
+ p = svncmd(cmd)
+ return p.stdout.read().strip()
+
+def svncmd_revprop(repo, revision, propname):
+ cmd = "%s propget -r %s --revprop %s %s" % (SVNLOOK, revision, repo, propname)
+ p = svncmd(cmd)
+ data = p.stdout.read()
+ #print data
+ return data
+
+def do_put(body):
+ opener = urllib2.build_opener(urllib2.HTTPHandler)
+ request = urllib2.Request("http://%s:%d/metadata" %(HOST, PORT), data=body)
+ request.add_header('Content-Type', 'application/json')
+ request.get_method = lambda: 'PUT'
+ url = opener.open(request)
+
+
+def main(repo, revision, author, propname, action):
+ revision = revision.lstrip('r')
+ if action in ('A', 'M'):
+ new_value = svncmd_revprop(repo, revision, propname)
+ elif action == 'D':
+ new_value = None
+ else:
+ sys.stderr.write('Unknown revprop change action "%s"\n' % action)
+ return
+ if action in ('D', 'M'):
+ old_value = sys.stdin.read()
+ else:
+ old_value = None
+ data = {'type': 'svn',
+ 'format': 1,
+ 'id': int(revision),
+ 'repository': svncmd_uuid(repo),
+ 'revprop': {
+ 'name': propname,
+ 'committer': author,
+ 'value': new_value,
+ 'old_value': old_value,
+ }
+ }
+ body = json.dumps(data)
+ do_put(body)
+
+if __name__ == "__main__":
+ if len(sys.argv) != 6:
+ sys.stderr.write("invalid args\n")
+ sys.exit(0)
+
+ main(*sys.argv[1:6])
Propchange: subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: subversion/trunk/tools/server-side/svnpubsub/revprop-change-hook.py
------------------------------------------------------------------------------
svn:executable = *
Modified: subversion/trunk/tools/server-side/svnpubsub/svnpubsub/client.py
URL: http://svn.apache.org/viewvc/subversion/trunk/tools/server-side/svnpubsub/svnpubsub/client.py?rev=1486463&r1=1486462&r2=1486463&view=diff
==============================================================================
--- subversion/trunk/tools/server-side/svnpubsub/svnpubsub/client.py (original)
+++ subversion/trunk/tools/server-side/svnpubsub/svnpubsub/client.py Sun May 26 20:06:46 2013
@@ -62,7 +62,8 @@ class SvnpubsubClientException(Exception
class Client(asynchat.async_chat):
- def __init__(self, url, commit_callback, event_callback):
+ def __init__(self, url, commit_callback, event_callback,
+ metadata_callback = None):
asynchat.async_chat.__init__(self)
self.last_activity = time.time()
@@ -82,7 +83,8 @@ class Client(asynchat.async_chat):
self.event_callback = event_callback
- self.parser = JSONRecordHandler(commit_callback, event_callback)
+ self.parser = JSONRecordHandler(commit_callback, event_callback,
+ metadata_callback)
# Wait for the end of headers. Then we start parsing JSON.
self.set_terminator(b'\r\n\r\n')
@@ -126,36 +128,50 @@ class Client(asynchat.async_chat):
self.ibuffer.append(data)
+class Notification(object):
+ def __init__(self, data):
+ self.__dict__.update(data)
+
+class Commit(Notification):
+ KIND = 'COMMIT'
+
+class Metadata(Notification):
+ KIND = 'METADATA'
+
+
class JSONRecordHandler:
- def __init__(self, commit_callback, event_callback):
+ def __init__(self, commit_callback, event_callback, metadata_callback):
self.commit_callback = commit_callback
self.event_callback = event_callback
+ self.metadata_callback = metadata_callback
+
+ EXPECTED_VERSION = 1
def feed(self, record):
obj = json.loads(record)
if 'svnpubsub' in obj:
actual_version = obj['svnpubsub'].get('version')
- EXPECTED_VERSION = 1
- if actual_version != EXPECTED_VERSION:
- raise SvnpubsubClientException("Unknown svnpubsub format: %r != %d"
- % (actual_format, expected_format))
+ if actual_version != self.EXPECTED_VERSION:
+ raise SvnpubsubClientException(
+ "Unknown svnpubsub format: %r != %d"
+ % (actual_version, self.EXPECTED_VERSION))
self.event_callback('version', obj['svnpubsub']['version'])
elif 'commit' in obj:
commit = Commit(obj['commit'])
self.commit_callback(commit)
elif 'stillalive' in obj:
self.event_callback('ping', obj['stillalive'])
-
-
-class Commit(object):
- def __init__(self, commit):
- self.__dict__.update(commit)
+ elif 'metadata' in obj and self.metadata_callback:
+ metadata = Metadata(obj['metadata'])
+ self.metadata_callback(metadata)
class MultiClient(object):
- def __init__(self, urls, commit_callback, event_callback):
+ def __init__(self, urls, commit_callback, event_callback,
+ metadata_callback = None):
self.commit_callback = commit_callback
self.event_callback = event_callback
+ self.metadata_callback = metadata_callback
# No target time, as no work to do
self.target_time = 0
@@ -185,9 +201,15 @@ class MultiClient(object):
def _add_channel(self, url):
# Simply instantiating the client will install it into the global map
# for processing in the main event loop.
- Client(url,
- functools.partial(self.commit_callback, url),
- functools.partial(self._reconnect, url))
+ if self.metadata_callback:
+ Client(url,
+ functools.partial(self.commit_callback, url),
+ functools.partial(self._reconnect, url),
+ functools.partial(self.metadata_callback, url))
+ else:
+ Client(url,
+ functools.partial(self.commit_callback, url),
+ functools.partial(self._reconnect, url))
def _check_stale(self):
now = time.time()
Modified: subversion/trunk/tools/server-side/svnpubsub/svnpubsub/server.py
URL: http://svn.apache.org/viewvc/subversion/trunk/tools/server-side/svnpubsub/svnpubsub/server.py?rev=1486463&r1=1486462&r2=1486463&view=diff
==============================================================================
--- subversion/trunk/tools/server-side/svnpubsub/svnpubsub/server.py (original)
+++ subversion/trunk/tools/server-side/svnpubsub/svnpubsub/server.py Sun May 26 20:06:46 2013
@@ -34,11 +34,18 @@
# curl -sN http://127.0.0.1:2069/commits/*/13f79535-47bb-0310-9956-ffa450edef68
# curl -sN http://127.0.0.1:2069/commits/svn/13f79535-47bb-0310-9956-ffa450edef68
#
-# URL is built into 2 parts:
-# /commits/${optional_type}/${optional_repository}
+# curl -sN http://127.0.0.1:2069/metadata
+# curl -sN http://127.0.0.1:2069/metadata/svn/*
+# curl -sN http://127.0.0.1:2069/metadata/svn
+# curl -sN http://127.0.0.1:2069/metadata/*/13f79535-47bb-0310-9956-ffa450edef68
+# curl -sN http://127.0.0.1:2069/metadata/svn/13f79535-47bb-0310-9956-ffa450edef68
#
-# If the type is included in the URL, you will only get commits of that type.
-# The type can be * and then you will receive commits of any type.
+# URLs are constructed from 3 parts:
+# /${notification}/${optional_type}/${optional_repository}
+#
+# Notifications can be sent for commits or metadata (e.g., revprop) changes.
+# If the type is included in the URL, you will only get notifications of that type.
+# The type can be * and then you will receive notifications of any type.
#
# If the repository is included in the URL, you will only receive
# messages about that repository. The repository can be * and then you
@@ -71,7 +78,7 @@ from twisted.python import log
import time
-class Commit:
+class Notification(object):
def __init__(self, r):
self.__dict__.update(r)
if not self.check_value('repository'):
@@ -86,7 +93,16 @@ class Commit:
def check_value(self, k):
return hasattr(self, k) and self.__dict__[k]
- def render_commit(self):
+ def render(self):
+ raise NotImplementedError
+
+ def render_log(self):
+ raise NotImplementedError
+
+class Commit(Notification):
+ KIND = 'COMMIT'
+
+ def render(self):
obj = {'commit': {}}
obj['commit'].update(self.__dict__)
return json.dumps(obj)
@@ -96,20 +112,32 @@ class Commit:
paths_changed = " %d paths changed" % len(self.changed)
except:
paths_changed = ""
- return "%s:%s repo '%s' id '%s'%s" % (self.type,
- self.format,
- self.repository,
- self.id,
- paths_changed)
+ return "commit %s:%s repo '%s' id '%s'%s" % (
+ self.type, self.format, self.repository, self.id,
+ paths_changed)
+
+class Metadata(Notification):
+ KIND = 'METADATA'
+
+ def render(self):
+ obj = {'metadata': {}}
+ obj['metadata'].update(self.__dict__)
+ return json.dumps(obj)
+
+ def render_log(self):
+ return "metadata %s:%s repo '%s' id '%s' revprop '%s'" % (
+ self.type, self.format, self.repository, self.id,
+ self.revprop.name)
HEARTBEAT_TIME = 15
class Client(object):
- def __init__(self, pubsub, r, type, repository):
+ def __init__(self, pubsub, r, kind, type, repository):
self.pubsub = pubsub
r.notifyFinish().addErrback(self.finished)
self.r = r
+ self.kind = kind
self.type = type
self.repository = repository
self.alive = True
@@ -123,11 +151,14 @@ class Client(object):
except ValueError:
pass
- def interested_in(self, commit):
- if self.type and self.type != commit.type:
+ def interested_in(self, notification):
+ if self.kind != notification.KIND:
return False
- if self.repository and self.repository != commit.repository:
+ if self.type and self.type != notification.type:
+ return False
+
+ if self.repository and self.repository != notification.repository:
return False
return True
@@ -163,6 +194,13 @@ class SvnPubSub(resource.Resource):
isLeaf = True
clients = []
+ __notification_uri_map = {'commits': Commit.KIND,
+ 'metadata': Metadata.KIND}
+
+ def __init__(self, notification_class):
+ resource.Resource.__init__(self)
+ self.__notification_class = notification_class
+
def cc(self):
return len(self.clients)
@@ -182,6 +220,11 @@ class SvnPubSub(resource.Resource):
request.setResponseCode(400)
return "Invalid path\n"
+ kind = self.__notification_uri_map.get(uri[1], None)
+ if kind is None:
+ request.setResponseCode(400)
+ return "Invalid path\n"
+
if uri_len >= 3:
type = uri[2]
@@ -194,17 +237,18 @@ class SvnPubSub(resource.Resource):
if repository == '*':
repository = None
- c = Client(self, request, type, repository)
+ c = Client(self, request, kind, type, repository)
self.clients.append(c)
c.start()
return twisted.web.server.NOT_DONE_YET
- def notifyAll(self, commit):
- data = commit.render_commit()
+ def notifyAll(self, notification):
+ data = notification.render()
- log.msg("COMMIT: %s (%d clients)" % (commit.render_log(), self.cc()))
+ log.msg("%s: %s (%d clients)"
+ % (notification.KIND, notification.render_log(), self.cc()))
for client in self.clients:
- if client.interested_in(commit):
+ if client.interested_in(notification):
client.write_data(data)
def render_PUT(self, request):
@@ -217,19 +261,23 @@ class SvnPubSub(resource.Resource):
#import pdb;pdb.set_trace()
#print "input: %s" % (input)
try:
- c = json.loads(input)
- commit = Commit(c)
+ data = json.loads(input)
+ notification = self.__notification_class(data)
except ValueError as e:
request.setResponseCode(400)
- log.msg("COMMIT: failed due to: %s" % str(e))
- return str(e)
- self.notifyAll(commit)
+ errstr = str(e)
+ log.msg("%s: failed due to: %s" % (notification.KIND, errstr))
+ return errstr
+ self.notifyAll(notification)
return "Ok"
+
def svnpubsub_server():
root = resource.Resource()
- s = SvnPubSub()
- root.putChild("commits", s)
+ c = SvnPubSub(Commit)
+ m = SvnPubSub(Metadata)
+ root.putChild('commits', c)
+ root.putChild('metadata', m)
return server.Site(root)
if __name__ == "__main__":
Modified: subversion/trunk/tools/server-side/svnpubsub/watcher.py
URL: http://svn.apache.org/viewvc/subversion/trunk/tools/server-side/svnpubsub/watcher.py?rev=1486463&r1=1486462&r2=1486463&view=diff
==============================================================================
--- subversion/trunk/tools/server-side/svnpubsub/watcher.py (original)
+++ subversion/trunk/tools/server-side/svnpubsub/watcher.py Sun May 26 20:06:46 2013
@@ -35,6 +35,9 @@ def _commit(url, commit):
print('COMMIT: from %s' % url)
pprint.pprint(vars(commit), indent=2)
+def _metadata(url, metadata):
+ print('METADATA: from %s' % url)
+ pprint.pprint(vars(metadata), indent=2)
def _event(url, event_name, event_arg):
if event_arg:
@@ -44,7 +47,7 @@ def _event(url, event_name, event_arg):
def main(urls):
- mc = svnpubsub.client.MultiClient(urls, _commit, _event)
+ mc = svnpubsub.client.MultiClient(urls, _commit, _event, _metadata)
mc.run_forever()
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub:
revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Branko Čibej <br...@wandisco.com>.
On 26.05.2013 23:22, Daniel Shahaf wrote:
> On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
>> +def main(repo, revision, author, propname, action):
>> + else:
>> + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
>> + return
> Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
> (libsvn_repos discards stderr when the exit code is zero).
>
>> +if __name__ == "__main__":
>> + if len(sys.argv) != 6:
>> + sys.stderr.write("invalid args\n")
>> + sys.exit(0)
>> +
> Same.
I was wondering about that myself. I basically just copied the behaviour
of the commit-hook.
-- Brane
--
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub:
revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Branko Čibej <br...@wandisco.com>.
On 27.05.2013 10:22, Branko Čibej wrote:
> On 27.05.2013 06:41, Daniel Shahaf wrote:
>> On Sun, May 26, 2013 at 09:22:48PM +0000, Daniel Shahaf wrote:
>>> On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
>>>> +def main(repo, revision, author, propname, action):
>>>> + else:
>>>> + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
>>>> + return
>>> Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
>>> (libsvn_repos discards stderr when the exit code is zero).
>> You haven't s/return/sys.exit(1)/ here. Do you disagree with that change?
> Ah, I just missed it. It was kind of late.
r1486597
-- Brane
--
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub:
revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Branko Čibej <br...@wandisco.com>.
On 27.05.2013 06:41, Daniel Shahaf wrote:
> On Sun, May 26, 2013 at 09:22:48PM +0000, Daniel Shahaf wrote:
>> On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
>>> +def main(repo, revision, author, propname, action):
>>> + else:
>>> + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
>>> + return
>> Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
>> (libsvn_repos discards stderr when the exit code is zero).
> You haven't s/return/sys.exit(1)/ here. Do you disagree with that change?
Ah, I just missed it. It was kind of late.
-- Brane
--
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub: revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Daniel Shahaf <da...@apache.org>.
On Sun, May 26, 2013 at 09:22:48PM +0000, Daniel Shahaf wrote:
> On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
> > +def main(repo, revision, author, propname, action):
> > + else:
> > + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
> > + return
>
> Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
> (libsvn_repos discards stderr when the exit code is zero).
You haven't s/return/sys.exit(1)/ here. Do you disagree with that change?
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub: revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Daniel Shahaf <da...@apache.org>.
On Sun, May 26, 2013 at 09:22:48PM +0000, Daniel Shahaf wrote:
> On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
> > +def main(repo, revision, author, propname, action):
> > + else:
> > + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
> > + return
>
> Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
> (libsvn_repos discards stderr when the exit code is zero).
You haven't s/return/sys.exit(1)/ here. Do you disagree with that change?
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub: revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Daniel Shahaf <da...@apache.org>.
On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
> +def main(repo, revision, author, propname, action):
> + else:
> + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
> + return
Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
(libsvn_repos discards stderr when the exit code is zero).
> +if __name__ == "__main__":
> + if len(sys.argv) != 6:
> + sys.stderr.write("invalid args\n")
> + sys.exit(0)
> +
Same.
Re: svn commit: r1486463 - in /subversion/trunk/tools/server-side/svnpubsub: revprop-change-hook.py svnpubsub/client.py svnpubsub/server.py watcher.py
Posted by Daniel Shahaf <da...@apache.org>.
On Sun, May 26, 2013 at 08:06:47PM -0000, brane@apache.org wrote:
> +def main(repo, revision, author, propname, action):
> + else:
> + sys.stderr.write('Unknown revprop change action "%s"\n' % action)
> + return
Maybe sys.exit(1)? Otherwise the stderr output will likely go unnoticed
(libsvn_repos discards stderr when the exit code is zero).
> +if __name__ == "__main__":
> + if len(sys.argv) != 6:
> + sys.stderr.write("invalid args\n")
> + sys.exit(0)
> +
Same.