[jira] [Created] (COMDEV-222) projects.a.o does not handle SVN problems well

["Sebb (JIRA)" <ji...@apache.org>]

Sebb created COMDEV-222:
---------------------------

             Summary: projects.a.o does not handle SVN problems well
                 Key: COMDEV-222
                 URL: https://issues.apache.org/jira/browse/COMDEV-222
             Project: Community Development
          Issue Type: Bug
          Components: Projects
            Reporter: Sebb


Recently there was some kind of a problem accessing SVN. This caused lots of errors [1] like:

Error when processing doap file http://svn.apache.org/repos/asf/xmlgraphics/site/trunk/content/fop/doap.rdf:
------------------------------------------------------------
Traceback (most recent call last):
  File "parseprojects.py", line 154, in <module>
    rdf = urllib.request.urlopen(url, timeout=URL_TIMEOUT).read()
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 507, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
URL: 'http://svn.apache.org/repos/asf/xmlgraphics/site/trunk/content/fop/doap.rdf'

Each such failure causes a message to site-dev
The failed gets also cause the doaps to be dropped:

Deleting obsolete file projects/xmlgraphics-fop.json

The script needs to be able to distinguish different kinds of errors and handle them accordingly.

For example, missing DOAPs are quite frequent, and should be retried next run.
However SVN errors such as the above should probably cause the run to be abandoned.

[1] https://reporter.apache.org/logs/2017-06/parseprojects_2017-06.log



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)