You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2015/02/14 16:22:13 UTC
[jira] [Comment Edited] (LUCENE-6231) smokeTestRelease.py should
retry failed downloads
[ https://issues.apache.org/jira/browse/LUCENE-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321491#comment-14321491 ]
Robert Muir edited comment on LUCENE-6231 at 2/14/15 3:22 PM:
--------------------------------------------------------------
Here is a listing showing binary release size over time for lucene and solr, each. If we don't fix this now, then when? My proposal on LUCENE-6247 would reduce this for lucene to less than 25M. I think similar improvements can be made to the solr artifacts.
Lucene 3.0: 42M
lucene-3.0.3.tar.gz 2010-12-01 15:06 17M
lucene-3.0.3.zip 2010-12-01 15:06 25M
Lucene 4.0: 104M
lucene-4.0.0.tgz 2013-01-15 15:13 47M
lucene-4.0.0.zip 2013-01-15 15:13 57M
Lucene 5.0: 138M
lucene-5.0.0.tgz 09-Feb-2015 17:50 64M
lucene-5.0.0.zip 09-Feb-2015 17:50 74M
Solr 1.4: 113M
apache-solr-1.4.0.tgz 2009-11-09 20:38 54M
apache-solr-1.4.0.zip 2009-11-09 20:38 59M
Solr 3.1: 156M
apache-solr-3.1.0.tgz 2011-03-30 13:21 76M
apache-solr-3.1.0.zip 2011-03-30 13:21 80M
Solr 4.0: 210M
apache-solr-4.0.0.tgz 2013-01-15 15:13 103M
apache-solr-4.0.0.zip 2013-01-15 15:13 107M
Solr 5.0: 250M
solr-5.0.0.tgz 09-Feb-2015 17:56 122M
solr-5.0.0.zip 09-Feb-2015 17:56 128M
edit: fix typo of lucene issue number
was (Author: rcmuir):
Here is a listing showing binary release size over time for lucene and solr, each. If we don't fix this now, then when? My proposal on LUCENE-6427 would reduce this for lucene to less than 25M. I think similar improvements can be made to the solr artifacts.
Lucene 3.0: 42M
lucene-3.0.3.tar.gz 2010-12-01 15:06 17M
lucene-3.0.3.zip 2010-12-01 15:06 25M
Lucene 4.0: 104M
lucene-4.0.0.tgz 2013-01-15 15:13 47M
lucene-4.0.0.zip 2013-01-15 15:13 57M
Lucene 5.0: 138M
lucene-5.0.0.tgz 09-Feb-2015 17:50 64M
lucene-5.0.0.zip 09-Feb-2015 17:50 74M
Solr 1.4: 113M
apache-solr-1.4.0.tgz 2009-11-09 20:38 54M
apache-solr-1.4.0.zip 2009-11-09 20:38 59M
Solr 3.1: 156M
apache-solr-3.1.0.tgz 2011-03-30 13:21 76M
apache-solr-3.1.0.zip 2011-03-30 13:21 80M
Solr 4.0: 210M
apache-solr-4.0.0.tgz 2013-01-15 15:13 103M
apache-solr-4.0.0.zip 2013-01-15 15:13 107M
Solr 5.0: 250M
solr-5.0.0.tgz 09-Feb-2015 17:56 122M
solr-5.0.0.zip 09-Feb-2015 17:56 128M
> smokeTestRelease.py should retry failed downloads
> -------------------------------------------------
>
> Key: LUCENE-6231
> URL: https://issues.apache.org/jira/browse/LUCENE-6231
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Steve Rowe
> Assignee: Steve Rowe
> Fix For: 5.0, Trunk, 5.1
>
> Attachments: LUCENE-6231-part-2.patch, LUCENE-6231-part-3.patch, LUCENE-6231-part-3.patch, LUCENE-6231.patch
>
>
> In the 5.0 RC2 vote thread, [~anshumg] mentioned that 6 attempts at running the smoke tester against the people.apache.org RC URL all failed because of download failures.
> I had the same problem - my first two attempts also failed because of failed downloads - here's the trace from one of them:
> {noformat}
> Traceback (most recent call last):
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 1248, in do_open
> h.request(req.get_method(), req.selector, req.data, headers)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 1061, in request
> self._send_request(method, url, body, headers)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 1099, in _send_request
> self.endheaders(body)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 1057, in endheaders
> self._send_output(message_body)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 902, in _send_output
> self.send(msg)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 840, in send
> self.connect()
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/http/client.py", line 818, in connect
> self.timeout, self.source_address)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 435, in create_connection
> raise err
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 426, in create_connection
> sock.connect(sa)
> TimeoutError: [Errno 60] Operation timed out
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "dev-tools/scripts/smokeTestRelease.py", line 117, in download
> fIn = urllib.request.urlopen(urlString)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 156, in urlopen
> return opener.open(url, data, timeout)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 469, in open
> response = self._open(req, data)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 487, in _open
> '_open', req)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 447, in _call_chain
> result = func(*args)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 1268, in http_open
> return self.do_open(http.client.HTTPConnection, req)
> File "/Users/sarowe/homebrew/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 1251, in do_open
> raise URLError(err)
> urllib.error.URLError: <urlopen error [Errno 60] Operation timed out>
> The above exception was the direct cause of the following exception:
> Traceback (most recent call last):
> File "dev-tools/scripts/smokeTestRelease.py", line 1523, in <module>
> main()
> File "dev-tools/scripts/smokeTestRelease.py", line 1468, in main
> smokeTest(c.java, c.url, c.revision, c.version, c.tmp_dir, c.is_signed, ' '.join(c.test_args))
> File "dev-tools/scripts/smokeTestRelease.py", line 1517, in smokeTest
> checkMaven(baseURL, tmpDir, svnRevision, version, isSigned)
> File "dev-tools/scripts/smokeTestRelease.py", line 1012, in checkMaven
> crawl(artifacts[project], artifactsURL, targetDir)
> File "dev-tools/scripts/smokeTestRelease.py", line 1280, in crawl
> crawl(downloadedFiles, subURL, path, exclusions)
> File "dev-tools/scripts/smokeTestRelease.py", line 1280, in crawl
> crawl(downloadedFiles, subURL, path, exclusions)
> File "dev-tools/scripts/smokeTestRelease.py", line 1283, in crawl
> download(text, subURL, targetDir, quiet=True)
> File "dev-tools/scripts/smokeTestRelease.py", line 139, in download
> raise RuntimeError('failed to download url "%s"' % urlString) from e
> RuntimeError: failed to download url "http://people.apache.org/~anshum/staging_area/lucene-solr-5.0.0-RC2-rev1658469//lucene/maven/org/apache/lucene/lucene-analyzers-uima/5.0.0/lucene-analyzers-uima-5.0.0.jar.asc"
> {noformat}
> I did a recursive download of the RC2 folder on people.apache.org using wget, and there were three download failures, which wget auto-retried, and succeeded in each case on the second attempt - two of these were timeouts and the third was a reset connection:
> {noformat}
> HTTP request sent, awaiting response... No data received.
> Retrying.
> [...]
> HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
> Retrying.
> {noformat}
> I think we should just automatically retry all failed downloads once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org