You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/10/06 08:10:46 UTC
[GitHub] [lucene] janhoy opened a new pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
janhoy opened a new pull request #355:
URL: https://github.com/apache/lucene/pull/355
https://issues.apache.org/jira/browse/LUCENE-9997
This PR mostly removes solr specific tests, but also
* Fix some python formatting, such as two LF before methods
* Initialize non-initialized variables
* remove `checkBrokenLinks` which I believe is now done in gradle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r727854794
##########
File path: lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
##########
@@ -175,7 +175,7 @@ public static void main(String[] args) throws Exception {
+ " documents in "
+ (end.getTime() - start.getTime())
+ " milliseconds");
- if (reader.numDocs() > 100 && vectorDictSize < 1_000_000) {
+ if (reader.numDocs() > 100 && vectorDictSize < 1_000_000 && System.getProperty("smoketester") == null) {
throw new RuntimeException(
"Are you (ab)using the toy vector dictionary? See the package javadocs to understand why you got this exception.");
}
Review comment:
The demo indexer failed due to this exception, so I add a 'smoketester' mode to it that will allow "abuse".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725915416
##########
File path: dev-tools/scripts/buildAndPushRelease.py
##########
@@ -329,7 +329,11 @@ def main():
print('Next run the smoker tester:')
p = re.compile(".*/")
m = p.match(sys.argv[0])
- print('%s -u %ssmokeTestRelease.py %s' % (sys.executable, m.group(), url))
+ if not c.sign:
Review comment:
This is another improvement from branch_8x that was never applied to main, bringing it in here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725370169
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -900,42 +701,19 @@ def testDemo(run_java, isSrc, version, jdk):
if removeTrailingZeros(actualVersion) != removeTrailingZeros(version):
raise RuntimeError('wrong version from CheckIndex: got "%s" but expected "%s"' % (actualVersion, version))
+
def removeTrailingZeros(version):
return re.sub(r'(\.0)*$', '', version)
-def checkMaven(solrSrcUnpackPath, baseURL, tmpDir, gitRevision, version, isSigned, keysFile):
Review comment:
Oops. I was too quick removing this, as it looked as if it was solr specific, but the `solrSrcUnpackPath` was only to get hold of the maven pom templates that we don't use anymore as we don't maintain a separate maven build.
I'm bringing the test(s) back, for lucene only. Have not tested yet, but hope to do that too...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725913472
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -1376,7 +1144,7 @@ def main():
c = parse_config()
scriptVersion = scriptutil.find_current_version()
- if not c.version.startswith(scriptVersion + '.'):
+ if not c.version.startswith(scriptVersion + '.') and c.version != scriptVersion:
Review comment:
Earlier the `find_current_version()` would return an `X.Y` version, but in some refactor sometime, when moving to `scriptutil.py` and using from other places, it now returns `X.Y.Z`. So I modified this so we once again compare release-version to "minor" script-version, e.g. `"9.0.0".startsWith("9.0")`. This allows the smokeTester in `branch_9_0` to be used to smoketest release 9.0.1, 9.0.2 etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] cpoerschke commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
cpoerschke commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r723514273
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -658,97 +616,41 @@ def verifyUnpacked(java, project, artifact, unpackPath, gitRevision, version, te
print(' run "%s"' % validateCmd)
java.run_java11(validateCmd, '%s/validate.log' % unpackPath)
- if project == 'lucene':
- print(" run tests w/ Java 11 and testArgs='%s'..." % testArgs)
- java.run_java11('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
- java.run_java11('ant jar', '%s/compile.log' % unpackPath)
- testDemo(java.run_java11, isSrc, version, '11')
-
- print(' generate javadocs w/ Java 11...')
- java.run_java11('ant javadocs', '%s/javadocs.log' % unpackPath)
- checkBrokenLinks('%s/build/docs' % unpackPath)
-
- if java.run_java12:
- print(" run tests w/ Java 12 and testArgs='%s'..." % testArgs)
- java.run_java12('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
- java.run_java12('ant jar', '%s/compile.log' % unpackPath)
- testDemo(java.run_java12, isSrc, version, '12')
-
- #print(' generate javadocs w/ Java 12...')
- #java.run_java12('ant javadocs', '%s/javadocs.log' % unpackPath)
- #checkBrokenLinks('%s/build/docs' % unpackPath)
-
- else:
- os.chdir('solr')
-
- print(" run tests w/ Java 11 and testArgs='%s'..." % testArgs)
- java.run_java11('ant clean test -Dtests.slow=false %s' % testArgs, '%s/test.log' % unpackPath)
-
- # test javadocs
- print(' generate javadocs w/ Java 11...')
- java.run_java11('ant clean javadocs', '%s/javadocs.log' % unpackPath)
- checkBrokenLinks('%s/solr/build/docs')
-
- print(' test solr example w/ Java 11...')
- java.run_java11('ant clean server', '%s/antexample.log' % unpackPath)
- testSolrExample(unpackPath, java.java11_home, True)
+ print(" run tests w/ Java 11 and testArgs='%s'..." % testArgs)
+ java.run_java11('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
+ java.run_java11('ant jar', '%s/compile.log' % unpackPath)
+ testDemo(java.run_java11, isSrc, version, '11')
- if java.run_java12:
- print(" run tests w/ Java 12 and testArgs='%s'..." % testArgs)
- java.run_java12('ant clean test -Dtests.slow=false %s' % testArgs, '%s/test.log' % unpackPath)
+ print(' generate javadocs w/ Java 11...')
+ java.run_java11('ant javadocs', '%s/javadocs.log' % unpackPath)
+ checkBrokenLinks('%s/build/docs' % unpackPath)
- #print(' generate javadocs w/ Java 12...')
- #java.run_java12('ant clean javadocs', '%s/javadocs.log' % unpackPath)
- #checkBrokenLinks('%s/solr/build/docs' % unpackPath)
+ if java.run_java12:
+ print(" run tests w/ Java 12 and testArgs='%s'..." % testArgs)
+ java.run_java12('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
+ java.run_java12('ant jar', '%s/compile.log' % unpackPath)
+ testDemo(java.run_java12, isSrc, version, '12')
- print(' test solr example w/ Java 12...')
- java.run_java12('ant clean server', '%s/antexample.log' % unpackPath)
- testSolrExample(unpackPath, java.java12_home, True)
-
- os.chdir('..')
- print(' check NOTICE')
- testNotice(unpackPath)
+ #print(' generate javadocs w/ Java 12...')
+ #java.run_java12('ant javadocs', '%s/javadocs.log' % unpackPath)
+ #checkBrokenLinks('%s/build/docs' % unpackPath)
else:
checkAllJARs(os.getcwd(), project, gitRevision, version, tmpDir, baseURL)
- if project == 'lucene':
- testDemo(java.run_java11, isSrc, version, '11')
- if java.run_java12:
- testDemo(java.run_java12, isSrc, version, '12')
-
- else:
- print(' copying unpacked distribution for Java 11 ...')
- java11UnpackPath = '%s-java11' % unpackPath
- if os.path.exists(java11UnpackPath):
- shutil.rmtree(java11UnpackPath)
- shutil.copytree(unpackPath, java11UnpackPath)
- os.chdir(java11UnpackPath)
- print(' test solr example w/ Java 11...')
- testSolrExample(java11UnpackPath, java.java11_home, False)
-
- if java.run_java12:
- print(' copying unpacked distribution for Java 12 ...')
- java12UnpackPath = '%s-java12' % unpackPath
- if os.path.exists(java12UnpackPath):
- shutil.rmtree(java12UnpackPath)
- shutil.copytree(unpackPath, java12UnpackPath)
- os.chdir(java12UnpackPath)
- print(' test solr example w/ Java 12...')
- testSolrExample(java12UnpackPath, java.java12_home, False)
-
- os.chdir(unpackPath)
+ testDemo(java.run_java11, isSrc, version, '11')
+ if java.run_java12:
+ testDemo(java.run_java12, isSrc, version, '12')
testChangesText('.', version, project)
- if project == 'lucene' and isSrc:
+ if isSrc:
print(' confirm all releases have coverage in TestBackwardsCompatibility')
confirmAllReleasesAreTestedForBackCompat(version, unpackPath)
def testNotice(unpackPath):
- solrNotice = open('%s/NOTICE.txt' % unpackPath, encoding='UTF-8').read()
luceneNotice = open('%s/lucene/NOTICE.txt' % unpackPath, encoding='UTF-8').read()
expected = """
Review comment:
Looks like this is no longer used now then, but the `testNotice` function itself still confirms that the `lucene/NOTICE.txt` is readable i.e. the local variable here could be removed (or put to use) but the function itself can stay (assuming no other tests do equivalent checks).
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -900,42 +701,19 @@ def testDemo(run_java, isSrc, version, jdk):
if removeTrailingZeros(actualVersion) != removeTrailingZeros(version):
raise RuntimeError('wrong version from CheckIndex: got "%s" but expected "%s"' % (actualVersion, version))
+
def removeTrailingZeros(version):
return re.sub(r'(\.0)*$', '', version)
-def checkMaven(solrSrcUnpackPath, baseURL, tmpDir, gitRevision, version, isSigned, keysFile):
Review comment:
Maybe also somehow mention this in the PR description since it's also removing some 'lucene' logic.
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -48,6 +48,7 @@
cygwin = platform.system().lower().startswith('cygwin')
cygwinWindowsRoot = os.popen('cygpath -w /').read().strip().replace('\\','/') if cygwin else ''
+
Review comment:
Line 44 above has an `and /solr` that can be removed.
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -1376,7 +1144,7 @@ def main():
c = parse_config()
scriptVersion = scriptutil.find_current_version()
- if not c.version.startswith(scriptVersion + '.'):
+ if not c.version.startswith(scriptVersion + '.') and c.version != scriptVersion:
Review comment:
Could this be just this?
```suggestion
if c.version != scriptVersion:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] cpoerschke commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
cpoerschke commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r727204711
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -141,21 +146,19 @@ def checkJARMetaData(desc, jarFile, gitRevision, version):
raise RuntimeError('%s is missing %s' % (desc, name))
except KeyError:
raise RuntimeError('%s is missing %s' % (desc, name))
-
+
s = decodeUTF8(z.read(MANIFEST_FILE_NAME))
-
+
for verify in (
'Specification-Vendor: The Apache Software Foundation',
'Implementation-Vendor: The Apache Software Foundation',
- # Make sure 1.8 compiler was used to build release bits:
+ 'Specification-Title: Lucene Search Engine:',
+ 'Implementation-Title: org.apache.lucene',
'X-Compile-Source-JDK: 11',
- # Make sure 1.8, 1.9 or 1.10 ant was used to build release bits: (this will match 1.8.x, 1.9.x, 1.10.x)
- ('Ant-Version: Apache Ant 1.8', 'Ant-Version: Apache Ant 1.9', 'Ant-Version: Apache Ant 1.10'),
- # Make sure .class files are 1.8 format:
Review comment:
`git grep -i ant` also finds some other ant reference e.g. line 70. could those be removed too maybe then?
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -1043,122 +823,81 @@ def getPOMcoordinate(treeRoot):
packaging = 'jar' if packaging is None else packaging.text.strip()
return groupId, artifactId, packaging, version
-def verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile):
- print(' verify maven artifact sigs', end=' ')
- for project in ('lucene', 'solr'):
-
- # Set up clean gpg world; import keys file:
- gpgHomeDir = '%s/%s.gpg' % (tmpDir, project)
- if os.path.exists(gpgHomeDir):
- shutil.rmtree(gpgHomeDir)
- os.makedirs(gpgHomeDir, 0o700)
- run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
- '%s/%s.gpg.import.log' % (tmpDir, project))
-
- reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
- for artifactFile in [a for a in artifacts[project] if reArtifacts.search(a)]:
- artifact = os.path.basename(artifactFile)
- sigFile = '%s.asc' % artifactFile
- # Test sig (this is done with a clean brand-new GPG world)
- logFile = '%s/%s.%s.gpg.verify.log' % (tmpDir, project, artifact)
- run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
- logFile)
- # Forward any GPG warnings, except the expected one (since it's a clean world)
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
- # Test trust (this is done with the real users config)
- run('gpg --import %s' % keysFile,
- '%s/%s.gpg.trust.import.log' % (tmpDir, project))
- logFile = '%s/%s.%s.gpg.trust.log' % (tmpDir, project, artifact)
- run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
- # Forward any GPG warnings:
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
+def verifyMavenSigs(tmpDir, artifacts, keysFile):
+ print(' verify maven artifact sigs', end=' ')
- sys.stdout.write('.')
+ # Set up clean gpg world; import keys file:
+ gpgHomeDir = '%s/lucene.gpg' % tmpDir
+ if os.path.exists(gpgHomeDir):
+ shutil.rmtree(gpgHomeDir)
+ os.makedirs(gpgHomeDir, 0o700)
+ run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
+ '%s/lucene.gpg.import.log' % tmpDir)
+
+ reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
+ for artifactFile in [a for a in artifacts if reArtifacts.search(a)]:
+ artifact = os.path.basename(artifactFile)
+ sigFile = '%s.asc' % artifactFile
+ # Test sig (this is done with a clean brand-new GPG world)
+ logFile = '%s/lucene.%s.gpg.verify.log' % (tmpDir, artifact)
+ run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
+ logFile)
+
+ # Forward any GPG warnings, except the expected one (since it's a clean world)
+ print_warnings_in_file(logFile)
+
+ # Test trust (this is done with the real users config)
+ run('gpg --import %s' % keysFile,
+ '%s/lucene.gpg.trust.import.log' % tmpDir)
+ logFile = '%s/lucene.%s.gpg.trust.log' % (tmpDir, artifact)
+ run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
+ # Forward any GPG warnings:
+ print_warnings_in_file(logFile)
+
+ sys.stdout.write('.')
print()
+
+def print_warnings_in_file(file):
+ f = open(file)
Review comment:
Thanks for factoring out a method here to remove the previous code duplication!
minor: `with open(file) as f:` could be a way to not need the explicit `f.close()` at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] cpoerschke commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
cpoerschke commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r727204711
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -141,21 +146,19 @@ def checkJARMetaData(desc, jarFile, gitRevision, version):
raise RuntimeError('%s is missing %s' % (desc, name))
except KeyError:
raise RuntimeError('%s is missing %s' % (desc, name))
-
+
s = decodeUTF8(z.read(MANIFEST_FILE_NAME))
-
+
for verify in (
'Specification-Vendor: The Apache Software Foundation',
'Implementation-Vendor: The Apache Software Foundation',
- # Make sure 1.8 compiler was used to build release bits:
+ 'Specification-Title: Lucene Search Engine:',
+ 'Implementation-Title: org.apache.lucene',
'X-Compile-Source-JDK: 11',
- # Make sure 1.8, 1.9 or 1.10 ant was used to build release bits: (this will match 1.8.x, 1.9.x, 1.10.x)
- ('Ant-Version: Apache Ant 1.8', 'Ant-Version: Apache Ant 1.9', 'Ant-Version: Apache Ant 1.10'),
- # Make sure .class files are 1.8 format:
Review comment:
`git grep -i ant` also finds some other ant reference e.g. line 70. could those be removed too maybe then?
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -1043,122 +823,81 @@ def getPOMcoordinate(treeRoot):
packaging = 'jar' if packaging is None else packaging.text.strip()
return groupId, artifactId, packaging, version
-def verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile):
- print(' verify maven artifact sigs', end=' ')
- for project in ('lucene', 'solr'):
-
- # Set up clean gpg world; import keys file:
- gpgHomeDir = '%s/%s.gpg' % (tmpDir, project)
- if os.path.exists(gpgHomeDir):
- shutil.rmtree(gpgHomeDir)
- os.makedirs(gpgHomeDir, 0o700)
- run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
- '%s/%s.gpg.import.log' % (tmpDir, project))
-
- reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
- for artifactFile in [a for a in artifacts[project] if reArtifacts.search(a)]:
- artifact = os.path.basename(artifactFile)
- sigFile = '%s.asc' % artifactFile
- # Test sig (this is done with a clean brand-new GPG world)
- logFile = '%s/%s.%s.gpg.verify.log' % (tmpDir, project, artifact)
- run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
- logFile)
- # Forward any GPG warnings, except the expected one (since it's a clean world)
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
- # Test trust (this is done with the real users config)
- run('gpg --import %s' % keysFile,
- '%s/%s.gpg.trust.import.log' % (tmpDir, project))
- logFile = '%s/%s.%s.gpg.trust.log' % (tmpDir, project, artifact)
- run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
- # Forward any GPG warnings:
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
+def verifyMavenSigs(tmpDir, artifacts, keysFile):
+ print(' verify maven artifact sigs', end=' ')
- sys.stdout.write('.')
+ # Set up clean gpg world; import keys file:
+ gpgHomeDir = '%s/lucene.gpg' % tmpDir
+ if os.path.exists(gpgHomeDir):
+ shutil.rmtree(gpgHomeDir)
+ os.makedirs(gpgHomeDir, 0o700)
+ run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
+ '%s/lucene.gpg.import.log' % tmpDir)
+
+ reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
+ for artifactFile in [a for a in artifacts if reArtifacts.search(a)]:
+ artifact = os.path.basename(artifactFile)
+ sigFile = '%s.asc' % artifactFile
+ # Test sig (this is done with a clean brand-new GPG world)
+ logFile = '%s/lucene.%s.gpg.verify.log' % (tmpDir, artifact)
+ run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
+ logFile)
+
+ # Forward any GPG warnings, except the expected one (since it's a clean world)
+ print_warnings_in_file(logFile)
+
+ # Test trust (this is done with the real users config)
+ run('gpg --import %s' % keysFile,
+ '%s/lucene.gpg.trust.import.log' % tmpDir)
+ logFile = '%s/lucene.%s.gpg.trust.log' % (tmpDir, artifact)
+ run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
+ # Forward any GPG warnings:
+ print_warnings_in_file(logFile)
+
+ sys.stdout.write('.')
print()
+
+def print_warnings_in_file(file):
+ f = open(file)
Review comment:
Thanks for factoring out a method here to remove the previous code duplication!
minor: `with open(file) as f:` could be a way to not need the explicit `f.close()` at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #355:
URL: https://github.com/apache/lucene/pull/355#issuecomment-941797977
I have started validating the smoke tester on the artifacts produced by the new build targets in #372
I find bugs in the gradle build, but also bugs in the smoketester :)
Once all lights are green, this should be ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #355:
URL: https://github.com/apache/lucene/pull/355#issuecomment-942298139
I intend to merge what we have so far, and then let others submit smaller followup PRs.
With the recent build improvements in #372 and the cleanup in wizard from #344, and this one, it should now be possible for anyone to attempt a full build and test the smoketester on real artifacts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #355:
URL: https://github.com/apache/lucene/pull/355#issuecomment-941984730
> I find bugs in the gradle build,
Please ping me if you do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #355:
URL: https://github.com/apache/lucene/pull/355#issuecomment-941990831
> Please ping me if you do!
Ping https://github.com/apache/lucene/pull/372#discussion_r727608870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy merged pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy merged pull request #355:
URL: https://github.com/apache/lucene/pull/355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] madrob commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725033367
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -900,42 +701,19 @@ def testDemo(run_java, isSrc, version, jdk):
if removeTrailingZeros(actualVersion) != removeTrailingZeros(version):
raise RuntimeError('wrong version from CheckIndex: got "%s" but expected "%s"' % (actualVersion, version))
+
def removeTrailingZeros(version):
return re.sub(r'(\.0)*$', '', version)
-def checkMaven(solrSrcUnpackPath, baseURL, tmpDir, gitRevision, version, isSigned, keysFile):
- POMtemplates = defaultdict()
- getPOMtemplates(solrSrcUnpackPath, POMtemplates, tmpDir)
- print(' download artifacts')
- artifacts = {'lucene': [], 'solr': []}
- for project in ('lucene', 'solr'):
- artifactsURL = '%s/%s/maven/org/apache/%s/' % (baseURL, project, project)
- targetDir = '%s/maven/org/apache/%s' % (tmpDir, project)
- if not os.path.exists(targetDir):
- os.makedirs(targetDir)
- crawl(artifacts[project], artifactsURL, targetDir)
- print()
- verifyPOMperBinaryArtifact(artifacts, version)
- verifyArtifactPerPOMtemplate(POMtemplates, artifacts, tmpDir, version)
- verifyMavenDigests(artifacts)
- checkJavadocAndSourceArtifacts(artifacts, version)
- verifyDeployedPOMsCoordinates(artifacts, version)
- if isSigned:
- verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile)
-
- distFiles = getBinaryDistFilesForMavenChecks(tmpDir, version, baseURL)
- checkIdenticalMavenArtifacts(distFiles, artifacts, version)
-
- checkAllJARs('%s/maven/org/apache/lucene' % tmpDir, 'lucene', gitRevision, version, tmpDir, baseURL)
- checkAllJARs('%s/maven/org/apache/solr' % tmpDir, 'solr', gitRevision, version, tmpDir, baseURL)
def getBinaryDistFilesForMavenChecks(tmpDir, version, baseURL):
# TODO: refactor distribution unpacking so that it only happens once per distribution per smoker run
distFiles = defaultdict()
- for project in ('lucene', 'solr'):
+ for project in ('lucene'):
Review comment:
for all of these, should we consider removing the for loop entirely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy merged pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy merged pull request #355:
URL: https://github.com/apache/lucene/pull/355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #355:
URL: https://github.com/apache/lucene/pull/355#issuecomment-941984730
> I find bugs in the gradle build,
Please ping me if you do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725401031
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -48,6 +48,7 @@
cygwin = platform.system().lower().startswith('cygwin')
cygwinWindowsRoot = os.popen('cygpath -w /').read().strip().replace('\\','/') if cygwin else ''
+
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on pull request #355:
URL: https://github.com/apache/lucene/pull/355#issuecomment-941797977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r727578265
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -1043,122 +823,81 @@ def getPOMcoordinate(treeRoot):
packaging = 'jar' if packaging is None else packaging.text.strip()
return groupId, artifactId, packaging, version
-def verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile):
- print(' verify maven artifact sigs', end=' ')
- for project in ('lucene', 'solr'):
-
- # Set up clean gpg world; import keys file:
- gpgHomeDir = '%s/%s.gpg' % (tmpDir, project)
- if os.path.exists(gpgHomeDir):
- shutil.rmtree(gpgHomeDir)
- os.makedirs(gpgHomeDir, 0o700)
- run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
- '%s/%s.gpg.import.log' % (tmpDir, project))
-
- reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
- for artifactFile in [a for a in artifacts[project] if reArtifacts.search(a)]:
- artifact = os.path.basename(artifactFile)
- sigFile = '%s.asc' % artifactFile
- # Test sig (this is done with a clean brand-new GPG world)
- logFile = '%s/%s.%s.gpg.verify.log' % (tmpDir, project, artifact)
- run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
- logFile)
- # Forward any GPG warnings, except the expected one (since it's a clean world)
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
- # Test trust (this is done with the real users config)
- run('gpg --import %s' % keysFile,
- '%s/%s.gpg.trust.import.log' % (tmpDir, project))
- logFile = '%s/%s.%s.gpg.trust.log' % (tmpDir, project, artifact)
- run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
- # Forward any GPG warnings:
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
+def verifyMavenSigs(tmpDir, artifacts, keysFile):
+ print(' verify maven artifact sigs', end=' ')
- sys.stdout.write('.')
+ # Set up clean gpg world; import keys file:
+ gpgHomeDir = '%s/lucene.gpg' % tmpDir
+ if os.path.exists(gpgHomeDir):
+ shutil.rmtree(gpgHomeDir)
+ os.makedirs(gpgHomeDir, 0o700)
+ run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
+ '%s/lucene.gpg.import.log' % tmpDir)
+
+ reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
+ for artifactFile in [a for a in artifacts if reArtifacts.search(a)]:
+ artifact = os.path.basename(artifactFile)
+ sigFile = '%s.asc' % artifactFile
+ # Test sig (this is done with a clean brand-new GPG world)
+ logFile = '%s/lucene.%s.gpg.verify.log' % (tmpDir, artifact)
+ run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
+ logFile)
+
+ # Forward any GPG warnings, except the expected one (since it's a clean world)
+ print_warnings_in_file(logFile)
+
+ # Test trust (this is done with the real users config)
+ run('gpg --import %s' % keysFile,
+ '%s/lucene.gpg.trust.import.log' % tmpDir)
+ logFile = '%s/lucene.%s.gpg.trust.log' % (tmpDir, artifact)
+ run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
+ # Forward any GPG warnings:
+ print_warnings_in_file(logFile)
+
+ sys.stdout.write('.')
print()
+
+def print_warnings_in_file(file):
+ f = open(file)
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r727577690
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -141,21 +146,19 @@ def checkJARMetaData(desc, jarFile, gitRevision, version):
raise RuntimeError('%s is missing %s' % (desc, name))
except KeyError:
raise RuntimeError('%s is missing %s' % (desc, name))
-
+
s = decodeUTF8(z.read(MANIFEST_FILE_NAME))
-
+
for verify in (
'Specification-Vendor: The Apache Software Foundation',
'Implementation-Vendor: The Apache Software Foundation',
- # Make sure 1.8 compiler was used to build release bits:
+ 'Specification-Title: Lucene Search Engine:',
+ 'Implementation-Title: org.apache.lucene',
'X-Compile-Source-JDK: 11',
- # Make sure 1.8, 1.9 or 1.10 ant was used to build release bits: (this will match 1.8.x, 1.9.x, 1.10.x)
- ('Ant-Version: Apache Ant 1.8', 'Ant-Version: Apache Ant 1.9', 'Ant-Version: Apache Ant 1.10'),
- # Make sure .class files are 1.8 format:
Review comment:
I fixed some more, including converting an ant command to gradle syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r726029721
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -900,42 +701,19 @@ def testDemo(run_java, isSrc, version, jdk):
if removeTrailingZeros(actualVersion) != removeTrailingZeros(version):
raise RuntimeError('wrong version from CheckIndex: got "%s" but expected "%s"' % (actualVersion, version))
+
def removeTrailingZeros(version):
return re.sub(r'(\.0)*$', '', version)
-def checkMaven(solrSrcUnpackPath, baseURL, tmpDir, gitRevision, version, isSigned, keysFile):
- POMtemplates = defaultdict()
- getPOMtemplates(solrSrcUnpackPath, POMtemplates, tmpDir)
- print(' download artifacts')
- artifacts = {'lucene': [], 'solr': []}
- for project in ('lucene', 'solr'):
- artifactsURL = '%s/%s/maven/org/apache/%s/' % (baseURL, project, project)
- targetDir = '%s/maven/org/apache/%s' % (tmpDir, project)
- if not os.path.exists(targetDir):
- os.makedirs(targetDir)
- crawl(artifacts[project], artifactsURL, targetDir)
- print()
- verifyPOMperBinaryArtifact(artifacts, version)
- verifyArtifactPerPOMtemplate(POMtemplates, artifacts, tmpDir, version)
- verifyMavenDigests(artifacts)
- checkJavadocAndSourceArtifacts(artifacts, version)
- verifyDeployedPOMsCoordinates(artifacts, version)
- if isSigned:
- verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile)
-
- distFiles = getBinaryDistFilesForMavenChecks(tmpDir, version, baseURL)
- checkIdenticalMavenArtifacts(distFiles, artifacts, version)
-
- checkAllJARs('%s/maven/org/apache/lucene' % tmpDir, 'lucene', gitRevision, version, tmpDir, baseURL)
- checkAllJARs('%s/maven/org/apache/solr' % tmpDir, 'solr', gitRevision, version, tmpDir, baseURL)
def getBinaryDistFilesForMavenChecks(tmpDir, version, baseURL):
# TODO: refactor distribution unpacking so that it only happens once per distribution per smoker run
distFiles = defaultdict()
- for project in ('lucene', 'solr'):
+ for project in ('lucene'):
Review comment:
Please review the latest commit 0748a7d6092af26c1be0556bef4c749ad040f813 where I have removed the unnecessary for loops with only 'lucene', and also removed 'project' from several functions. I think it justifies a separate review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725396122
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -900,42 +701,19 @@ def testDemo(run_java, isSrc, version, jdk):
if removeTrailingZeros(actualVersion) != removeTrailingZeros(version):
raise RuntimeError('wrong version from CheckIndex: got "%s" but expected "%s"' % (actualVersion, version))
+
def removeTrailingZeros(version):
return re.sub(r'(\.0)*$', '', version)
-def checkMaven(solrSrcUnpackPath, baseURL, tmpDir, gitRevision, version, isSigned, keysFile):
- POMtemplates = defaultdict()
- getPOMtemplates(solrSrcUnpackPath, POMtemplates, tmpDir)
- print(' download artifacts')
- artifacts = {'lucene': [], 'solr': []}
- for project in ('lucene', 'solr'):
- artifactsURL = '%s/%s/maven/org/apache/%s/' % (baseURL, project, project)
- targetDir = '%s/maven/org/apache/%s' % (tmpDir, project)
- if not os.path.exists(targetDir):
- os.makedirs(targetDir)
- crawl(artifacts[project], artifactsURL, targetDir)
- print()
- verifyPOMperBinaryArtifact(artifacts, version)
- verifyArtifactPerPOMtemplate(POMtemplates, artifacts, tmpDir, version)
- verifyMavenDigests(artifacts)
- checkJavadocAndSourceArtifacts(artifacts, version)
- verifyDeployedPOMsCoordinates(artifacts, version)
- if isSigned:
- verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile)
-
- distFiles = getBinaryDistFilesForMavenChecks(tmpDir, version, baseURL)
- checkIdenticalMavenArtifacts(distFiles, artifacts, version)
-
- checkAllJARs('%s/maven/org/apache/lucene' % tmpDir, 'lucene', gitRevision, version, tmpDir, baseURL)
- checkAllJARs('%s/maven/org/apache/solr' % tmpDir, 'solr', gitRevision, version, tmpDir, baseURL)
def getBinaryDistFilesForMavenChecks(tmpDir, version, baseURL):
# TODO: refactor distribution unpacking so that it only happens once per distribution per smoker run
distFiles = defaultdict()
- for project in ('lucene', 'solr'):
+ for project in ('lucene'):
Review comment:
The function was dead code, but then by the careful eye of @cpoerschke the validation of maven artifacts is back and targeting lucene only.
In this first iteration I just get rid of solr, but agree that we can collapse some loops and also get rid of some `project` function arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r727577690
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -141,21 +146,19 @@ def checkJARMetaData(desc, jarFile, gitRevision, version):
raise RuntimeError('%s is missing %s' % (desc, name))
except KeyError:
raise RuntimeError('%s is missing %s' % (desc, name))
-
+
s = decodeUTF8(z.read(MANIFEST_FILE_NAME))
-
+
for verify in (
'Specification-Vendor: The Apache Software Foundation',
'Implementation-Vendor: The Apache Software Foundation',
- # Make sure 1.8 compiler was used to build release bits:
+ 'Specification-Title: Lucene Search Engine:',
+ 'Implementation-Title: org.apache.lucene',
'X-Compile-Source-JDK: 11',
- # Make sure 1.8, 1.9 or 1.10 ant was used to build release bits: (this will match 1.8.x, 1.9.x, 1.10.x)
- ('Ant-Version: Apache Ant 1.8', 'Ant-Version: Apache Ant 1.9', 'Ant-Version: Apache Ant 1.10'),
- # Make sure .class files are 1.8 format:
Review comment:
I fixed some more, including converting an ant command to gradle syntax.
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -1043,122 +823,81 @@ def getPOMcoordinate(treeRoot):
packaging = 'jar' if packaging is None else packaging.text.strip()
return groupId, artifactId, packaging, version
-def verifyMavenSigs(baseURL, tmpDir, artifacts, keysFile):
- print(' verify maven artifact sigs', end=' ')
- for project in ('lucene', 'solr'):
-
- # Set up clean gpg world; import keys file:
- gpgHomeDir = '%s/%s.gpg' % (tmpDir, project)
- if os.path.exists(gpgHomeDir):
- shutil.rmtree(gpgHomeDir)
- os.makedirs(gpgHomeDir, 0o700)
- run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
- '%s/%s.gpg.import.log' % (tmpDir, project))
-
- reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
- for artifactFile in [a for a in artifacts[project] if reArtifacts.search(a)]:
- artifact = os.path.basename(artifactFile)
- sigFile = '%s.asc' % artifactFile
- # Test sig (this is done with a clean brand-new GPG world)
- logFile = '%s/%s.%s.gpg.verify.log' % (tmpDir, project, artifact)
- run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
- logFile)
- # Forward any GPG warnings, except the expected one (since it's a clean world)
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
- # Test trust (this is done with the real users config)
- run('gpg --import %s' % keysFile,
- '%s/%s.gpg.trust.import.log' % (tmpDir, project))
- logFile = '%s/%s.%s.gpg.trust.log' % (tmpDir, project, artifact)
- run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
- # Forward any GPG warnings:
- f = open(logFile)
- for line in f.readlines():
- if line.lower().find('warning') != -1 \
- and line.find('WARNING: This key is not certified with a trusted signature') == -1 \
- and line.find('WARNING: using insecure memory') == -1:
- print(' GPG: %s' % line.strip())
- f.close()
+def verifyMavenSigs(tmpDir, artifacts, keysFile):
+ print(' verify maven artifact sigs', end=' ')
- sys.stdout.write('.')
+ # Set up clean gpg world; import keys file:
+ gpgHomeDir = '%s/lucene.gpg' % tmpDir
+ if os.path.exists(gpgHomeDir):
+ shutil.rmtree(gpgHomeDir)
+ os.makedirs(gpgHomeDir, 0o700)
+ run('gpg --homedir %s --import %s' % (gpgHomeDir, keysFile),
+ '%s/lucene.gpg.import.log' % tmpDir)
+
+ reArtifacts = re.compile(r'\.(?:pom|[jw]ar)$')
+ for artifactFile in [a for a in artifacts if reArtifacts.search(a)]:
+ artifact = os.path.basename(artifactFile)
+ sigFile = '%s.asc' % artifactFile
+ # Test sig (this is done with a clean brand-new GPG world)
+ logFile = '%s/lucene.%s.gpg.verify.log' % (tmpDir, artifact)
+ run('gpg --homedir %s --verify %s %s' % (gpgHomeDir, sigFile, artifactFile),
+ logFile)
+
+ # Forward any GPG warnings, except the expected one (since it's a clean world)
+ print_warnings_in_file(logFile)
+
+ # Test trust (this is done with the real users config)
+ run('gpg --import %s' % keysFile,
+ '%s/lucene.gpg.trust.import.log' % tmpDir)
+ logFile = '%s/lucene.%s.gpg.trust.log' % (tmpDir, artifact)
+ run('gpg --verify %s %s' % (sigFile, artifactFile), logFile)
+ # Forward any GPG warnings:
+ print_warnings_in_file(logFile)
+
+ sys.stdout.write('.')
print()
+
+def print_warnings_in_file(file):
+ f = open(file)
Review comment:
Done
##########
File path: lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
##########
@@ -175,7 +175,7 @@ public static void main(String[] args) throws Exception {
+ " documents in "
+ (end.getTime() - start.getTime())
+ " milliseconds");
- if (reader.numDocs() > 100 && vectorDictSize < 1_000_000) {
+ if (reader.numDocs() > 100 && vectorDictSize < 1_000_000 && System.getProperty("smoketester") == null) {
throw new RuntimeException(
"Are you (ab)using the toy vector dictionary? See the package javadocs to understand why you got this exception.");
}
Review comment:
The demo indexer failed due to this exception, so I add a 'smoketester' mode to it that will allow "abuse".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725350490
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -658,97 +616,41 @@ def verifyUnpacked(java, project, artifact, unpackPath, gitRevision, version, te
print(' run "%s"' % validateCmd)
java.run_java11(validateCmd, '%s/validate.log' % unpackPath)
- if project == 'lucene':
- print(" run tests w/ Java 11 and testArgs='%s'..." % testArgs)
- java.run_java11('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
- java.run_java11('ant jar', '%s/compile.log' % unpackPath)
- testDemo(java.run_java11, isSrc, version, '11')
-
- print(' generate javadocs w/ Java 11...')
- java.run_java11('ant javadocs', '%s/javadocs.log' % unpackPath)
- checkBrokenLinks('%s/build/docs' % unpackPath)
-
- if java.run_java12:
- print(" run tests w/ Java 12 and testArgs='%s'..." % testArgs)
- java.run_java12('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
- java.run_java12('ant jar', '%s/compile.log' % unpackPath)
- testDemo(java.run_java12, isSrc, version, '12')
-
- #print(' generate javadocs w/ Java 12...')
- #java.run_java12('ant javadocs', '%s/javadocs.log' % unpackPath)
- #checkBrokenLinks('%s/build/docs' % unpackPath)
-
- else:
- os.chdir('solr')
-
- print(" run tests w/ Java 11 and testArgs='%s'..." % testArgs)
- java.run_java11('ant clean test -Dtests.slow=false %s' % testArgs, '%s/test.log' % unpackPath)
-
- # test javadocs
- print(' generate javadocs w/ Java 11...')
- java.run_java11('ant clean javadocs', '%s/javadocs.log' % unpackPath)
- checkBrokenLinks('%s/solr/build/docs')
-
- print(' test solr example w/ Java 11...')
- java.run_java11('ant clean server', '%s/antexample.log' % unpackPath)
- testSolrExample(unpackPath, java.java11_home, True)
+ print(" run tests w/ Java 11 and testArgs='%s'..." % testArgs)
+ java.run_java11('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
+ java.run_java11('ant jar', '%s/compile.log' % unpackPath)
+ testDemo(java.run_java11, isSrc, version, '11')
- if java.run_java12:
- print(" run tests w/ Java 12 and testArgs='%s'..." % testArgs)
- java.run_java12('ant clean test -Dtests.slow=false %s' % testArgs, '%s/test.log' % unpackPath)
+ print(' generate javadocs w/ Java 11...')
+ java.run_java11('ant javadocs', '%s/javadocs.log' % unpackPath)
+ checkBrokenLinks('%s/build/docs' % unpackPath)
- #print(' generate javadocs w/ Java 12...')
- #java.run_java12('ant clean javadocs', '%s/javadocs.log' % unpackPath)
- #checkBrokenLinks('%s/solr/build/docs' % unpackPath)
+ if java.run_java12:
+ print(" run tests w/ Java 12 and testArgs='%s'..." % testArgs)
+ java.run_java12('ant clean test %s' % testArgs, '%s/test.log' % unpackPath)
+ java.run_java12('ant jar', '%s/compile.log' % unpackPath)
+ testDemo(java.run_java12, isSrc, version, '12')
- print(' test solr example w/ Java 12...')
- java.run_java12('ant clean server', '%s/antexample.log' % unpackPath)
- testSolrExample(unpackPath, java.java12_home, True)
-
- os.chdir('..')
- print(' check NOTICE')
- testNotice(unpackPath)
+ #print(' generate javadocs w/ Java 12...')
+ #java.run_java12('ant javadocs', '%s/javadocs.log' % unpackPath)
+ #checkBrokenLinks('%s/build/docs' % unpackPath)
else:
checkAllJARs(os.getcwd(), project, gitRevision, version, tmpDir, baseURL)
- if project == 'lucene':
- testDemo(java.run_java11, isSrc, version, '11')
- if java.run_java12:
- testDemo(java.run_java12, isSrc, version, '12')
-
- else:
- print(' copying unpacked distribution for Java 11 ...')
- java11UnpackPath = '%s-java11' % unpackPath
- if os.path.exists(java11UnpackPath):
- shutil.rmtree(java11UnpackPath)
- shutil.copytree(unpackPath, java11UnpackPath)
- os.chdir(java11UnpackPath)
- print(' test solr example w/ Java 11...')
- testSolrExample(java11UnpackPath, java.java11_home, False)
-
- if java.run_java12:
- print(' copying unpacked distribution for Java 12 ...')
- java12UnpackPath = '%s-java12' % unpackPath
- if os.path.exists(java12UnpackPath):
- shutil.rmtree(java12UnpackPath)
- shutil.copytree(unpackPath, java12UnpackPath)
- os.chdir(java12UnpackPath)
- print(' test solr example w/ Java 12...')
- testSolrExample(java12UnpackPath, java.java12_home, False)
-
- os.chdir(unpackPath)
+ testDemo(java.run_java11, isSrc, version, '11')
+ if java.run_java12:
+ testDemo(java.run_java12, isSrc, version, '12')
testChangesText('.', version, project)
- if project == 'lucene' and isSrc:
+ if isSrc:
print(' confirm all releases have coverage in TestBackwardsCompatibility')
confirmAllReleasesAreTestedForBackCompat(version, unpackPath)
def testNotice(unpackPath):
- solrNotice = open('%s/NOTICE.txt' % unpackPath, encoding='UTF-8').read()
luceneNotice = open('%s/lucene/NOTICE.txt' % unpackPath, encoding='UTF-8').read()
expected = """
Review comment:
You are right. I only partly removed it. What the test does is to verify that Lucene's NOTICE file is contained in Solr's NOTICE file. Removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] janhoy commented on a change in pull request #355: LUCENE-9997 Revisit smoketester for 9.0 build
Posted by GitBox <gi...@apache.org>.
janhoy commented on a change in pull request #355:
URL: https://github.com/apache/lucene/pull/355#discussion_r725914818
##########
File path: dev-tools/scripts/smokeTestRelease.py
##########
@@ -173,11 +173,15 @@ def checkJARMetaData(desc, jarFile, gitRevision, version):
if gitRevision != 'skip':
# Make sure this matches the version and git revision we think we are releasing:
- # TODO: LUCENE-7023: is it OK that Implementation-Version's value now spans two lines?
- verifyRevision = 'Implementation-Version: %s %s' % (version, gitRevision)
- if s.find(verifyRevision) == -1:
- raise RuntimeError('%s is missing "%s" inside its META-INF/MANIFEST.MF (wrong git revision?)' % \
+ match = re.search("Implementation-Version: (.+\r\n .+)", s, re.MULTILINE)
Review comment:
I ported this change from branch_8x even if that was targeted to 8.x only (don't know why), since we'll end up in same situation with 10.x some time..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org