You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/05/01 16:28:06 UTC
[GitHub] [lucene-solr] uschindler opened a new pull request #1477: LUCENE-9321: Port markdown task to Gradle
uschindler opened a new pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477
This PR handles the markdown specific parts of the Lucene/Solr documentation:
- copy static assets (unfortunetely those are not yet unified between borth projects)
- convert .md files to HTML using flexmark
- create the index files by first iterating through the projects and collecting all links, creating a markdown representation of all links locally and then write as index.html to docus folder
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425774732
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import groovy.text.SimpleTemplateEngine;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex(type: MarkdownTemplateTask) {
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+ inputs.file(defaultCodecFile)
+
+ outputFile = file("${project.docroot}/index.html")
+ templateFile = file('site/xsl/index.template.md')
+
+ binding = {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def majorVersion = project.version.split(/\./)[0] as int;
+ return [
+ defaultCodecPackage : matcher.group(1).toLowerCase(Locale.ROOT),
+ version : project.version,
+ majorVersion : majorVersion,
+ projectList : getListOfProjectsAsMarkdown(':lucene:')
+ ]
+ }
+ }
+}
+
+configure(project(':solr')) {
+ markdownToHtml {
+ from('site') {
+ include '**/*.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+// filter that can be used with the "copy" task of Gradle that transforms Markdown files
+// from source location to HTML (adding HTML header, styling,...)
+class MarkdownFilter extends FilterReader {
+
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is not really a filter: it reads whole file in ctor,
+ // converts it and provides result downstream as a StringReader
+ super(new StringReader(convert(reader.text)));
+ }
+
+ public static String convert(String markdownSource) {
+ // first replace LUCENE and SOLR issue numbers with a markdown link
+ markdownSource = markdownSource.replaceAll(/(?s)\b(LUCENE|SOLR)\-\d+\b/,
+ '[$0](https://issues.apache.org/jira/browse/$0)');
+
+ // convert the markdown
+ MutableDataSet options = new MutableDataSet();
+ options.setFrom(ParserEmulationProfile.MARKDOWN);
+ options.set(Parser.EXTENSIONS, [ AbbreviationExtension.create(), AutolinkExtension.create(), AttributesExtension.create() ]);
+ options.set(HtmlRenderer.RENDER_HEADER_ID, true);
+ options.set(HtmlRenderer.MAX_TRAILING_BLANK_LINES, 0);
+ Document parsed = Parser.builder(options).build().parse(markdownSource);
+
+ StringBuilder html = new StringBuilder('<html>\n<head>\n');
Review comment:
> I would add CSS in a separate issue.
Yes I think so too. I will open separate issue for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r418783353
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ def flexmarkVersion = '0.61.24'
+
+ classpath 'com.vladsch.flexmark:flexmark:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-autolink:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-abbreviation:' + flexmarkVersion
+ }
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ from('.') { // lucene
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ from('site') { // solr
+ include '**/*.md'
+ }
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+class MarkdownFilter extends FilterReader {
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is a hack: it reads whole file, converts it and provides result as a StringReader
Review comment:
It looks like misuse of a FilterReader, because you do everything in the constructor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622650749
I am not sure if it's worth here, "render-javadoc.gradle" has a utility method to convert project path into output doc folder path: https://github.com/apache/lucene-solr/blob/master/gradle/render-javadoc.gradle#L21
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425286729
##########
File path: gradle/documentation/documentation.gradle
##########
@@ -34,4 +36,11 @@ configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
ext {
docroot = "${project.buildDir}/documentation"
}
+
+ task copyDocumentationAssets(type: Copy) {
+ includeEmptyDirs = false
+ from('site/html') // lucene
Review comment:
I did this for the main task. At this place we can also do it, but that's less to maintain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425375832
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ def outputFile = file("${project.docroot}/index.html");
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+
+ inputs.file(defaultCodecFile)
+ outputs.file(outputFile)
+
+ doLast {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def defaultCodecPackage = matcher.group(1).toLowerCase(Locale.ROOT)
+ def markdown = """
Review comment:
https://docs.groovy-lang.org/latest/html/api/groovy/text/SimpleTemplateEngine.html
Should work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425371129
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ def outputFile = file("${project.docroot}/index.html");
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+
+ inputs.file(defaultCodecFile)
+ outputs.file(outputFile)
+
+ doLast {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def defaultCodecPackage = matcher.group(1).toLowerCase(Locale.ROOT)
+ def markdown = """
Review comment:
That was my question. Can we include it in a way so the groovy variables are parsed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622927763
Hi, thanks for the discussion. @mocobeta explains it correct: You just have to change one thing to change the destination dir in the render-javadoc.
> I was not aware of that this task depends on projects' relative paths. To me, before proceeding it we need to manage to reach a consensus about the destination (output) directory for "renderJavadoc" anyway...?
Actually this is my main concern. I will comment on this on the issue. I have an idea for that. The reason for the issue is because there is conflicting interests: javadocs-jar on Maven central vs. documentation folder on web site and inside tar.gz of whole Lucene bundle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628832740
The title and metadata of a project should IMHO be part of project's gradle file. You just add a `description = "..."` next to the dependencies, an done. Is this OK to you, or do you want them central?
In Ant this information is part of the root element of the `build.xml`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629104491
It's very clean and nice with external file. I love it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622879655
Thanks for clarifying, Tomoko. I'm much in favor of keeping the javadocs in per-project build folders but if Uwe insists this is a problem then would it be a large patch to build those docs under target documentation location?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r418783252
##########
File path: gradle/documentation/documentation.gradle
##########
@@ -34,4 +36,11 @@ configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
ext {
docroot = "${project.buildDir}/documentation"
}
+
+ task copyDocumentationAssets(type: Copy) {
+ includeEmptyDirs = false
+ from('site/html') // lucene
Review comment:
I was thinking about that too, but this was easier to begin with. I did not want to clone the whole "Copy" task several times. If you tell me how to do this easier, give me a hint!
Should I remove the `from` here and then add 2 separate configures for lucene and solr just adding the `from`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629632183
I also figured out that the assets folder in Solr's site contains some files from the getting started guide which was deleted a while ago. I think we should delete all that, except logo and maybe css.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler edited a comment on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler edited a comment on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622931376
See my lengthly comment here: https://issues.apache.org/jira/browse/LUCENE-9321?focusedCommentId=17097899&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17097899
Please read it fully concentrated!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425355108
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ def outputFile = file("${project.docroot}/index.html");
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+
+ inputs.file(defaultCodecFile)
+ outputs.file(outputFile)
+
+ doLast {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def defaultCodecPackage = matcher.group(1).toLowerCase(Locale.ROOT)
+ def markdown = """
Review comment:
Should we move it to an external file? Then we can have a proper suffix and editor support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425749526
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import groovy.text.SimpleTemplateEngine;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex(type: MarkdownTemplateTask) {
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+ inputs.file(defaultCodecFile)
+
+ outputFile = file("${project.docroot}/index.html")
+ templateFile = file('site/xsl/index.template.md')
+
+ binding = {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def majorVersion = project.version.split(/\./)[0] as int;
+ return [
+ defaultCodecPackage : matcher.group(1).toLowerCase(Locale.ROOT),
+ version : project.version,
+ majorVersion : majorVersion,
+ projectList : getListOfProjectsAsMarkdown(':lucene:')
+ ]
+ }
+ }
+}
+
+configure(project(':solr')) {
+ markdownToHtml {
+ from('site') {
+ include '**/*.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+// filter that can be used with the "copy" task of Gradle that transforms Markdown files
+// from source location to HTML (adding HTML header, styling,...)
+class MarkdownFilter extends FilterReader {
+
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is not really a filter: it reads whole file in ctor,
+ // converts it and provides result downstream as a StringReader
+ super(new StringReader(convert(reader.text)));
+ }
+
+ public static String convert(String markdownSource) {
+ // first replace LUCENE and SOLR issue numbers with a markdown link
+ markdownSource = markdownSource.replaceAll(/(?s)\b(LUCENE|SOLR)\-\d+\b/,
+ '[$0](https://issues.apache.org/jira/browse/$0)');
+
+ // convert the markdown
+ MutableDataSet options = new MutableDataSet();
+ options.setFrom(ParserEmulationProfile.MARKDOWN);
+ options.set(Parser.EXTENSIONS, [ AbbreviationExtension.create(), AutolinkExtension.create(), AttributesExtension.create() ]);
+ options.set(HtmlRenderer.RENDER_HEADER_ID, true);
+ options.set(HtmlRenderer.MAX_TRAILING_BLANK_LINES, 0);
+ Document parsed = Parser.builder(options).build().parse(markdownSource);
+
+ StringBuilder html = new StringBuilder('<html>\n<head>\n');
Review comment:
Would it be better inserting `<!DOCTYPE html>` at the beginning (as a valid html5)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler merged pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler merged pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628820984
bq. Should I simple add a project.description to all .gradle files, so it's consistent with Ant?
Hmm... What do you mean by adding project.description? We don't need to have per-project build.gradle file (for example if there are no dependencies). If you're thinking of a project property to hold the description then this could be configured from a single file too -- whatever is more convenient, really. I guess this description will come in handy for maven artifact (pom) as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629630945
Hi,
I think all is done now. The documentation is generated for both Solr and Lucene.
For Solr this also generates the "minimal" documentation folder as zipped in the release distribution, which just contains a link to the online docs (`solr/build/docs-online`). When zipping we have to make sure to package the right stuff (see ant task).
I think this is ready for merge. I am running the whole stuff one more time with empty build dir.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628742753
Hi, I added a first implemntation of `index.html`: 8dca5bb (Lucene only)
It works quite well:
- I added a method that collects all sub-projects with a specific path prefix (`:lucene:`) that have javadocs
- The list is then formatted as markdown (identical to the XSL). The only missing thing is the project description (we have it in Ant, but it's misisng in Gradle)
- The top part of the documentation was also converted to Markdown and is using project variabales with version numbers and the default codec
- default codec is extarcted like with ant, just simpler
- inputs/output of task are trivial
Open questions:
- Should I simple add a `project.description` to all `.gradle` files, so it's consistent with Ant?
- Where to place the 'templated' markdown source code. The XSL is currently a separate file. But as markdown contains Groovy `${...}` I would like to leave it in the gradle file.
Here is how it looks:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622489584
For now I have no idea how to handle the last task, so I left it as nocommit. @mocobeta: any ideas how to get the relative path and the title of all projects below lucene/solr? I am also not sure if the links hardcoded as absolute links in `render-javadoc.gradle` are consistent with the intended folder structure below `build/documentation`.
These are all open questions and before a decission how to build the f*cking javadocs wthout copying all stuff to another folder (this costs imense amounts of disk space) I refuse to work on that issue/PR - sorry! @dweiss?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629638174
I just run `gradlew documentation` on linux PC and manually checked the two index.html files and all hyperlinks; works fine for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622931376
See my langthly comment here: https://issues.apache.org/jira/browse/LUCENE-9321?focusedCommentId=17097899&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17097899
Please read it fully concentrated!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628834560
I don't mind either way. Can be per-project gradle file. Seems to fit there nicely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629789289
Hi,
I split the documentation tasks of Lucene and Solr, so you can also just build lucene or just solr documentation (still building all dependencies in a correct way).
This helps with the documentation linter (see #1522), as it can depend on solr/lucene specific documentation.
On top-level we still have the documentation, task too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629178396
I now added a task, so it can be reused in Solr. In solr we have 2 variants: One HTML files that's shipped in ZIP/TGZ and the full blown one on the web.
I also added more variables to the markdown: I added a binding for `majorVersion`. This one is used to format the line behind the Migration Guide: "What changed in Lucene 9, how to update 8.x code".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622866628
Just for clarification... Eventually, the `build/documentation` folder should look like this regardless of which way we choose - keep per-project javadoc outputs, or outputs docs into top-level `documentation` folder from the beginning:
**Lucene**
```
lucene/build/documentation
├── JRE_VERSION_MIGRATION.html
├── MIGRATE.html
├── SYSTEM_REQUIREMENTS.html
├── index.html
├── lucene_green_300.gif
├── analysis
│ ├── common
│ ├── icu
│ ├── kuromoji
│ ├── morfologik
│ ├── nori
│ ├── opennlp
│ ├── phonetic
│ ├── smartcn
│ └── stempel
├── backward-codecs
├── benchmark
├── classification
├── codecs
├── core
├── demo
├── expressions
├── facet
├── grouping
├── highlighter
├── join
├── luke
├── memory
├── misc
├── monitor
├── queries
├── queryparser
├── replicator
├── sandbox
├── spatial-extras
├── spatial3d
├── suggest
└── test-framework
```
**Solr**
```
solr/build/documentation
├── SYSTEM_REQUIREMENTS.html
├── index.html
├── contrib
│ ├── analysis-extras
│ ├── analytics
│ ├── clustering
│ ├── dataimporthandler
│ ├── dataimporthandler-extras
│ ├── extraction
│ ├── jaegertracer-configurator
│ ├── langid
│ ├── ltr
│ ├── prometheus-exporter
│ └── velocity
├── core
├── solrj
└── test-framework
```
The each subproject's javadoc folder structure is consistent with Gradle project path (as I emphasized on LUCENE-9278). Both `build/documentation` folder should be uploaded to lucene.apache.org website on an as-is basis (it's the final purpose of the `documentation` task).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r418783252
##########
File path: gradle/documentation/documentation.gradle
##########
@@ -34,4 +36,11 @@ configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
ext {
docroot = "${project.buildDir}/documentation"
}
+
+ task copyDocumentationAssets(type: Copy) {
+ includeEmptyDirs = false
+ from('site/html') // lucene
Review comment:
Here it's easy to separate, will do!
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ def flexmarkVersion = '0.61.24'
+
+ classpath 'com.vladsch.flexmark:flexmark:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-autolink:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-abbreviation:' + flexmarkVersion
+ }
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ from('.') { // lucene
Review comment:
I was thinking about that too, but this was easier to begin with. I did not want to clone the whole "Copy" task several times. If you tell me how to do this easier, give me a hint!
Should I remove the `from` here and then add 2 separate configures for lucene and solr just adding the `from`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425763977
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import groovy.text.SimpleTemplateEngine;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex(type: MarkdownTemplateTask) {
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+ inputs.file(defaultCodecFile)
+
+ outputFile = file("${project.docroot}/index.html")
+ templateFile = file('site/xsl/index.template.md')
+
+ binding = {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def majorVersion = project.version.split(/\./)[0] as int;
+ return [
+ defaultCodecPackage : matcher.group(1).toLowerCase(Locale.ROOT),
+ version : project.version,
+ majorVersion : majorVersion,
+ projectList : getListOfProjectsAsMarkdown(':lucene:')
+ ]
+ }
+ }
+}
+
+configure(project(':solr')) {
+ markdownToHtml {
+ from('site') {
+ include '**/*.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+// filter that can be used with the "copy" task of Gradle that transforms Markdown files
+// from source location to HTML (adding HTML header, styling,...)
+class MarkdownFilter extends FilterReader {
+
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is not really a filter: it reads whole file in ctor,
+ // converts it and provides result downstream as a StringReader
+ super(new StringReader(convert(reader.text)));
+ }
+
+ public static String convert(String markdownSource) {
+ // first replace LUCENE and SOLR issue numbers with a markdown link
+ markdownSource = markdownSource.replaceAll(/(?s)\b(LUCENE|SOLR)\-\d+\b/,
+ '[$0](https://issues.apache.org/jira/browse/$0)');
+
+ // convert the markdown
+ MutableDataSet options = new MutableDataSet();
+ options.setFrom(ParserEmulationProfile.MARKDOWN);
+ options.set(Parser.EXTENSIONS, [ AbbreviationExtension.create(), AutolinkExtension.create(), AttributesExtension.create() ]);
+ options.set(HtmlRenderer.RENDER_HEADER_ID, true);
+ options.set(HtmlRenderer.MAX_TRAILING_BLANK_LINES, 0);
+ Document parsed = Parser.builder(options).build().parse(markdownSource);
+
+ StringBuilder html = new StringBuilder('<html>\n<head>\n');
Review comment:
Applying CSS was in my mind. I will add the tag later when I add the style file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425384992
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ def outputFile = file("${project.docroot}/index.html");
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+
+ inputs.file(defaultCodecFile)
+ outputs.file(outputFile)
+
+ doLast {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def defaultCodecPackage = matcher.group(1).toLowerCase(Locale.ROOT)
+ def markdown = """
Review comment:
Cool. So I will place the file next to the legacy XSL file used by Ant. Once Ant is retired, the latter can go away.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628929672
I pushed a first mockup using Groovy's `SimpleTemplateEngine`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425769125
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import groovy.text.SimpleTemplateEngine;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex(type: MarkdownTemplateTask) {
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+ inputs.file(defaultCodecFile)
+
+ outputFile = file("${project.docroot}/index.html")
+ templateFile = file('site/xsl/index.template.md')
+
+ binding = {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def majorVersion = project.version.split(/\./)[0] as int;
+ return [
+ defaultCodecPackage : matcher.group(1).toLowerCase(Locale.ROOT),
+ version : project.version,
+ majorVersion : majorVersion,
+ projectList : getListOfProjectsAsMarkdown(':lucene:')
+ ]
+ }
+ }
+}
+
+configure(project(':solr')) {
+ markdownToHtml {
+ from('site') {
+ include '**/*.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+// filter that can be used with the "copy" task of Gradle that transforms Markdown files
+// from source location to HTML (adding HTML header, styling,...)
+class MarkdownFilter extends FilterReader {
+
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is not really a filter: it reads whole file in ctor,
+ // converts it and provides result downstream as a StringReader
+ super(new StringReader(convert(reader.text)));
+ }
+
+ public static String convert(String markdownSource) {
+ // first replace LUCENE and SOLR issue numbers with a markdown link
+ markdownSource = markdownSource.replaceAll(/(?s)\b(LUCENE|SOLR)\-\d+\b/,
+ '[$0](https://issues.apache.org/jira/browse/$0)');
+
+ // convert the markdown
+ MutableDataSet options = new MutableDataSet();
+ options.setFrom(ParserEmulationProfile.MARKDOWN);
+ options.set(Parser.EXTENSIONS, [ AbbreviationExtension.create(), AutolinkExtension.create(), AttributesExtension.create() ]);
+ options.set(HtmlRenderer.RENDER_HEADER_ID, true);
+ options.set(HtmlRenderer.MAX_TRAILING_BLANK_LINES, 0);
+ Document parsed = Parser.builder(options).build().parse(markdownSource);
+
+ StringBuilder html = new StringBuilder('<html>\n<head>\n');
Review comment:
I would add CSS in a separate issue. I just want to get everything running first. What do you think about current state?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r418669624
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ def flexmarkVersion = '0.61.24'
+
+ classpath 'com.vladsch.flexmark:flexmark:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-autolink:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-abbreviation:' + flexmarkVersion
+ }
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ from('.') { // lucene
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ from('site') { // solr
+ include '**/*.md'
+ }
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+class MarkdownFilter extends FilterReader {
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is a hack: it reads whole file, converts it and provides result as a StringReader
Review comment:
Looks legit to me, not a hack? :)
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ def flexmarkVersion = '0.61.24'
+
+ classpath 'com.vladsch.flexmark:flexmark:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-autolink:' + flexmarkVersion
+ classpath 'com.vladsch.flexmark:flexmark-ext-abbreviation:' + flexmarkVersion
+ }
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ from('.') { // lucene
Review comment:
Right... same here.
##########
File path: gradle/documentation/documentation.gradle
##########
@@ -34,4 +36,11 @@ configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
ext {
docroot = "${project.buildDir}/documentation"
}
+
+ task copyDocumentationAssets(type: Copy) {
+ includeEmptyDirs = false
+ from('site/html') // lucene
Review comment:
I'd rather have it as separate configure(":lucene"), configure(":solr") blocks... would be easier to separate the logic between the two?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta edited a comment on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta edited a comment on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629638174
I just ran `gradlew documentation` (snapshot build) on linux PC and manually checked the two index.html files and all hyperlinks; works fine for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-629210291
I added `description` metadata for root, core, analysis projects. Looks like it works as expected. I will proceed when coming back (out of office now):
- open build.xml and build.gradle in parallel
- add `description = ''` line
- copypaste stuff from build.xml's `<description/>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628833244
> I guess this description will come in handy for maven artifact (pom) as well.
Yes, there it's public and searchable in e.g., search.maven.org
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622650279
I was not aware of that this task depends on projects' relative paths. To me, before proceeding it we need to manage to reach a consensus about the destination (output) directory for "renderJavadoc" anyway...?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-628571533
I can now proceed with this one as #1488 is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] uschindler commented on a change in pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
uschindler commented on a change in pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#discussion_r425756436
##########
File path: gradle/documentation/markdown.gradle
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.vladsch.flexmark.ast.Heading;
+import com.vladsch.flexmark.ext.abbreviation.AbbreviationExtension;
+import com.vladsch.flexmark.ext.attributes.AttributesExtension;
+import com.vladsch.flexmark.ext.autolink.AutolinkExtension;
+import com.vladsch.flexmark.html.HtmlRenderer;
+import com.vladsch.flexmark.parser.Parser;
+import com.vladsch.flexmark.parser.ParserEmulationProfile;
+import com.vladsch.flexmark.util.ast.Document;
+import com.vladsch.flexmark.util.data.MutableDataSet;
+import com.vladsch.flexmark.util.sequence.Escaping;
+import groovy.text.SimpleTemplateEngine;
+
+buildscript {
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+ classpath "com.vladsch.flexmark:flexmark:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-abbreviation:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-attributes:${scriptDepVersions['flexmark']}"
+ classpath "com.vladsch.flexmark:flexmark-ext-autolink:${scriptDepVersions['flexmark']}"
+ }
+}
+
+def getListOfProjectsAsMarkdown = { prefix ->
+ def projects = allprojects.findAll{ it.path.startsWith(prefix) && it.tasks.findByName('renderSiteJavadoc') }
+ .sort(false, Comparator.comparing{ (it.name != 'core') as Boolean }
+ .thenComparing(Comparator.comparing{ (it.name == 'test-framework') as Boolean })
+ .thenComparing(Comparator.comparing{ it.path }));
+ return projects.collect{ project ->
+ def text = "**[${project.path.substring(prefix.length()).replace(':','-')}](${project.relativeDocPath}/index.html):** ${project.description}"
+ if (project.name == 'core') {
+ text = text.concat(' {style="font-size:larger; margin-bottom:.5em"}')
+ }
+ return '* ' + text;
+ }.join('\n')
+}
+
+configure(subprojects.findAll { it.path == ':lucene' || it.path == ':solr' }) {
+ task markdownToHtml(type: Copy) {
+ filteringCharset = 'UTF-8'
+ includeEmptyDirs = false
+ into project.docroot
+ rename(/\.md$/, '.html')
+ filter(MarkdownFilter)
+ }
+}
+
+configure(project(':lucene')) {
+ markdownToHtml {
+ from('.') {
+ include 'MIGRATE.md'
+ include 'JRE_VERSION_MIGRATION.md'
+ include 'SYSTEM_REQUIREMENTS.md'
+ }
+ }
+
+ task createDocumentationIndex(type: MarkdownTemplateTask) {
+ def defaultCodecFile = project(':lucene:core').file('src/java/org/apache/lucene/codecs/Codec.java')
+ inputs.file(defaultCodecFile)
+
+ outputFile = file("${project.docroot}/index.html")
+ templateFile = file('site/xsl/index.template.md')
+
+ binding = {
+ // static Codec defaultCodec = LOADER . lookup ( "LuceneXXX" ) ;
+ def regex = ~/\bdefaultCodec\s*=\s*LOADER\s*\.\s*lookup\s*\(\s*"([^"]+)"\s*\)\s*;/
+ def matcher = regex.matcher(defaultCodecFile.getText('UTF-8'))
+ if (!matcher.find()) {
+ throw GradleException("Cannot determine default codec from file ${defaultCodecFile}")
+ }
+ def majorVersion = project.version.split(/\./)[0] as int;
+ return [
+ defaultCodecPackage : matcher.group(1).toLowerCase(Locale.ROOT),
+ version : project.version,
+ majorVersion : majorVersion,
+ projectList : getListOfProjectsAsMarkdown(':lucene:')
+ ]
+ }
+ }
+}
+
+configure(project(':solr')) {
+ markdownToHtml {
+ from('site') {
+ include '**/*.md'
+ }
+ }
+
+ task createDocumentationIndex {
+ // nocommit: this needs to be implemented next
+ }
+}
+
+// filter that can be used with the "copy" task of Gradle that transforms Markdown files
+// from source location to HTML (adding HTML header, styling,...)
+class MarkdownFilter extends FilterReader {
+
+ public MarkdownFilter(Reader reader) throws IOException {
+ // this is not really a filter: it reads whole file in ctor,
+ // converts it and provides result downstream as a StringReader
+ super(new StringReader(convert(reader.text)));
+ }
+
+ public static String convert(String markdownSource) {
+ // first replace LUCENE and SOLR issue numbers with a markdown link
+ markdownSource = markdownSource.replaceAll(/(?s)\b(LUCENE|SOLR)\-\d+\b/,
+ '[$0](https://issues.apache.org/jira/browse/$0)');
+
+ // convert the markdown
+ MutableDataSet options = new MutableDataSet();
+ options.setFrom(ParserEmulationProfile.MARKDOWN);
+ options.set(Parser.EXTENSIONS, [ AbbreviationExtension.create(), AutolinkExtension.create(), AttributesExtension.create() ]);
+ options.set(HtmlRenderer.RENDER_HEADER_ID, true);
+ options.set(HtmlRenderer.MAX_TRAILING_BLANK_LINES, 0);
+ Document parsed = Parser.builder(options).build().parse(markdownSource);
+
+ StringBuilder html = new StringBuilder('<html>\n<head>\n');
Review comment:
Yes, if we add CSS that's a good idea. With the current unstyled stuff this was the simplest we can do.
If we change that here, we should also do this in tools/ where Ant's script is located.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta commented on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta commented on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622903717
> if Uwe insists this is a problem then would it be a large patch to build those docs under target documentation location?
I think all we need is replacing `project.javadoc.destinationDir` variables in this file with the target documentation folder, `_docroot_/${pathToDocdir(project.path)}`. (For now the renderJavadoc task does not care about where is the final destination, so we have to teach it about the value of _docroot_ to the task in some way.)
https://github.com/apache/lucene-solr/blob/master/gradle/render-javadoc.gradle
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mocobeta edited a comment on pull request #1477: LUCENE-9321: Port markdown task to Gradle
Posted by GitBox <gi...@apache.org>.
mocobeta edited a comment on pull request #1477:
URL: https://github.com/apache/lucene-solr/pull/1477#issuecomment-622866628
Just for clarification... Eventually, the `build/documentation` folder should look like this regardless of which way we choose - keep per-project javadoc outputs, or outputs docs into top-level `documentation` folder from the beginning:
**Lucene**
```
lucene/build/documentation
├── JRE_VERSION_MIGRATION.html
├── MIGRATE.html
├── SYSTEM_REQUIREMENTS.html
├── index.html
├── lucene_green_300.gif
├── changes
├── analysis
│ ├── common
│ ├── icu
│ ├── kuromoji
│ ├── morfologik
│ ├── nori
│ ├── opennlp
│ ├── phonetic
│ ├── smartcn
│ └── stempel
├── backward-codecs
├── benchmark
├── classification
├── codecs
├── core
├── demo
├── expressions
├── facet
├── grouping
├── highlighter
├── join
├── luke
├── memory
├── misc
├── monitor
├── queries
├── queryparser
├── replicator
├── sandbox
├── spatial-extras
├── spatial3d
├── suggest
└── test-framework
```
**Solr**
```
solr/build/documentation
├── SYSTEM_REQUIREMENTS.html
├── index.html
├── changes
├── images
├── contrib
│ ├── analysis-extras
│ ├── analytics
│ ├── clustering
│ ├── dataimporthandler
│ ├── dataimporthandler-extras
│ ├── extraction
│ ├── jaegertracer-configurator
│ ├── langid
│ ├── ltr
│ ├── prometheus-exporter
│ └── velocity
├── core
├── solrj
└── test-framework
```
The each subproject's javadoc folder structure is consistent with Gradle project path (as I emphasized on LUCENE-9278). Both `build/documentation` folder should be uploaded to lucene.apache.org website on an as-is basis (it's the final purpose of the `documentation` task).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org