You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@any23.apache.org by HansBrende <gi...@git.apache.org> on 2018/01/24 12:48:33 UTC
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
GitHub user HansBrende opened a pull request:
https://github.com/apache/any23/pull/59
ANY23-326 fixed rdfa issue with unclosed input & meta tags
This PR should also fix ANY23-317, ANY23-273, ANY23-267, ANY23-271, and ANY23-227 (this time, for realz).
These all have to do with the RDFa implementation failing to parse HTML.
My previous commit attempted to fix these issues by changing the default parser from NekoHTML to Jsoup. But alas, it turns out the RDFa implementation is using a completely different html parser under the hood, and it's the RDFa parser that's too strict, not ours, so changing ours from NekoHTML to Jsoup had no effect (although it did come with a nice 20% speed increase, so there's that). It seems that, for rio parsers, the document is parsed with Jsoup *only to get the document language* and then parsed **again** under the hood with who knows what.
Now, I simply check the RDF format to see if we're putting out XHTML. If we are, I first XHTML-ify the stream with Jsoup before sending it on to the rio RDF parser.
mvn clean install -> all tests passed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HansBrende/any23 ANY23-326
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/any23/pull/59.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #59
----
commit 74b2909b6d91cc4989093d90a38baef1c34c603f
Author: Hans <fi...@...>
Date: 2018-01-24T12:26:40Z
ANY23-326 fixed rdfa issue with unclosed input & meta tags
----
---
[GitHub] any23 issue #59: ANY23-326 fixed rdfa issue with unclosed input & meta tags
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on the issue:
https://github.com/apache/any23/pull/59
@lewismc my latest commit should fix ANY23-227 and ANY23-268.
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/any23/pull/59#discussion_r163680939
--- Diff: core/src/main/java/org/apache/any23/extractor/rdf/BaseRDFExtractor.java ---
@@ -105,7 +109,24 @@ public void run(
parser.getParserConfig().addNonFatalError(BasicParserSettings.NORMALIZE_DATATYPE_VALUES);
//ByteBuffer seems to represent incorrect content. Need to make sure it is the content
//of the <script> node and not anything else!
- parser.parse(in, extractionContext.getDocumentIRI().stringValue());
+ RDFFormat format = parser.getRDFFormat();
+ String iri = extractionContext.getDocumentIRI().stringValue();
+
+ if (format.hasFileExtension("xhtml")) {
--- End diff --
ack, please ignore
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/any23/pull/59#discussion_r163678949
--- Diff: core/src/main/java/org/apache/any23/extractor/rdf/BaseRDFExtractor.java ---
@@ -22,16 +22,20 @@
import org.apache.any23.extractor.ExtractionParameters;
import org.apache.any23.extractor.ExtractionResult;
import org.apache.any23.extractor.Extractor;
-import org.eclipse.rdf4j.rio.RDFHandlerException;
-import org.eclipse.rdf4j.rio.RDFParseException;
-import org.eclipse.rdf4j.rio.RDFParser;
-import org.eclipse.rdf4j.rio.RioSetting;
+import org.eclipse.rdf4j.rio.*;
--- End diff --
Can you please make explicit imports rather than wildcard.
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on a diff in the pull request:
https://github.com/apache/any23/pull/59#discussion_r163679410
--- Diff: core/src/main/java/org/apache/any23/extractor/rdf/BaseRDFExtractor.java ---
@@ -22,16 +22,20 @@
import org.apache.any23.extractor.ExtractionParameters;
import org.apache.any23.extractor.ExtractionResult;
import org.apache.any23.extractor.Extractor;
-import org.eclipse.rdf4j.rio.RDFHandlerException;
-import org.eclipse.rdf4j.rio.RDFParseException;
-import org.eclipse.rdf4j.rio.RDFParser;
-import org.eclipse.rdf4j.rio.RioSetting;
+import org.eclipse.rdf4j.rio.*;
--- End diff --
Sure, sorry, that was something that IntelliJ did. I have no idea why. Will fix.
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/any23/pull/59#discussion_r163679147
--- Diff: core/src/main/java/org/apache/any23/extractor/rdf/BaseRDFExtractor.java ---
@@ -105,7 +109,24 @@ public void run(
parser.getParserConfig().addNonFatalError(BasicParserSettings.NORMALIZE_DATATYPE_VALUES);
//ByteBuffer seems to represent incorrect content. Need to make sure it is the content
//of the <script> node and not anything else!
- parser.parse(in, extractionContext.getDocumentIRI().stringValue());
+ RDFFormat format = parser.getRDFFormat();
+ String iri = extractionContext.getDocumentIRI().stringValue();
+
+ if (format.hasFileExtension("xhtml")) {
--- End diff --
What happens when the file suffix is just .html?
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on a diff in the pull request:
https://github.com/apache/any23/pull/59#discussion_r163680710
--- Diff: core/src/main/java/org/apache/any23/extractor/rdf/BaseRDFExtractor.java ---
@@ -105,7 +109,24 @@ public void run(
parser.getParserConfig().addNonFatalError(BasicParserSettings.NORMALIZE_DATATYPE_VALUES);
//ByteBuffer seems to represent incorrect content. Need to make sure it is the content
//of the <script> node and not anything else!
- parser.parse(in, extractionContext.getDocumentIRI().stringValue());
+ RDFFormat format = parser.getRDFFormat();
+ String iri = extractionContext.getDocumentIRI().stringValue();
+
+ if (format.hasFileExtension("xhtml")) {
--- End diff --
All of the RDF formats defined in the RDFFormat class which have file extension "html" also have file extension "xhtml". (And there is exactly one of them: the RDFA format--which is the one that is giving us problems.)
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on a diff in the pull request:
https://github.com/apache/any23/pull/59#discussion_r163683520
--- Diff: core/src/main/java/org/apache/any23/extractor/rdf/BaseRDFExtractor.java ---
@@ -105,7 +109,24 @@ public void run(
parser.getParserConfig().addNonFatalError(BasicParserSettings.NORMALIZE_DATATYPE_VALUES);
//ByteBuffer seems to represent incorrect content. Need to make sure it is the content
//of the <script> node and not anything else!
- parser.parse(in, extractionContext.getDocumentIRI().stringValue());
+ RDFFormat format = parser.getRDFFormat();
+ String iri = extractionContext.getDocumentIRI().stringValue();
+
+ if (format.hasFileExtension("xhtml")) {
--- End diff --
@lewismc I can also do the check via pattern matching on MIME types instead of file extensions, if you'd prefer. This way was just the easiest & quickest, and either way we'd get the same results.
---
[GitHub] any23 pull request #59: ANY23-326 fixed rdfa issue with unclosed input & met...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/any23/pull/59
---
[GitHub] any23 issue #59: ANY23-326 fixed rdfa issue with unclosed input & meta tags
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on the issue:
https://github.com/apache/any23/pull/59
@lewismc I rebased all the commits into a single commit to make it easier to see what has changed. Everything should be fully functional now.
---