You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2024/11/22 07:36:04 UTC

[PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

rzo1 opened a new pull request, #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417

   See details: https://lists.apache.org/thread/xljxn3901mffhyjhdx7orwy938kf2vjv


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1854356820


##########
external/tika/pom.xml:
##########
@@ -60,6 +60,11 @@ under the License.
 					<groupId>org.slf4j</groupId>
 					<artifactId>slf4j-log4j12</artifactId>
 				</exclusion>
+				<!-- This comes in through the Tika Image Parser module and has an incompatible BSD license -->

Review Comment:
   Great to have the link!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#issuecomment-2493108174

   Replaced the RSS Feeds as well as the compressed Tripadvisor-based Sitemap tests with original stormcrawler.apache.org content.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1853435960


##########
core/src/test/java/org/apache/stormcrawler/bolt/FeedParserBoltTest.java:
##########
@@ -51,7 +51,10 @@ void testFeedParsing() throws IOException {
         Metadata metadata = new Metadata();
         // specify that it is a Feed file
         metadata.setValue(FeedParserBolt.isFeedKey, "true");
-        parse("http://www.guardian.com/Feed.xml", "guardian.rss", metadata);
+        parse(
+                "https://de.wikipedia.org/w/index.php?title=Apache_Software_Foundation&action=history&feed=rss",

Review Comment:
   Thanks. Think we are good now. I replaced it with handmade rss based upon the sitemap of our actual stormcrawler website.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1854343012


##########
external/tika/pom.xml:
##########
@@ -60,6 +60,11 @@ under the License.
 					<groupId>org.slf4j</groupId>
 					<artifactId>slf4j-log4j12</artifactId>
 				</exclusion>
+				<!-- This comes in through the Tika Image Parser module and has an incompatible BSD license -->

Review Comment:
   Apache Tika cleared the use of `jai-imageio-core` here: https://issues.apache.org/jira/browse/LEGAL-304.
   
   I'm ok excluding it to make licensing cleaner for stormcrawler, and I don't think there will be large consequences for stormcrawler.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1853416242


##########
core/src/test/java/org/apache/stormcrawler/bolt/FeedParserBoltTest.java:
##########
@@ -51,7 +51,10 @@ void testFeedParsing() throws IOException {
         Metadata metadata = new Metadata();
         // specify that it is a Feed file
         metadata.setValue(FeedParserBolt.isFeedKey, "true");
-        parse("http://www.guardian.com/Feed.xml", "guardian.rss", metadata);
+        parse(
+                "https://de.wikipedia.org/w/index.php?title=Apache_Software_Foundation&action=history&feed=rss",

Review Comment:
   can we use some page listed under apache scope instead? not sure if using wikipedia might upset some as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1853419849


##########
core/src/test/java/org/apache/stormcrawler/bolt/FeedParserBoltTest.java:
##########
@@ -51,7 +51,10 @@ void testFeedParsing() throws IOException {
         Metadata metadata = new Metadata();
         // specify that it is a Feed file
         metadata.setValue(FeedParserBolt.isFeedKey, "true");
-        parse("http://www.guardian.com/Feed.xml", "guardian.rss", metadata);
+        parse(
+                "https://de.wikipedia.org/w/index.php?title=Apache_Software_Foundation&action=history&feed=rss",

Review Comment:
   Alternative is I will just create something random myself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1853421533


##########
core/src/test/java/org/apache/stormcrawler/bolt/FeedParserBoltTest.java:
##########
@@ -51,7 +51,10 @@ void testFeedParsing() throws IOException {
         Metadata metadata = new Metadata();
         // specify that it is a Feed file
         metadata.setValue(FeedParserBolt.isFeedKey, "true");
-        parse("http://www.guardian.com/Feed.xml", "guardian.rss", metadata);
+        parse(
+                "https://de.wikipedia.org/w/index.php?title=Apache_Software_Foundation&action=history&feed=rss",

Review Comment:
   Let me create one for stormcrawler.apache.org (just to be save)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "tballison (via GitHub)" <gi...@apache.org>.
tballison merged PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1853417153


##########
core/src/test/java/org/apache/stormcrawler/bolt/FeedParserBoltTest.java:
##########
@@ -51,7 +51,10 @@ void testFeedParsing() throws IOException {
         Metadata metadata = new Metadata();
         // specify that it is a Feed file
         metadata.setValue(FeedParserBolt.isFeedKey, "true");
-        parse("http://www.guardian.com/Feed.xml", "guardian.rss", metadata);
+        parse(
+                "https://de.wikipedia.org/w/index.php?title=Apache_Software_Foundation&action=history&feed=rss",

Review Comment:
   Do you have an example for an RSS feed on an apache site ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fixes Issues mentioned in IPMC Vote [incubator-stormcrawler]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #1417:
URL: https://github.com/apache/incubator-stormcrawler/pull/1417#discussion_r1853429177


##########
core/src/test/java/org/apache/stormcrawler/bolt/FeedParserBoltTest.java:
##########
@@ -51,7 +51,10 @@ void testFeedParsing() throws IOException {
         Metadata metadata = new Metadata();
         // specify that it is a Feed file
         metadata.setValue(FeedParserBolt.isFeedKey, "true");
-        parse("http://www.guardian.com/Feed.xml", "guardian.rss", metadata);
+        parse(
+                "https://de.wikipedia.org/w/index.php?title=Apache_Software_Foundation&action=history&feed=rss",

Review Comment:
   maybe comdev has some
   https://community.apache.org/blog/
   
   the latest says
   https://community.apache.org/blog/rss-feeds-activated.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org