You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "TamimEhsan (via GitHub)" <gi...@apache.org> on 2025/08/16 05:02:06 UTC

[PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

TamimEhsan opened a new pull request, #1631:
URL: https://github.com/apache/stormcrawler/pull/1631

   This PR adds readability in the BasicURLFilter#filterPathRepeat by removing the magic number 4 and adding comments to explain the number


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 merged PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280320379


##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
     public final String filterPathRepeat(String urlToFilter) {
         // check whether a path element is repeated N times
         String[] paths = urlToFilter.split("/");
-        if (paths.length <= 4) return urlToFilter;
+
+        final int minimumPathLength = 5;

Review Comment:
   Either a constant or a configurable options (if it makes sense to be able to configure it), imho.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "TamimEhsan (via GitHub)" <gi...@apache.org>.
TamimEhsan commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280389236


##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
     public final String filterPathRepeat(String urlToFilter) {
         // check whether a path element is repeated N times
         String[] paths = urlToFilter.split("/");
-        if (paths.length <= 4) return urlToFilter;
+
+        final int minimumPathLength = 5;

Review Comment:
   @rzo1 converted it to static final int class level variable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "TamimEhsan (via GitHub)" <gi...@apache.org>.
TamimEhsan commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280256931


##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
     public final String filterPathRepeat(String urlToFilter) {
         // check whether a path element is repeated N times
         String[] paths = urlToFilter.split("/");
-        if (paths.length <= 4) return urlToFilter;
+
+        final int minimumPathLength = 5;

Review Comment:
   @sigee I looked up for other instances like this and followed those. for example 
   https://github.com/apache/stormcrawler/blob/9d4b0302a7859f9f24caa1fe3cdabb498f968112/core/src/main/java/org/apache/stormcrawler/bolt/FeedParserBolt.java#L93-L96
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "sigee (via GitHub)" <gi...@apache.org>.
sigee commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280250410


##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
     public final String filterPathRepeat(String urlToFilter) {
         // check whether a path element is repeated N times
         String[] paths = urlToFilter.split("/");
-        if (paths.length <= 4) return urlToFilter;
+
+        final int minimumPathLength = 5;

Review Comment:
   To maintain consistency shouldn't it be extracted as Class variable as the other two?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "TamimEhsan (via GitHub)" <gi...@apache.org>.
TamimEhsan commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280389236


##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
     public final String filterPathRepeat(String urlToFilter) {
         // check whether a path element is repeated N times
         String[] paths = urlToFilter.split("/");
-        if (paths.length <= 4) return urlToFilter;
+
+        final int minimumPathLength = 5;

Review Comment:
   converted it to static final int class level variable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore: remove magic number from filterPathRepeat method [stormcrawler]

Posted by "TamimEhsan (via GitHub)" <gi...@apache.org>.
TamimEhsan commented on code in PR #1631:
URL: https://github.com/apache/stormcrawler/pull/1631#discussion_r2280257913


##########
core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLFilter.java:
##########
@@ -48,7 +48,11 @@ public String filter(
     public final String filterPathRepeat(String urlToFilter) {
         // check whether a path element is repeated N times
         String[] paths = urlToFilter.split("/");
-        if (paths.length <= 4) return urlToFilter;
+
+        final int minimumPathLength = 5;

Review Comment:
   I think similar occurances can be refactored together and moved to class variable as const int?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org