[I] http.agent.name is dropped from the robots.txt match set when http.robots.agents is set [stormcrawler]

["rzo1 (via GitHub)" <gi...@apache.org>]

rzo1 opened a new issue, #2026:
URL: https://github.com/apache/stormcrawler/issues/2026

   `RobotRulesParser.setConf()` never adds `http.agent.name` to `agentNames` when `http.robots.agents` is non-empty:
   
   - if the advertised name is listed first, `index++` skips it and the loop appends only `agents[1..]`, so the name is dropped entirely;
   - if it is not listed first, the code logs *"Agent we advertise (X) not listed first"* and appends the configured list, still without adding the name.
   
   `agentNames` is only ever written in the empty-list branch and in that loop, which contradicts the in-code comment: *"If both are present, our agent-string should be the first one we advertise to robots-parsing."*
   
   ### Impact
   
   With `http.agent.name: mybot` and `http.robots.agents: mybot,mybot-uk`, the match set handed to crawler-commons is `[mybot-uk]`. A `User-agent: mybot` group in a robots.txt does not match and the parser falls back to `*`. Rules addressed specifically to the crawler are ignored, which is a compliance issue rather than a cosmetic one.
   
   ### Reproducer
   
   With a robots.txt of:
   
   ```
   User-agent: mybot
   Disallow: /restricted/
   
   User-agent: *
   Disallow: /
   ```
   
   and `http.agent.name: mybot`, `http.robots.agents: mybot,mybot-uk`, `/index.html` is reported as disallowed because the wildcard group is selected.
   
   Nothing in `core/src/test` currently references `http.robots.agents`.
   
   ### Fix
   
   Add `agentNames.add(agentName)` before the append loop. `agentNames` is a `LinkedHashSet`, so the existing `index++` can stay and no duplicate is introduced.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] http.agent.name is dropped from the robots.txt match set when http.robots.agents is set [stormcrawler]

["jnioche (via GitHub)" <gi...@apache.org>]

jnioche closed issue #2026: http.agent.name is dropped from the robots.txt match set when http.robots.agents is set
URL: https://github.com/apache/stormcrawler/issues/2026


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org