You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "JaspreetSinghChahal (via GitHub)" <gi...@apache.org> on 2026/02/03 02:25:27 UTC
[D] Don't index sitemap inopensearch [stormcrawler]
GitHub user JaspreetSinghChahal created a discussion: Don't index sitemap inopensearch
In stormcrawler, I want to fetch and parse sitemaps to discover new urls but don't want to index it in opensearch.
I can't exclude it in regex as new urls won't be discovered then.
GitHub link: https://github.com/apache/stormcrawler/discussions/1803
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap in opensearch [stormcrawler]
Posted by "JaspreetSinghChahal (via GitHub)" <gi...@apache.org>.
GitHub user JaspreetSinghChahal added a comment to the discussion: Don't index sitemap in opensearch
I fixed this for me.
For PRI have a doubt , I needed to exclude documents with isFeed=true or isSitemap=true in metadata. That meant documents with this value as false or not present should be included. It is in contradiction to current implementation . If I make it inclusive
i.e say isFeed=false then documents were this field is no present is also excluded.
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15743217
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap in opensearch [stormcrawler]
Posted by "JaspreetSinghChahal (via GitHub)" <gi...@apache.org>.
GitHub user JaspreetSinghChahal edited a comment on the discussion: Don't index sitemap in opensearch
I fixed this for me.
For PR I have a doubt , I needed to exclude documents with isFeed=true or isSitemap=true in metadata. That meant documents with this value as false or not present should be included. It is in contradiction to current implementation . If I make it inclusive
i.e say isFeed=false then documents were this field is no present is also excluded.
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15743217
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap inopensearch [stormcrawler]
Posted by "JaspreetSinghChahal (via GitHub)" <gi...@apache.org>.
GitHub user JaspreetSinghChahal added a comment to the discussion: Don't index sitemap inopensearch
It seems to be taking only single key. How to do it for more example "isSitemap=false", "isFeed=false"
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15705025
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap inopensearch [stormcrawler]
Posted by "JaspreetSinghChahal (via GitHub)" <gi...@apache.org>.
GitHub user JaspreetSinghChahal edited a comment on the discussion: Don't index sitemap inopensearch
It seems to be taking only single key. How to do it for more keys, example "isSitemap=false", "isFeed=false"?
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15705025
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap in opensearch [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
GitHub user rzo1 added a comment to the discussion: Don't index sitemap in opensearch
@JaspreetSinghChahal There’s a mismatch between the current implementation and the documentation. From my perspective, it makes more sense to align the implementation with what the documentation specifies. Therefore, it would be useful to allow this parameter to accept a YAML list instead of just a single key-value.
Any implementation should remain backward compatible, still supporting the previous configuration format. Would you be interested in submitting a PR with the fix and a test?
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15705459
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap in opensearch [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
GitHub user rzo1 added a comment to the discussion: Don't index sitemap in opensearch
Yes, you are right. The current implementation is a bit weird in that sense.
From my POV, filtering is exclusion-based, not inclusion-based, i.e. “exclude documents matching these conditions”.
That means:
- If a document has key=value matching the filter, it should be filtered out
- If the key is not present, the document should not be filtered
- Multiple filters should be supported (as the docs claim) - guess it would be **or** between multiple filters.
For each filter entry:
- If key not present -> ignore
- If key present and value matches -> reject document
In that sense, I think, that we should fix it (or introduce a new config option to be backward compatbile).
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15744130
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap in opensearch [stormcrawler]
Posted by "JaspreetSinghChahal (via GitHub)" <gi...@apache.org>.
GitHub user JaspreetSinghChahal edited a discussion: Don't index sitemap in opensearch
In stormcrawler, I want to fetch and parse sitemaps to discover new urls but don't want to index it in opensearch.
I can't exclude it in regex as new urls won't be discovered then.
GitHub link: https://github.com/apache/stormcrawler/discussions/1803
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org
Re: [D] Don't index sitemap inopensearch [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
GitHub user rzo1 added a comment to the discussion: Don't index sitemap inopensearch
The sitemap parser bolt will set a metadata key [`isSitemap`](https://stormcrawler.apache.org/docs/3.5.1/index.html#_sitemapparserbolt) and use filter keys in the indexer's configuration to remove them from indexing: [`indexer.md.filter`](https://stormcrawler.apache.org/docs/3.5.1/index.html#_indexing)
GitHub link: https://github.com/apache/stormcrawler/discussions/1803#discussioncomment-15679470
----
This is an automatically sent email for dev@stormcrawler.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@stormcrawler.apache.org