You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2026/04/01 22:09:03 UTC
Re: Algolia search on website is broken
It’s broken again. This is the third breakage I am reporting in the past couple of years.
Is there some sort of alert or CI test we could setup to catch or prevent this going forward?
> On Dec 21, 2025, at 1:35 PM, Gengliang Wang <lt...@gmail.com> wrote:
>
> Hi all,
>
>
> The crawler issue has been identified and fixed.
>
> The root cause was that by the crawler fails when the latest result contains less than 90% of the previous result. Increasing the `maxLostRecordsPercentage` threshold resolves the issue.
>
> https://www.algolia.com/doc/tools/crawler/apis/configuration/safety-checks
>
>
> On Wed, Dec 17, 2025 at 10:03 PM Xiao Li <gatorsmile@gmail.com <ma...@gmail.com>> wrote:
>> Thanks for reporting it! Will take a look
>>
>> Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> 于2025年12月5日周五 04:19写道：
>>> Bueller?
>>>
>>> Is anyone on this list able to fix the crawler?
>>>
>>>
>>>> On Dec 1, 2025, at 12:19 PM, Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> This seems to be happening again.
>>>>
>>>> Perhaps we should add a new test (but where, I wonder?) to ensure that Algolia search doesn’t break without us knowing.
>>>>
>>>> Nick
>>>>
>>>>
>>>>> On Dec 11, 2023, at 5:02 AM, Gengliang Wang <ltnwgl@gmail.com <ma...@gmail.com>> wrote:
>>>>>
>>>>> Hi Nick,
>>>>>
>>>>> Thank you for reporting the issue with our web crawler.
>>>>>
>>>>> I've found that the issue was due to a change(specifically, pull request #40269 <https://github.com/apache/spark/pull/40269>) in the website's HTML structure, where the JavaScript selector ".container-wrapper" is now ".container". I've updated the crawler accordingly, and it's working properly now.
>>>>>
>>>>> Gengliang
>>>>>
>>>>> On Sun, Dec 10, 2023 at 8:15 AM Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>> Pinging Gengliang and Xiao about this, per these docs <https://github.com/apache/spark-website/blob/0ceaaaf528ec1d0201e1eab1288f37cce607268b/release-process.md#update-the-configuration-of-algolia-crawler>.
>>>>>>
>>>>>> It looks like to fix this problem you need access to the Algolia Crawler Admin Console.
>>>>>>
>>>>>>
>>>>>>> On Dec 5, 2023, at 11:28 AM, Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>> Should I report this instead on Jira? Apologies if the dev list is not the right place.
>>>>>>>
>>>>>>> Search on the website appears to be broken. For example, here is a search for “analyze”:
>>>>>>>
>>>>>>> <Image 12-5-23 at 11.26 AM.jpeg>
>>>>>>>
>>>>>>> And here is the same search using DDG <https://duckduckgo.com/?q=site:https://spark.apache.org/docs/latest/+analyze&t=osx&ia=web>.
>>>>>>>
>>>>>>> Nick
>>>>>>>
>>>>>>
>>>>
>>>
Re: Algolia search on website is broken
Posted by Nicholas Chammas <ni...@gmail.com>.
It’s broken again…
> On Apr 3, 2026, at 12:18 PM, Nicholas Chammas <ni...@gmail.com> wrote:
>
> Thanks for fixing this. I can confirm it’s working from my side.
>
> Looks like we need some kind of alert on Algolia's crawl status <https://www.algolia.com/doc/tools/crawler/troubleshooting/crawl-status>. If there’s a way a non-committer can help with this, let me know.
>
>
>> On Apr 3, 2026, at 1:39 AM, Gengliang Wang <lt...@gmail.com> wrote:
>>
>> Hi Nicholas,
>>
>> The crawler configuration was not updated after the Spark 4.1.1 release, as documented in the release process <https://spark.apache.org/release-process.html>. I've fixed it.
>>
>> A unit test isn't really feasible here since the doc search is powered by Algolia, but we could set up an Algolia monitoring alert to catch this proactively. I'll look into it when I have the bandwidth.
>>
>> Gengliang
>>
>> On Wed, Apr 1, 2026 at 3:09 PM Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>> It’s broken again. This is the third breakage I am reporting in the past couple of years.
>>>
>>> Is there some sort of alert or CI test we could setup to catch or prevent this going forward?
>>>
>>>
>>>> On Dec 21, 2025, at 1:35 PM, Gengliang Wang <ltnwgl@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>>
>>>> The crawler issue has been identified and fixed.
>>>>
>>>> The root cause was that by the crawler fails when the latest result contains less than 90% of the previous result. Increasing the `maxLostRecordsPercentage` threshold resolves the issue.
>>>>
>>>> https://www.algolia.com/doc/tools/crawler/apis/configuration/safety-checks
>>>>
>>>>
>>>> On Wed, Dec 17, 2025 at 10:03 PM Xiao Li <gatorsmile@gmail.com <ma...@gmail.com>> wrote:
>>>>> Thanks for reporting it! Will take a look
>>>>>
>>>>> Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> 于2025年12月5日周五 04:19写道：
>>>>>> Bueller?
>>>>>>
>>>>>> Is anyone on this list able to fix the crawler?
>>>>>>
>>>>>>
>>>>>>> On Dec 1, 2025, at 12:19 PM, Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> This seems to be happening again.
>>>>>>>
>>>>>>> Perhaps we should add a new test (but where, I wonder?) to ensure that Algolia search doesn’t break without us knowing.
>>>>>>>
>>>>>>> Nick
>>>>>>>
>>>>>>>
>>>>>>>> On Dec 11, 2023, at 5:02 AM, Gengliang Wang <ltnwgl@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>
>>>>>>>> Hi Nick,
>>>>>>>>
>>>>>>>> Thank you for reporting the issue with our web crawler.
>>>>>>>>
>>>>>>>> I've found that the issue was due to a change(specifically, pull request #40269 <https://github.com/apache/spark/pull/40269>) in the website's HTML structure, where the JavaScript selector ".container-wrapper" is now ".container". I've updated the crawler accordingly, and it's working properly now.
>>>>>>>>
>>>>>>>> Gengliang
>>>>>>>>
>>>>>>>> On Sun, Dec 10, 2023 at 8:15 AM Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>> Pinging Gengliang and Xiao about this, per these docs <https://github.com/apache/spark-website/blob/0ceaaaf528ec1d0201e1eab1288f37cce607268b/release-process.md#update-the-configuration-of-algolia-crawler>.
>>>>>>>>>
>>>>>>>>> It looks like to fix this problem you need access to the Algolia Crawler Admin Console.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Dec 5, 2023, at 11:28 AM, Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>>
>>>>>>>>>> Should I report this instead on Jira? Apologies if the dev list is not the right place.
>>>>>>>>>>
>>>>>>>>>> Search on the website appears to be broken. For example, here is a search for “analyze”:
>>>>>>>>>>
>>>>>>>>>> <Image 12-5-23 at 11.26 AM.jpeg>
>>>>>>>>>>
>>>>>>>>>> And here is the same search using DDG <https://duckduckgo.com/?q=site:https://spark.apache.org/docs/latest/+analyze&t=osx&ia=web>.
>>>>>>>>>>
>>>>>>>>>> Nick
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>
>
Re: Algolia search on website is broken
Posted by Nicholas Chammas <ni...@gmail.com>.
Thanks for fixing this. I can confirm it’s working from my side.
Looks like we need some kind of alert on Algolia's crawl status <https://www.algolia.com/doc/tools/crawler/troubleshooting/crawl-status>. If there’s a way a non-committer can help with this, let me know.
> On Apr 3, 2026, at 1:39 AM, Gengliang Wang <lt...@gmail.com> wrote:
>
> Hi Nicholas,
>
> The crawler configuration was not updated after the Spark 4.1.1 release, as documented in the release process <https://spark.apache.org/release-process.html>. I've fixed it.
>
> A unit test isn't really feasible here since the doc search is powered by Algolia, but we could set up an Algolia monitoring alert to catch this proactively. I'll look into it when I have the bandwidth.
>
> Gengliang
>
> On Wed, Apr 1, 2026 at 3:09 PM Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>> It’s broken again. This is the third breakage I am reporting in the past couple of years.
>>
>> Is there some sort of alert or CI test we could setup to catch or prevent this going forward?
>>
>>
>>> On Dec 21, 2025, at 1:35 PM, Gengliang Wang <ltnwgl@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi all,
>>>
>>>
>>> The crawler issue has been identified and fixed.
>>>
>>> The root cause was that by the crawler fails when the latest result contains less than 90% of the previous result. Increasing the `maxLostRecordsPercentage` threshold resolves the issue.
>>>
>>> https://www.algolia.com/doc/tools/crawler/apis/configuration/safety-checks
>>>
>>>
>>> On Wed, Dec 17, 2025 at 10:03 PM Xiao Li <gatorsmile@gmail.com <ma...@gmail.com>> wrote:
>>>> Thanks for reporting it! Will take a look
>>>>
>>>> Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> 于2025年12月5日周五 04:19写道：
>>>>> Bueller?
>>>>>
>>>>> Is anyone on this list able to fix the crawler?
>>>>>
>>>>>
>>>>>> On Dec 1, 2025, at 12:19 PM, Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> This seems to be happening again.
>>>>>>
>>>>>> Perhaps we should add a new test (but where, I wonder?) to ensure that Algolia search doesn’t break without us knowing.
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>>> On Dec 11, 2023, at 5:02 AM, Gengliang Wang <ltnwgl@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>> Hi Nick,
>>>>>>>
>>>>>>> Thank you for reporting the issue with our web crawler.
>>>>>>>
>>>>>>> I've found that the issue was due to a change(specifically, pull request #40269 <https://github.com/apache/spark/pull/40269>) in the website's HTML structure, where the JavaScript selector ".container-wrapper" is now ".container". I've updated the crawler accordingly, and it's working properly now.
>>>>>>>
>>>>>>> Gengliang
>>>>>>>
>>>>>>> On Sun, Dec 10, 2023 at 8:15 AM Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>> Pinging Gengliang and Xiao about this, per these docs <https://github.com/apache/spark-website/blob/0ceaaaf528ec1d0201e1eab1288f37cce607268b/release-process.md#update-the-configuration-of-algolia-crawler>.
>>>>>>>>
>>>>>>>> It looks like to fix this problem you need access to the Algolia Crawler Admin Console.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Dec 5, 2023, at 11:28 AM, Nicholas Chammas <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>
>>>>>>>>> Should I report this instead on Jira? Apologies if the dev list is not the right place.
>>>>>>>>>
>>>>>>>>> Search on the website appears to be broken. For example, here is a search for “analyze”:
>>>>>>>>>
>>>>>>>>> <Image 12-5-23 at 11.26 AM.jpeg>
>>>>>>>>>
>>>>>>>>> And here is the same search using DDG <https://duckduckgo.com/?q=site:https://spark.apache.org/docs/latest/+analyze&t=osx&ia=web>.
>>>>>>>>>
>>>>>>>>> Nick
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>
Re: Algolia search on website is broken
Posted by Gengliang Wang <lt...@gmail.com>.
Hi Nicholas,
The crawler configuration was not updated after the Spark 4.1.1 release, as
documented in the release process
<https://spark.apache.org/release-process.html>. I've fixed it.
A unit test isn't really feasible here since the doc search is powered by
Algolia, but we could set up an Algolia monitoring alert to catch this
proactively. I'll look into it when I have the bandwidth.
Gengliang
On Wed, Apr 1, 2026 at 3:09 PM Nicholas Chammas <ni...@gmail.com>
wrote:
> It’s broken again. This is the third breakage I am reporting in the past
> couple of years.
>
> Is there some sort of alert or CI test we could setup to catch or prevent
> this going forward?
>
>
> On Dec 21, 2025, at 1:35 PM, Gengliang Wang <lt...@gmail.com> wrote:
>
> Hi all,
>
> The crawler issue has been identified and fixed.
>
> The root cause was that by the crawler fails when the latest result
> contains less than 90% of the previous result. Increasing the
> `maxLostRecordsPercentage` threshold resolves the issue.
>
> https://www.algolia.com/doc/tools/crawler/apis/configuration/safety-checks
>
> On Wed, Dec 17, 2025 at 10:03 PM Xiao Li <ga...@gmail.com> wrote:
>
>> Thanks for reporting it! Will take a look
>>
>> Nicholas Chammas <ni...@gmail.com> 于2025年12月5日周五 04:19写道：
>>
>>> Bueller?
>>>
>>> Is anyone on this list able to fix the crawler?
>>>
>>>
>>> On Dec 1, 2025, at 12:19 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> This seems to be happening again.
>>>
>>> Perhaps we should add a new test (but where, I wonder?) to ensure that
>>> Algolia search doesn’t break without us knowing.
>>>
>>> Nick
>>>
>>>
>>> On Dec 11, 2023, at 5:02 AM, Gengliang Wang <lt...@gmail.com> wrote:
>>>
>>> Hi Nick,
>>>
>>> Thank you for reporting the issue with our web crawler.
>>>
>>> I've found that the issue was due to a change(specifically, pull
>>> request #40269 <https://github.com/apache/spark/pull/40269>) in the
>>> website's HTML structure, where the JavaScript selector
>>> ".container-wrapper" is now ".container". I've updated the crawler
>>> accordingly, and it's working properly now.
>>>
>>> Gengliang
>>>
>>> On Sun, Dec 10, 2023 at 8:15 AM Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> Pinging Gengliang and Xiao about this, per these docs
>>>> <https://github.com/apache/spark-website/blob/0ceaaaf528ec1d0201e1eab1288f37cce607268b/release-process.md#update-the-configuration-of-algolia-crawler>
>>>> .
>>>>
>>>> It looks like to fix this problem you need access to the Algolia
>>>> Crawler Admin Console.
>>>>
>>>>
>>>> On Dec 5, 2023, at 11:28 AM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>> Should I report this instead on Jira? Apologies if the dev list is not
>>>> the right place.
>>>>
>>>> Search on the website appears to be broken. For example, here is a
>>>> search for “analyze”:
>>>>
>>>> <Image 12-5-23 at 11.26 AM.jpeg>
>>>>
>>>> And here is the same search using DDG
>>>> <https://duckduckgo.com/?q=site:https://spark.apache.org/docs/latest/+analyze&t=osx&ia=web>
>>>> .
>>>>
>>>> Nick
>>>>
>>>>
>>>>
>>>
>>>
>