You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Nic Crane <th...@gmail.com> on 2025/02/17 11:18:05 UTC

Docs AI bot

I was chatting earlier with Alenka, who mentioned this LinkedIn post[1] by
Ritchie Vink from Polars, which talks about the Polars project's use of an
AI service which ingests their docs and answers user questions based on the
results.

It's provided by an organisation called kapa.ai which offers this service
often free to open source projects [2], and I was wondering if it's
something we might consider trying?

It'd give us useful insights into where we have gaps in our docs where we
can improve things, or what are common things that users struggle with.  It
claims to have better results than basic LLM search due to using and citing
the docs directly, and acknowledging when it doesn't know the answer -
though we'd need to test it out to

There's an example of the widget on the Django CMS docs [3] under the "Ask
AI" link which provides a popup where the AI bot appears and users can ask
questions.

I had suspected there might be conflict with the website policy [4] but it
appears that the ASF already allows its usage [5][6].

What do folks think of the idea of us experimenting with this for Arrow?

Nic

[1]
https://www.linkedin.com/posts/ritchievink_python-api-reference-polars-documentation-activity-7296175345987170305-ZEUI/
[2] https://docs.kapa.ai/kapa-for-open-source
[3] https://www.django-cms.org/en/support/
[4] https://privacy.apache.org/policies/website-policy.html
[5] https://privacy.apache.org/policies/privacy-policy-public.html
[6] https://privacy.apache.org/faq/committers.html

Re: Docs AI bot

Posted by Nic Crane <th...@gmail.com>.
Hi Antoine,

> Would Kapa provide us with stats about usage of their service?

Yep, I think that would be one of the most valuable things for us.  In the
docs [1] it shows a UI where we can view the questions which have been
asked.

> we should check at some point that the provided service is useful enough.

Absolutely - perhaps we can try it for 1 major release and evaluate before
the next one whether we've got anything useful out of it and then make the
decision about whether to retain it or not.

Nic

[1]
https://docs.kapa.ai/integrations/website-widget/user-tracking#anonymous-user-tracking-with-fingerprint

On Mon, 17 Feb 2025 at 11:59, Antoine Pitrou <an...@python.org> wrote:

>
> Hi Nic,
>
> Le 17/02/2025 à 12:18, Nic Crane a écrit :> It'd give us useful insights
> into where we have gaps in our docs where we
> > can improve things, or what are common things that users struggle with.
>
> Would Kapa provide us with stats about usage of their service?
>
> > What do folks think of the idea of us experimenting with this for Arrow?
>
> I would say "why not". It also sounds like it would turn our users into
> beta testers of that company's technology, so we should check at some
> point that the provided service is useful enough.
>
> (note: kapa.ai itself does not look open source at all)
>
> Regards
>
> Antoine.
>

Re: Docs AI bot

Posted by Antoine Pitrou <an...@python.org>.
Hi Nic,

Le 17/02/2025 à 12:18, Nic Crane a écrit :> It'd give us useful insights into where we have gaps in our docs where we
> can improve things, or what are common things that users struggle with.

Would Kapa provide us with stats about usage of their service?

> What do folks think of the idea of us experimenting with this for Arrow?

I would say "why not". It also sounds like it would turn our users into
beta testers of that company's technology, so we should check at some
point that the provided service is useful enough.

(note: kapa.ai itself does not look open source at all)

Regards

Antoine.

Re: Docs AI bot

Posted by Nic Crane <th...@gmail.com>.
OK, great, thanks both for your comments. Given there don't seem to be any
overall objections, I'll push ahead with getting this set up and make a PR
to integrate it into the docs.

We can assess its suitability once it's up there and we've got some data
from testing it out.

Cheers,

Nic

On Tue, 18 Feb 2025, 09:02 Krisztián Szűcs, <sz...@gmail.com>
wrote:

> Hi Nic,
>
> I am definitely in favor of the proposal because:
> - arrow can be confusing due to the various subprojects / specifications /
> implementations, so context-specific answers could help reduce this
> confusion
> - even if the LLM's answer is not entirely accurate, it provides useful
> pointers to the relevant sections of the documentation
> - it aggregates content from multiple sources [1] (zulip is not supported
> yet, but we can submit a request)
> - it is already being used by several large projects
> - as far as I know, they are actively working on improving accuracy
> - analytics [2] could help us to improve the documentation
>
> Regards, Krisztian
>
> [1]: https://docs.kapa.ai/data-sources/supported-data-sources
> [2]: https://docs.kapa.ai/analytics/overview
>
> > On 2025. Feb 17., at 12:18, Nic Crane <th...@gmail.com> wrote:
> >
> > I was chatting earlier with Alenka, who mentioned this LinkedIn post[1]
> by
> > Ritchie Vink from Polars, which talks about the Polars project's use of
> an
> > AI service which ingests their docs and answers user questions based on
> the
> > results.
> >
> > It's provided by an organisation called kapa.ai which offers this
> service
> > often free to open source projects [2], and I was wondering if it's
> > something we might consider trying?
> >
> > It'd give us useful insights into where we have gaps in our docs where we
> > can improve things, or what are common things that users struggle with.
> It
> > claims to have better results than basic LLM search due to using and
> citing
> > the docs directly, and acknowledging when it doesn't know the answer -
> > though we'd need to test it out to
> >
> > There's an example of the widget on the Django CMS docs [3] under the
> "Ask
> > AI" link which provides a popup where the AI bot appears and users can
> ask
> > questions.
> >
> > I had suspected there might be conflict with the website policy [4] but
> it
> > appears that the ASF already allows its usage [5][6].
> >
> > What do folks think of the idea of us experimenting with this for Arrow?
> >
> > Nic
> >
> > [1]
> >
> https://www.linkedin.com/posts/ritchievink_python-api-reference-polars-documentation-activity-7296175345987170305-ZEUI/
> > [2] https://docs.kapa.ai/kapa-for-open-source
> > [3] https://www.django-cms.org/en/support/
> > [4] https://privacy.apache.org/policies/website-policy.html
> > [5] https://privacy.apache.org/policies/privacy-policy-public.html
> > [6] https://privacy.apache.org/faq/committers.html
>
>

Re: Docs AI bot

Posted by Krisztián Szűcs <sz...@gmail.com>.
Hi Nic,

I am definitely in favor of the proposal because:
- arrow can be confusing due to the various subprojects / specifications / implementations, so context-specific answers could help reduce this confusion
- even if the LLM's answer is not entirely accurate, it provides useful pointers to the relevant sections of the documentation
- it aggregates content from multiple sources [1] (zulip is not supported yet, but we can submit a request)
- it is already being used by several large projects
- as far as I know, they are actively working on improving accuracy
- analytics [2] could help us to improve the documentation

Regards, Krisztian

[1]: https://docs.kapa.ai/data-sources/supported-data-sources
[2]: https://docs.kapa.ai/analytics/overview

> On 2025. Feb 17., at 12:18, Nic Crane <th...@gmail.com> wrote:
> 
> I was chatting earlier with Alenka, who mentioned this LinkedIn post[1] by
> Ritchie Vink from Polars, which talks about the Polars project's use of an
> AI service which ingests their docs and answers user questions based on the
> results.
> 
> It's provided by an organisation called kapa.ai which offers this service
> often free to open source projects [2], and I was wondering if it's
> something we might consider trying?
> 
> It'd give us useful insights into where we have gaps in our docs where we
> can improve things, or what are common things that users struggle with.  It
> claims to have better results than basic LLM search due to using and citing
> the docs directly, and acknowledging when it doesn't know the answer -
> though we'd need to test it out to
> 
> There's an example of the widget on the Django CMS docs [3] under the "Ask
> AI" link which provides a popup where the AI bot appears and users can ask
> questions.
> 
> I had suspected there might be conflict with the website policy [4] but it
> appears that the ASF already allows its usage [5][6].
> 
> What do folks think of the idea of us experimenting with this for Arrow?
> 
> Nic
> 
> [1]
> https://www.linkedin.com/posts/ritchievink_python-api-reference-polars-documentation-activity-7296175345987170305-ZEUI/
> [2] https://docs.kapa.ai/kapa-for-open-source
> [3] https://www.django-cms.org/en/support/
> [4] https://privacy.apache.org/policies/website-policy.html
> [5] https://privacy.apache.org/policies/privacy-policy-public.html
> [6] https://privacy.apache.org/faq/committers.html