You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Mike Adamson <ma...@datastax.com> on 2023/09/13 16:08:42 UTC
[DISCUSS] Addition of smile-nlp test dependency for CEP-30
CEP-30: [Approximate Nearest Neighbor(ANN) Vector Search via
Storage-Attached Indexes] uses the smile-nlp library
(com.github.haifengl.smile-nlp) in its testing to allow the creation of
word2vec embeddings for valid input into the HNSW graph index.
The reason for this library is that we found that using random vectors in
testing produced very inconsistent results. Using the smile-nlp word2vec
implementation with the glove.3k.50d library produces repeatable results.
Does anyone have any objections to the use of this library as a test only
dependency?
--
[image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
Engineering
+1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
Find DataStax Online: [image: LinkedIn Logo]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
[image: Facebook Logo]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
[image: Twitter Logo] <https://twitter.com/DataStax> [image: RSS Feed]
<https://www.datastax.com/blog/rss.xml> [image: Github Logo]
<https://github.com/datastax>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Mike Adamson <ma...@datastax.com>.
We can't use open-nlp because it is JDK 17 only. I'll pull the smile-nlp
dependency and write something to do the same thing. Our usage was trivial.
On Thu, 14 Sept 2023 at 00:10, J. D. Jordan <je...@gmail.com>
wrote:
> Reading through smile license again, it is licensed pure GPL 3, not GPL
> with classpath exception. So I think that kills all debate here.
>
> -1 on inclusion
>
> On Sep 13, 2023, at 2:30 PM, Jeremiah Jordan <je...@gmail.com>
> wrote:
>
>
> I wonder if it can easily be replaced with Apache open-nlp? It also
> provides an implementation of GloVe.
>
>
> https://opennlp.apache.org/docs/2.3.0/apidocs/opennlp-tools/opennlp/tools/util/wordvector/Glove.html
>
>
> On Sep 13, 2023 at 1:17:46 PM, Benedict <be...@apache.org> wrote:
>
>> There’s a distinction for spotbugs and other build related tools where
>> they can be downloaded and used during the build so long as they’re not
>> critical to the build process.
>>
>> They have to be downloaded dynamically in binary form I believe though,
>> they cannot be included in the release.
>>
>> So it’s not really in conflict with what Jeff is saying, and my
>> recollection accords with Jeff’s
>>
>> On 13 Sep 2023, at 17:42, Brandon Williams <dr...@gmail.com> wrote:
>>
>>
>>
>> On Wed, Sep 13, 2023 at 11:37 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> You can open a legal JIRA to confirm, but based on my understanding (and
>>> re-confirming reading
>>> https://www.apache.org/legal/resolved.html#category-a ):
>>>
>>>
>> We should probably get clarification here regardless, iirc this came up
>> when we were considering SpotBugs too.
>>
>>
--
[image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
Engineering
+1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
Find DataStax Online: [image: LinkedIn Logo]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
[image: Facebook Logo]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
[image: Twitter Logo] <https://twitter.com/DataStax> [image: RSS Feed]
<https://www.datastax.com/blog/rss.xml> [image: Github Logo]
<https://github.com/datastax>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by "J. D. Jordan" <je...@gmail.com>.
Reading through smile license again, it is licensed pure GPL 3, not GPL with
classpath exception. So I think that kills all debate here.
-1 on inclusion
> On Sep 13, 2023, at 2:30 PM, Jeremiah Jordan <je...@gmail.com>
> wrote:
>
>
>
>
> I wonder if it can easily be replaced with Apache open-nlp? It also
> provides an implementation of GloVe.
>
>
>
>
> <https://opennlp.apache.org/docs/2.3.0/apidocs/opennlp-
> tools/opennlp/tools/util/wordvector/Glove.html>
>
>
>
>
>
>
>
>
> On Sep 13, 2023 at 1:17:46 PM, Benedict
> <[benedict@apache.org](mailto:benedict@apache.org)> wrote:
>
>
>> There’s a distinction for spotbugs and other build related tools where they
can be downloaded and used during the build so long as they’re not critical to
the build process.
>>
>>
>
>>
>> They have to be downloaded dynamically in binary form I believe though,
they cannot be included in the release.
>>
>>
>
>>
>> So it’s not really in conflict with what Jeff is saying, and my
recollection accords with Jeff’s
>>
>>
>
>>
>>> On 13 Sep 2023, at 17:42, Brandon Williams
<[driftx@gmail.com](mailto:driftx@gmail.com)> wrote:
>
>
>>
>>>
>>>
>>>
>
>>>
>>> On Wed, Sep 13, 2023 at 11:37 AM Jeff Jirsa
<[jjirsa@gmail.com](mailto:jjirsa@gmail.com)> wrote:
>
>>>
>>>> You can open a legal JIRA to confirm, but based on my understanding (and
re-confirming reading <https://www.apache.org/legal/resolved.html#category-a>
):
> |
>>>> ---|---
>>>
>>>
>>>
>>> We should probably get clarification here regardless, iirc this came up
when we were considering SpotBugs too.
>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Jeremiah Jordan <je...@gmail.com>.
I wonder if it can easily be replaced with Apache open-nlp? It also
provides an implementation of GloVe.
https://opennlp.apache.org/docs/2.3.0/apidocs/opennlp-tools/opennlp/tools/util/wordvector/Glove.html
On Sep 13, 2023 at 1:17:46 PM, Benedict <be...@apache.org> wrote:
> There’s a distinction for spotbugs and other build related tools where
> they can be downloaded and used during the build so long as they’re not
> critical to the build process.
>
> They have to be downloaded dynamically in binary form I believe though,
> they cannot be included in the release.
>
> So it’s not really in conflict with what Jeff is saying, and my
> recollection accords with Jeff’s
>
> On 13 Sep 2023, at 17:42, Brandon Williams <dr...@gmail.com> wrote:
>
>
>
> On Wed, Sep 13, 2023 at 11:37 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> You can open a legal JIRA to confirm, but based on my understanding (and
>> re-confirming reading
>> https://www.apache.org/legal/resolved.html#category-a ):
>>
>>
> We should probably get clarification here regardless, iirc this came up
> when we were considering SpotBugs too.
>
>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Benedict <be...@apache.org>.
There’s a distinction for spotbugs and other build related tools where they
can be downloaded and used during the build so long as they’re not critical to
the build process.
They have to be downloaded dynamically in binary form I believe though, they
cannot be included in the release.
So it’s not really in conflict with what Jeff is saying, and my recollection
accords with Jeff’s
> On 13 Sep 2023, at 17:42, Brandon Williams <dr...@gmail.com> wrote:
>
>
>
>
>
>
>
> On Wed, Sep 13, 2023 at 11:37 AM Jeff Jirsa
> <[jjirsa@gmail.com](mailto:jjirsa@gmail.com)> wrote:
>
>
>> You can open a legal JIRA to confirm, but based on my understanding (and
re-confirming reading <https://www.apache.org/legal/resolved.html#category-a>
):
> |
>> ---|---
>
>
>
> We should probably get clarification here regardless, iirc this came up when
> we were considering SpotBugs too.
>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Brandon Williams <dr...@gmail.com>.
On Wed, Sep 13, 2023 at 11:37 AM Jeff Jirsa <jj...@gmail.com> wrote:
> You can open a legal JIRA to confirm, but based on my understanding (and
> re-confirming reading
> https://www.apache.org/legal/resolved.html#category-a ):
>
>
We should probably get clarification here regardless, iirc this came up
when we were considering SpotBugs too.
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Jeff Jirsa <jj...@gmail.com>.
You can open a legal JIRA to confirm, but based on my understanding (and
re-confirming reading https://www.apache.org/legal/resolved.html#category-a
):
- The restriction is on what can be in A PROJECT. tests are part of the
project, and not distinguished from the compiled product (especially since
the PROJECT ships SOURCE to build the product, if the SOURCE to build
requires the test, the test is clearly non-optional)
- GPL is cat X https://www.apache.org/legal/resolved.html#category-x
Cat X mixes "project" and "product" a few times, but again, the product is
still the source distribution, which would include the test, which means
it's excluded.
"Apache projects may not distribute Category X licensed components, in
source or binary form" doesnt seem ambiguous to me, but if someone wants to
ask ASF legal if I'm wrong, that's totally ok.
On Wed, Sep 13, 2023 at 9:25 AM Ekaterina Dimitrova <e....@gmail.com>
wrote:
> Jeff, isn’t this ok as long as it is used only in tests? If we are not
> sure we can open a Jira to legal?
>
> On Wed, 13 Sep 2023 at 12:23, Jeff Jirsa <jj...@gmail.com> wrote:
>
>> Just to be clear - this repo?
>> https://github.com/haifengl/smile/blob/master/LICENSE
>>
>> That shows GPL + Commercial?
>>
>>
>>
>> On Wed, Sep 13, 2023 at 9:10 AM Brandon Williams <dr...@gmail.com>
>> wrote:
>>
>>> I don't see any problem with this, +1.
>>>
>>> Kind Regards,
>>> Brandon
>>>
>>>
>>> On Wed, Sep 13, 2023 at 11:09 AM Mike Adamson <ma...@datastax.com>
>>> wrote:
>>>
>>>> CEP-30: [Approximate Nearest Neighbor(ANN) Vector Search via
>>>> Storage-Attached Indexes] uses the smile-nlp library
>>>> (com.github.haifengl.smile-nlp) in its testing to allow the creation of
>>>> word2vec embeddings for valid input into the HNSW graph index.
>>>>
>>>> The reason for this library is that we found that using random vectors
>>>> in testing produced very inconsistent results. Using the smile-nlp word2vec
>>>> implementation with the glove.3k.50d library produces repeatable results.
>>>>
>>>> Does anyone have any objections to the use of this library as a test
>>>> only dependency?
>>>> --
>>>> [image: DataStax Logo Square] <https://www.datastax.com/> *Mike
>>>> Adamson*
>>>> Engineering
>>>>
>>>> +1 650 389 6000 <16503896000> | datastax.com
>>>> <https://www.datastax.com/>
>>>> Find DataStax Online: [image: LinkedIn Logo]
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
>>>> [image: Facebook Logo]
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
>>>> [image: Twitter Logo] <https://twitter.com/DataStax> [image: RSS
>>>> Feed] <https://www.datastax.com/blog/rss.xml> [image: Github Logo]
>>>> <https://github.com/datastax>
>>>>
>>>>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Ekaterina Dimitrova <e....@gmail.com>.
Jeff, isn’t this ok as long as it is used only in tests? If we are not sure
we can open a Jira to legal?
On Wed, 13 Sep 2023 at 12:23, Jeff Jirsa <jj...@gmail.com> wrote:
> Just to be clear - this repo?
> https://github.com/haifengl/smile/blob/master/LICENSE
>
> That shows GPL + Commercial?
>
>
>
> On Wed, Sep 13, 2023 at 9:10 AM Brandon Williams <dr...@gmail.com> wrote:
>
>> I don't see any problem with this, +1.
>>
>> Kind Regards,
>> Brandon
>>
>>
>> On Wed, Sep 13, 2023 at 11:09 AM Mike Adamson <ma...@datastax.com>
>> wrote:
>>
>>> CEP-30: [Approximate Nearest Neighbor(ANN) Vector Search via
>>> Storage-Attached Indexes] uses the smile-nlp library
>>> (com.github.haifengl.smile-nlp) in its testing to allow the creation of
>>> word2vec embeddings for valid input into the HNSW graph index.
>>>
>>> The reason for this library is that we found that using random vectors
>>> in testing produced very inconsistent results. Using the smile-nlp word2vec
>>> implementation with the glove.3k.50d library produces repeatable results.
>>>
>>> Does anyone have any objections to the use of this library as a test
>>> only dependency?
>>> --
>>> [image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
>>> Engineering
>>>
>>> +1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
>>> Find DataStax Online: [image: LinkedIn Logo]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
>>> [image: Facebook Logo]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
>>> [image: Twitter Logo] <https://twitter.com/DataStax> [image: RSS
>>> Feed] <https://www.datastax.com/blog/rss.xml> [image: Github Logo]
>>> <https://github.com/datastax>
>>>
>>>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Jeff Jirsa <jj...@gmail.com>.
Just to be clear - this repo?
https://github.com/haifengl/smile/blob/master/LICENSE
That shows GPL + Commercial?
On Wed, Sep 13, 2023 at 9:10 AM Brandon Williams <dr...@gmail.com> wrote:
> I don't see any problem with this, +1.
>
> Kind Regards,
> Brandon
>
>
> On Wed, Sep 13, 2023 at 11:09 AM Mike Adamson <ma...@datastax.com>
> wrote:
>
>> CEP-30: [Approximate Nearest Neighbor(ANN) Vector Search via
>> Storage-Attached Indexes] uses the smile-nlp library
>> (com.github.haifengl.smile-nlp) in its testing to allow the creation of
>> word2vec embeddings for valid input into the HNSW graph index.
>>
>> The reason for this library is that we found that using random vectors in
>> testing produced very inconsistent results. Using the smile-nlp word2vec
>> implementation with the glove.3k.50d library produces repeatable results.
>>
>> Does anyone have any objections to the use of this library as a test only
>> dependency?
>> --
>> [image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
>> Engineering
>>
>> +1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
>> Find DataStax Online: [image: LinkedIn Logo]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
>> [image: Facebook Logo]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
>> [image: Twitter Logo] <https://twitter.com/DataStax> [image: RSS
>> Feed] <https://www.datastax.com/blog/rss.xml> [image: Github Logo]
>> <https://github.com/datastax>
>>
>>
Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30
Posted by Brandon Williams <dr...@gmail.com>.
I don't see any problem with this, +1.
Kind Regards,
Brandon
On Wed, Sep 13, 2023 at 11:09 AM Mike Adamson <ma...@datastax.com> wrote:
> CEP-30: [Approximate Nearest Neighbor(ANN) Vector Search via
> Storage-Attached Indexes] uses the smile-nlp library
> (com.github.haifengl.smile-nlp) in its testing to allow the creation of
> word2vec embeddings for valid input into the HNSW graph index.
>
> The reason for this library is that we found that using random vectors in
> testing produced very inconsistent results. Using the smile-nlp word2vec
> implementation with the glove.3k.50d library produces repeatable results.
>
> Does anyone have any objections to the use of this library as a test only
> dependency?
> --
> [image: DataStax Logo Square] <https://www.datastax.com/> *Mike Adamson*
> Engineering
>
> +1 650 389 6000 <16503896000> | datastax.com <https://www.datastax.com/>
> Find DataStax Online: [image: LinkedIn Logo]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=akx0E6l2bnTjOvA-YxtonbW0M4b6bNg4nRwmcHNDo4Q&e=>
> [image: Facebook Logo]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_datastax&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=IFj3MdIKYLLXIUhYdUGB0cTzTlxyCb7_VUmICBaYilU&m=uHzE4WhPViSF0rsjSxKhfwGDU1Bo7USObSc_aIcgelo&s=ncMlB41-6hHuqx-EhnM83-KVtjMegQ9c2l2zDzHAxiU&e=>
> [image: Twitter Logo] <https://twitter.com/DataStax> [image: RSS
> Feed] <https://www.datastax.com/blog/rss.xml> [image: Github Logo]
> <https://github.com/datastax>
>
>