You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Wes McKinney <we...@gmail.com> on 2021/05/14 13:30:03 UTC
[C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
hi folks,
As we build more functions (kernels) in the project, I note that the
amount of hand-coded C++ code relating to testing function correctness
is growing significantly. Many of these tests are quite simple and
could be expressed in a text format that can be parsed and evaluated.
Thoughts about building something like that to make it easier to write
functional correctness tests?
- Wes
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Benjamin Kietzman <be...@gmail.com>.
I've added https://issues.apache.org/jira/browse/ARROW-13013
to track moving kernel unit tests to Python since that seems easily
doable and worthwhile
On Sun, May 16, 2021 at 3:35 PM Wes McKinney <we...@gmail.com> wrote:
> I agree there are pros and cons here (up front investment hopefully
> yielding future productivity gains). If a test harness and format
> meeting the requirements could be created without too much pain (I'm
> thinking less than a week of full time effort, including refactoring
> some existing tests), it would save developers a lot of time going
> forward. I look for example at the large number of hand-coded
> functional tests found in this file as one example to use to guide the
> effort:
>
>
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/scalar_arithmetic_test.cc
>
> On Sat, May 15, 2021 at 5:02 AM Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > I think people who think this would be beneficial should try to devise a
> > text format to represent compute test data. As Eduardo pointed out,
> > there are various complications that need to be catered for.
> >
> > To me, it's not obvious that building the necessary infrastructure in
> > C++ to ingest that text format will be more pleasant than our current
> > way of writing tests. As a data point, the JSON integration code in C++
> > is really annoying to maintain.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 15/05/2021 à 00:03, Wes McKinney a écrit :
> > > In C++, we have the "ArrayFromJSON" function which is an even simpler
> > > way of specifying input data compared with the integration tests.
> > > That's one possible starting point.
> > >
> > > The "interpreted tests" could be all specified and driven by minimal
> > > dependency Python code, as one possible way to approach things.
> > >
> > > On Fri, May 14, 2021 at 1:57 PM Jorge Cardoso Leitão
> > > <jo...@gmail.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> (this problem also exists in Rust, btw)
> > >>
> > >> Couldn't we use something like we do for our integration tests?
> Create a
> > >> separate binary that would allow us to call e.g.
> > >>
> > >> test-compute --method equal --json-file <path> --arg "column1" --arg
> > >> "column2" --expected "column3"
> > >>
> > >> (or simply pass the input via stdin)
> > >>
> > >> and then use Python to call the binary?
> > >>
> > >> The advantage I see here is that we would compile the binary with
> flags to
> > >> disable unnecessary code, use debug, etc, thereby reducing compile
> time if
> > >> the kernel needs
> > >> to be changed.
> > >>
> > >> IMO our "json integration" format is a reliable way of passing data
> across,
> > >> it is very easy to read and write, and all our implementations can
> already
> > >> read it for integration tests.
> > >>
> > >> wrt to the "cross implementation", the equality operation seems a
> natural
> > >> candidate for across implementations checks, as that one has important
> > >> implications in all our integration tests. filter, take, slice,
> boolean ops
> > >> may also be easy to agree upon. "add" and the like are a bit more
> difficult
> > >> due to how overflow should be handled (abort vs saturate vs None), but
> > >> nothing that we can't take. ^_^
> > >>
> > >> Best,
> > >> Jorge
> > >>
> > >> On Fri, May 14, 2021 at 8:25 PM David Li <li...@apache.org> wrote:
> > >>
> > >>> I think even if it's not (easily) generalizable across languages,
> it'd
> > >>> still be a win for C++ (and hopefully languages that bind to
> > >>> C++). Also, I don't think they're meant to completely replace
> > >>> language-specific tests, but rather complement them, and make it
> > >>> easier to add and maintain tests in the overwhelmingly common case.
> > >>>
> > >>> I do feel it's somewhat painful to write these kinds of tests in C++,
> > >>> largely because of the iteration time and the difficulty of repeating
> > >>> tests across various configurations. I also think this could be an
> > >>> opportunity to leverage things like Hypothesis/property-based testing
> > >>> or perhaps fuzzing to make the kernels even more robust.
> > >>>
> > >>> -David
> > >>>
> > >>> On 2021/05/14 18:09:45, Eduardo Ponce <ed...@gmail.com> wrote:
> > >>>> Another aspect to keep in mind is that some tests require internal
> > >>> options
> > >>>> to be changed before executing the compute functions (e.g., check
> > >>> overflow,
> > >>>> allow NaN comparisons, change validity bits, etc.). Also, there are
> tests
> > >>>> that take randomized inputs and others make use of the min/max
> values for
> > >>>> each specific data type. Certainly, these details can be generalized
> > >>> across
> > >>>> languages/testing frameworks but not without careful treatment.
> > >>>>
> > >>>> Moreover, each language implementation still needs to test
> > >>>> language-specific or internal functions, so having a meta test
> framework
> > >>>> will not necessarily get rid of language-specific tests.
> > >>>>
> > >>>> ~Eduardo
> > >>>>
> > >>>> On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com>
> > >>> wrote:
> > >>>>
> > >>>>> With that in mind it seems the somewhat recurring discussion on
> coming
> > >>>>> up with a language independent standard for logical query plans
> > >>>>> (
> > >>>>>
> > >>>
> https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
> > >>>>> )
> > >>>>> would be relevant. Each test case would then be a triplet of
> (Input
> > >>>>> Dataframe, Logical Plan, Output Dataframe). So perhaps tackling
> this
> > >>>>> effort would be to make progress on both fronts.
> > >>>>>
> > >>>>> On Fri, May 14, 2021 at 7:39 AM Julian Hyde <
> jhyde.apache@gmail.com>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> Do these any of these compute functions have analogs in other
> > >>>>> implementations of Arrow (e.g. Rust)?
> > >>>>>>
> > >>>>>> I believe that as much as possible of Arrow’s compute
> functionality
> > >>>>> should be cross-language. Perhaps there are language-specific
> > >>> differences
> > >>>>> in how functions are invoked, but the basic functionality is the
> same.
> > >>>>>>
> > >>>>>> If people buy into that premise, then a single suite of tests is a
> > >>>>> powerful way to make that happen. The tests can be written in a
> > >>> high-level
> > >>>>> language and can generate tests in each implementation language.
> (For
> > >>> these
> > >>>>> purposes, the “high-level language” could be a special text format,
> > >>> could
> > >>>>> be a data language such as JSON, or could be a programming language
> > >>> such as
> > >>>>> Python; it doesn’t matter much.)
> > >>>>>>
> > >>>>>> For example,
> > >>>>>>
> > >>>>>> assertThatCall(“foo(1, 2)”, returns(“3”))
> > >>>>>>
> > >>>>>> might actually call foo with arguments 1 and 2, or it might
> generate
> > >>> a
> > >>>>> C++ or Rust test that does the same.
> > >>>>>>
> > >>>>>>
> > >>>>>> Julian
> > >>>>>>
> > >>>>>>
> > >>>>>>> On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> > >>>>>>>> hi folks,
> > >>>>>>>> As we build more functions (kernels) in the project, I note that
> > >>> the
> > >>>>>>>> amount of hand-coded C++ code relating to testing function
> > >>> correctness
> > >>>>>>>> is growing significantly. Many of these tests are quite simple
> and
> > >>>>>>>> could be expressed in a text format that can be parsed and
> > >>> evaluated.
> > >>>>>>>> Thoughts about building something like that to make it easier to
> > >>> write
> > >>>>>>>> functional correctness tests?
> > >>>>>>>
> > >>>>>>> Or perhaps build-up higher level C++ functions if desired?
> > >>>>>>>
> > >>>>>>> Or even write some of those tests as part of the PyArrow test
> > >>> suite.
> > >>>>>>>
> > >>>>>>> I'm not sure adding a custom (and probably inflexible) text
> format
> > >>> is
> > >>>>> really a good use of our time.
> > >>>>>>>
> > >>>>>>> Regards
> > >>>>>>>
> > >>>>>>> Antoine.
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Wes McKinney <we...@gmail.com>.
I agree there are pros and cons here (up front investment hopefully
yielding future productivity gains). If a test harness and format
meeting the requirements could be created without too much pain (I'm
thinking less than a week of full time effort, including refactoring
some existing tests), it would save developers a lot of time going
forward. I look for example at the large number of hand-coded
functional tests found in this file as one example to use to guide the
effort:
https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/scalar_arithmetic_test.cc
On Sat, May 15, 2021 at 5:02 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> I think people who think this would be beneficial should try to devise a
> text format to represent compute test data. As Eduardo pointed out,
> there are various complications that need to be catered for.
>
> To me, it's not obvious that building the necessary infrastructure in
> C++ to ingest that text format will be more pleasant than our current
> way of writing tests. As a data point, the JSON integration code in C++
> is really annoying to maintain.
>
> Regards
>
> Antoine.
>
>
> Le 15/05/2021 à 00:03, Wes McKinney a écrit :
> > In C++, we have the "ArrayFromJSON" function which is an even simpler
> > way of specifying input data compared with the integration tests.
> > That's one possible starting point.
> >
> > The "interpreted tests" could be all specified and driven by minimal
> > dependency Python code, as one possible way to approach things.
> >
> > On Fri, May 14, 2021 at 1:57 PM Jorge Cardoso Leitão
> > <jo...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> (this problem also exists in Rust, btw)
> >>
> >> Couldn't we use something like we do for our integration tests? Create a
> >> separate binary that would allow us to call e.g.
> >>
> >> test-compute --method equal --json-file <path> --arg "column1" --arg
> >> "column2" --expected "column3"
> >>
> >> (or simply pass the input via stdin)
> >>
> >> and then use Python to call the binary?
> >>
> >> The advantage I see here is that we would compile the binary with flags to
> >> disable unnecessary code, use debug, etc, thereby reducing compile time if
> >> the kernel needs
> >> to be changed.
> >>
> >> IMO our "json integration" format is a reliable way of passing data across,
> >> it is very easy to read and write, and all our implementations can already
> >> read it for integration tests.
> >>
> >> wrt to the "cross implementation", the equality operation seems a natural
> >> candidate for across implementations checks, as that one has important
> >> implications in all our integration tests. filter, take, slice, boolean ops
> >> may also be easy to agree upon. "add" and the like are a bit more difficult
> >> due to how overflow should be handled (abort vs saturate vs None), but
> >> nothing that we can't take. ^_^
> >>
> >> Best,
> >> Jorge
> >>
> >> On Fri, May 14, 2021 at 8:25 PM David Li <li...@apache.org> wrote:
> >>
> >>> I think even if it's not (easily) generalizable across languages, it'd
> >>> still be a win for C++ (and hopefully languages that bind to
> >>> C++). Also, I don't think they're meant to completely replace
> >>> language-specific tests, but rather complement them, and make it
> >>> easier to add and maintain tests in the overwhelmingly common case.
> >>>
> >>> I do feel it's somewhat painful to write these kinds of tests in C++,
> >>> largely because of the iteration time and the difficulty of repeating
> >>> tests across various configurations. I also think this could be an
> >>> opportunity to leverage things like Hypothesis/property-based testing
> >>> or perhaps fuzzing to make the kernels even more robust.
> >>>
> >>> -David
> >>>
> >>> On 2021/05/14 18:09:45, Eduardo Ponce <ed...@gmail.com> wrote:
> >>>> Another aspect to keep in mind is that some tests require internal
> >>> options
> >>>> to be changed before executing the compute functions (e.g., check
> >>> overflow,
> >>>> allow NaN comparisons, change validity bits, etc.). Also, there are tests
> >>>> that take randomized inputs and others make use of the min/max values for
> >>>> each specific data type. Certainly, these details can be generalized
> >>> across
> >>>> languages/testing frameworks but not without careful treatment.
> >>>>
> >>>> Moreover, each language implementation still needs to test
> >>>> language-specific or internal functions, so having a meta test framework
> >>>> will not necessarily get rid of language-specific tests.
> >>>>
> >>>> ~Eduardo
> >>>>
> >>>> On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> With that in mind it seems the somewhat recurring discussion on coming
> >>>>> up with a language independent standard for logical query plans
> >>>>> (
> >>>>>
> >>> https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
> >>>>> )
> >>>>> would be relevant. Each test case would then be a triplet of (Input
> >>>>> Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
> >>>>> effort would be to make progress on both fronts.
> >>>>>
> >>>>> On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Do these any of these compute functions have analogs in other
> >>>>> implementations of Arrow (e.g. Rust)?
> >>>>>>
> >>>>>> I believe that as much as possible of Arrow’s compute functionality
> >>>>> should be cross-language. Perhaps there are language-specific
> >>> differences
> >>>>> in how functions are invoked, but the basic functionality is the same.
> >>>>>>
> >>>>>> If people buy into that premise, then a single suite of tests is a
> >>>>> powerful way to make that happen. The tests can be written in a
> >>> high-level
> >>>>> language and can generate tests in each implementation language. (For
> >>> these
> >>>>> purposes, the “high-level language” could be a special text format,
> >>> could
> >>>>> be a data language such as JSON, or could be a programming language
> >>> such as
> >>>>> Python; it doesn’t matter much.)
> >>>>>>
> >>>>>> For example,
> >>>>>>
> >>>>>> assertThatCall(“foo(1, 2)”, returns(“3”))
> >>>>>>
> >>>>>> might actually call foo with arguments 1 and 2, or it might generate
> >>> a
> >>>>> C++ or Rust test that does the same.
> >>>>>>
> >>>>>>
> >>>>>> Julian
> >>>>>>
> >>>>>>
> >>>>>>> On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> >>>>>>>> hi folks,
> >>>>>>>> As we build more functions (kernels) in the project, I note that
> >>> the
> >>>>>>>> amount of hand-coded C++ code relating to testing function
> >>> correctness
> >>>>>>>> is growing significantly. Many of these tests are quite simple and
> >>>>>>>> could be expressed in a text format that can be parsed and
> >>> evaluated.
> >>>>>>>> Thoughts about building something like that to make it easier to
> >>> write
> >>>>>>>> functional correctness tests?
> >>>>>>>
> >>>>>>> Or perhaps build-up higher level C++ functions if desired?
> >>>>>>>
> >>>>>>> Or even write some of those tests as part of the PyArrow test
> >>> suite.
> >>>>>>>
> >>>>>>> I'm not sure adding a custom (and probably inflexible) text format
> >>> is
> >>>>> really a good use of our time.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>>
> >>>>>>> Antoine.
> >>>>>>
> >>>>>
> >>>>
> >>>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Antoine Pitrou <an...@python.org>.
I think people who think this would be beneficial should try to devise a
text format to represent compute test data. As Eduardo pointed out,
there are various complications that need to be catered for.
To me, it's not obvious that building the necessary infrastructure in
C++ to ingest that text format will be more pleasant than our current
way of writing tests. As a data point, the JSON integration code in C++
is really annoying to maintain.
Regards
Antoine.
Le 15/05/2021 à 00:03, Wes McKinney a écrit :
> In C++, we have the "ArrayFromJSON" function which is an even simpler
> way of specifying input data compared with the integration tests.
> That's one possible starting point.
>
> The "interpreted tests" could be all specified and driven by minimal
> dependency Python code, as one possible way to approach things.
>
> On Fri, May 14, 2021 at 1:57 PM Jorge Cardoso Leitão
> <jo...@gmail.com> wrote:
>>
>> Hi,
>>
>> (this problem also exists in Rust, btw)
>>
>> Couldn't we use something like we do for our integration tests? Create a
>> separate binary that would allow us to call e.g.
>>
>> test-compute --method equal --json-file <path> --arg "column1" --arg
>> "column2" --expected "column3"
>>
>> (or simply pass the input via stdin)
>>
>> and then use Python to call the binary?
>>
>> The advantage I see here is that we would compile the binary with flags to
>> disable unnecessary code, use debug, etc, thereby reducing compile time if
>> the kernel needs
>> to be changed.
>>
>> IMO our "json integration" format is a reliable way of passing data across,
>> it is very easy to read and write, and all our implementations can already
>> read it for integration tests.
>>
>> wrt to the "cross implementation", the equality operation seems a natural
>> candidate for across implementations checks, as that one has important
>> implications in all our integration tests. filter, take, slice, boolean ops
>> may also be easy to agree upon. "add" and the like are a bit more difficult
>> due to how overflow should be handled (abort vs saturate vs None), but
>> nothing that we can't take. ^_^
>>
>> Best,
>> Jorge
>>
>> On Fri, May 14, 2021 at 8:25 PM David Li <li...@apache.org> wrote:
>>
>>> I think even if it's not (easily) generalizable across languages, it'd
>>> still be a win for C++ (and hopefully languages that bind to
>>> C++). Also, I don't think they're meant to completely replace
>>> language-specific tests, but rather complement them, and make it
>>> easier to add and maintain tests in the overwhelmingly common case.
>>>
>>> I do feel it's somewhat painful to write these kinds of tests in C++,
>>> largely because of the iteration time and the difficulty of repeating
>>> tests across various configurations. I also think this could be an
>>> opportunity to leverage things like Hypothesis/property-based testing
>>> or perhaps fuzzing to make the kernels even more robust.
>>>
>>> -David
>>>
>>> On 2021/05/14 18:09:45, Eduardo Ponce <ed...@gmail.com> wrote:
>>>> Another aspect to keep in mind is that some tests require internal
>>> options
>>>> to be changed before executing the compute functions (e.g., check
>>> overflow,
>>>> allow NaN comparisons, change validity bits, etc.). Also, there are tests
>>>> that take randomized inputs and others make use of the min/max values for
>>>> each specific data type. Certainly, these details can be generalized
>>> across
>>>> languages/testing frameworks but not without careful treatment.
>>>>
>>>> Moreover, each language implementation still needs to test
>>>> language-specific or internal functions, so having a meta test framework
>>>> will not necessarily get rid of language-specific tests.
>>>>
>>>> ~Eduardo
>>>>
>>>> On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com>
>>> wrote:
>>>>
>>>>> With that in mind it seems the somewhat recurring discussion on coming
>>>>> up with a language independent standard for logical query plans
>>>>> (
>>>>>
>>> https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
>>>>> )
>>>>> would be relevant. Each test case would then be a triplet of (Input
>>>>> Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
>>>>> effort would be to make progress on both fronts.
>>>>>
>>>>> On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Do these any of these compute functions have analogs in other
>>>>> implementations of Arrow (e.g. Rust)?
>>>>>>
>>>>>> I believe that as much as possible of Arrow’s compute functionality
>>>>> should be cross-language. Perhaps there are language-specific
>>> differences
>>>>> in how functions are invoked, but the basic functionality is the same.
>>>>>>
>>>>>> If people buy into that premise, then a single suite of tests is a
>>>>> powerful way to make that happen. The tests can be written in a
>>> high-level
>>>>> language and can generate tests in each implementation language. (For
>>> these
>>>>> purposes, the “high-level language” could be a special text format,
>>> could
>>>>> be a data language such as JSON, or could be a programming language
>>> such as
>>>>> Python; it doesn’t matter much.)
>>>>>>
>>>>>> For example,
>>>>>>
>>>>>> assertThatCall(“foo(1, 2)”, returns(“3”))
>>>>>>
>>>>>> might actually call foo with arguments 1 and 2, or it might generate
>>> a
>>>>> C++ or Rust test that does the same.
>>>>>>
>>>>>>
>>>>>> Julian
>>>>>>
>>>>>>
>>>>>>> On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Le 14/05/2021 à 15:30, Wes McKinney a écrit :
>>>>>>>> hi folks,
>>>>>>>> As we build more functions (kernels) in the project, I note that
>>> the
>>>>>>>> amount of hand-coded C++ code relating to testing function
>>> correctness
>>>>>>>> is growing significantly. Many of these tests are quite simple and
>>>>>>>> could be expressed in a text format that can be parsed and
>>> evaluated.
>>>>>>>> Thoughts about building something like that to make it easier to
>>> write
>>>>>>>> functional correctness tests?
>>>>>>>
>>>>>>> Or perhaps build-up higher level C++ functions if desired?
>>>>>>>
>>>>>>> Or even write some of those tests as part of the PyArrow test
>>> suite.
>>>>>>>
>>>>>>> I'm not sure adding a custom (and probably inflexible) text format
>>> is
>>>>> really a good use of our time.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Antoine.
>>>>>>
>>>>>
>>>>
>>>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Wes McKinney <we...@gmail.com>.
In C++, we have the "ArrayFromJSON" function which is an even simpler
way of specifying input data compared with the integration tests.
That's one possible starting point.
The "interpreted tests" could be all specified and driven by minimal
dependency Python code, as one possible way to approach things.
On Fri, May 14, 2021 at 1:57 PM Jorge Cardoso Leitão
<jo...@gmail.com> wrote:
>
> Hi,
>
> (this problem also exists in Rust, btw)
>
> Couldn't we use something like we do for our integration tests? Create a
> separate binary that would allow us to call e.g.
>
> test-compute --method equal --json-file <path> --arg "column1" --arg
> "column2" --expected "column3"
>
> (or simply pass the input via stdin)
>
> and then use Python to call the binary?
>
> The advantage I see here is that we would compile the binary with flags to
> disable unnecessary code, use debug, etc, thereby reducing compile time if
> the kernel needs
> to be changed.
>
> IMO our "json integration" format is a reliable way of passing data across,
> it is very easy to read and write, and all our implementations can already
> read it for integration tests.
>
> wrt to the "cross implementation", the equality operation seems a natural
> candidate for across implementations checks, as that one has important
> implications in all our integration tests. filter, take, slice, boolean ops
> may also be easy to agree upon. "add" and the like are a bit more difficult
> due to how overflow should be handled (abort vs saturate vs None), but
> nothing that we can't take. ^_^
>
> Best,
> Jorge
>
> On Fri, May 14, 2021 at 8:25 PM David Li <li...@apache.org> wrote:
>
> > I think even if it's not (easily) generalizable across languages, it'd
> > still be a win for C++ (and hopefully languages that bind to
> > C++). Also, I don't think they're meant to completely replace
> > language-specific tests, but rather complement them, and make it
> > easier to add and maintain tests in the overwhelmingly common case.
> >
> > I do feel it's somewhat painful to write these kinds of tests in C++,
> > largely because of the iteration time and the difficulty of repeating
> > tests across various configurations. I also think this could be an
> > opportunity to leverage things like Hypothesis/property-based testing
> > or perhaps fuzzing to make the kernels even more robust.
> >
> > -David
> >
> > On 2021/05/14 18:09:45, Eduardo Ponce <ed...@gmail.com> wrote:
> > > Another aspect to keep in mind is that some tests require internal
> > options
> > > to be changed before executing the compute functions (e.g., check
> > overflow,
> > > allow NaN comparisons, change validity bits, etc.). Also, there are tests
> > > that take randomized inputs and others make use of the min/max values for
> > > each specific data type. Certainly, these details can be generalized
> > across
> > > languages/testing frameworks but not without careful treatment.
> > >
> > > Moreover, each language implementation still needs to test
> > > language-specific or internal functions, so having a meta test framework
> > > will not necessarily get rid of language-specific tests.
> > >
> > > ~Eduardo
> > >
> > > On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com>
> > wrote:
> > >
> > > > With that in mind it seems the somewhat recurring discussion on coming
> > > > up with a language independent standard for logical query plans
> > > > (
> > > >
> > https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
> > > > )
> > > > would be relevant. Each test case would then be a triplet of (Input
> > > > Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
> > > > effort would be to make progress on both fronts.
> > > >
> > > > On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Do these any of these compute functions have analogs in other
> > > > implementations of Arrow (e.g. Rust)?
> > > > >
> > > > > I believe that as much as possible of Arrow’s compute functionality
> > > > should be cross-language. Perhaps there are language-specific
> > differences
> > > > in how functions are invoked, but the basic functionality is the same.
> > > > >
> > > > > If people buy into that premise, then a single suite of tests is a
> > > > powerful way to make that happen. The tests can be written in a
> > high-level
> > > > language and can generate tests in each implementation language. (For
> > these
> > > > purposes, the “high-level language” could be a special text format,
> > could
> > > > be a data language such as JSON, or could be a programming language
> > such as
> > > > Python; it doesn’t matter much.)
> > > > >
> > > > > For example,
> > > > >
> > > > > assertThatCall(“foo(1, 2)”, returns(“3”))
> > > > >
> > > > > might actually call foo with arguments 1 and 2, or it might generate
> > a
> > > > C++ or Rust test that does the same.
> > > > >
> > > > >
> > > > > Julian
> > > > >
> > > > >
> > > > > > On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
> > > > wrote:
> > > > > >
> > > > > >
> > > > > > Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> > > > > >> hi folks,
> > > > > >> As we build more functions (kernels) in the project, I note that
> > the
> > > > > >> amount of hand-coded C++ code relating to testing function
> > correctness
> > > > > >> is growing significantly. Many of these tests are quite simple and
> > > > > >> could be expressed in a text format that can be parsed and
> > evaluated.
> > > > > >> Thoughts about building something like that to make it easier to
> > write
> > > > > >> functional correctness tests?
> > > > > >
> > > > > > Or perhaps build-up higher level C++ functions if desired?
> > > > > >
> > > > > > Or even write some of those tests as part of the PyArrow test
> > suite.
> > > > > >
> > > > > > I'm not sure adding a custom (and probably inflexible) text format
> > is
> > > > really a good use of our time.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Antoine.
> > > > >
> > > >
> > >
> >
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
Hi,
(this problem also exists in Rust, btw)
Couldn't we use something like we do for our integration tests? Create a
separate binary that would allow us to call e.g.
test-compute --method equal --json-file <path> --arg "column1" --arg
"column2" --expected "column3"
(or simply pass the input via stdin)
and then use Python to call the binary?
The advantage I see here is that we would compile the binary with flags to
disable unnecessary code, use debug, etc, thereby reducing compile time if
the kernel needs
to be changed.
IMO our "json integration" format is a reliable way of passing data across,
it is very easy to read and write, and all our implementations can already
read it for integration tests.
wrt to the "cross implementation", the equality operation seems a natural
candidate for across implementations checks, as that one has important
implications in all our integration tests. filter, take, slice, boolean ops
may also be easy to agree upon. "add" and the like are a bit more difficult
due to how overflow should be handled (abort vs saturate vs None), but
nothing that we can't take. ^_^
Best,
Jorge
On Fri, May 14, 2021 at 8:25 PM David Li <li...@apache.org> wrote:
> I think even if it's not (easily) generalizable across languages, it'd
> still be a win for C++ (and hopefully languages that bind to
> C++). Also, I don't think they're meant to completely replace
> language-specific tests, but rather complement them, and make it
> easier to add and maintain tests in the overwhelmingly common case.
>
> I do feel it's somewhat painful to write these kinds of tests in C++,
> largely because of the iteration time and the difficulty of repeating
> tests across various configurations. I also think this could be an
> opportunity to leverage things like Hypothesis/property-based testing
> or perhaps fuzzing to make the kernels even more robust.
>
> -David
>
> On 2021/05/14 18:09:45, Eduardo Ponce <ed...@gmail.com> wrote:
> > Another aspect to keep in mind is that some tests require internal
> options
> > to be changed before executing the compute functions (e.g., check
> overflow,
> > allow NaN comparisons, change validity bits, etc.). Also, there are tests
> > that take randomized inputs and others make use of the min/max values for
> > each specific data type. Certainly, these details can be generalized
> across
> > languages/testing frameworks but not without careful treatment.
> >
> > Moreover, each language implementation still needs to test
> > language-specific or internal functions, so having a meta test framework
> > will not necessarily get rid of language-specific tests.
> >
> > ~Eduardo
> >
> > On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com>
> wrote:
> >
> > > With that in mind it seems the somewhat recurring discussion on coming
> > > up with a language independent standard for logical query plans
> > > (
> > >
> https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
> > > )
> > > would be relevant. Each test case would then be a triplet of (Input
> > > Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
> > > effort would be to make progress on both fronts.
> > >
> > > On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com>
> > > wrote:
> > > >
> > > > Do these any of these compute functions have analogs in other
> > > implementations of Arrow (e.g. Rust)?
> > > >
> > > > I believe that as much as possible of Arrow’s compute functionality
> > > should be cross-language. Perhaps there are language-specific
> differences
> > > in how functions are invoked, but the basic functionality is the same.
> > > >
> > > > If people buy into that premise, then a single suite of tests is a
> > > powerful way to make that happen. The tests can be written in a
> high-level
> > > language and can generate tests in each implementation language. (For
> these
> > > purposes, the “high-level language” could be a special text format,
> could
> > > be a data language such as JSON, or could be a programming language
> such as
> > > Python; it doesn’t matter much.)
> > > >
> > > > For example,
> > > >
> > > > assertThatCall(“foo(1, 2)”, returns(“3”))
> > > >
> > > > might actually call foo with arguments 1 and 2, or it might generate
> a
> > > C++ or Rust test that does the same.
> > > >
> > > >
> > > > Julian
> > > >
> > > >
> > > > > On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
> > > wrote:
> > > > >
> > > > >
> > > > > Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> > > > >> hi folks,
> > > > >> As we build more functions (kernels) in the project, I note that
> the
> > > > >> amount of hand-coded C++ code relating to testing function
> correctness
> > > > >> is growing significantly. Many of these tests are quite simple and
> > > > >> could be expressed in a text format that can be parsed and
> evaluated.
> > > > >> Thoughts about building something like that to make it easier to
> write
> > > > >> functional correctness tests?
> > > > >
> > > > > Or perhaps build-up higher level C++ functions if desired?
> > > > >
> > > > > Or even write some of those tests as part of the PyArrow test
> suite.
> > > > >
> > > > > I'm not sure adding a custom (and probably inflexible) text format
> is
> > > really a good use of our time.
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > >
> > >
> >
>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for compute functions
Posted by David Li <li...@apache.org>.
I think even if it's not (easily) generalizable across languages, it'd
still be a win for C++ (and hopefully languages that bind to
C++). Also, I don't think they're meant to completely replace
language-specific tests, but rather complement them, and make it
easier to add and maintain tests in the overwhelmingly common case.
I do feel it's somewhat painful to write these kinds of tests in C++,
largely because of the iteration time and the difficulty of repeating
tests across various configurations. I also think this could be an
opportunity to leverage things like Hypothesis/property-based testing
or perhaps fuzzing to make the kernels even more robust.
-David
On 2021/05/14 18:09:45, Eduardo Ponce <ed...@gmail.com> wrote:
> Another aspect to keep in mind is that some tests require internal options
> to be changed before executing the compute functions (e.g., check overflow,
> allow NaN comparisons, change validity bits, etc.). Also, there are tests
> that take randomized inputs and others make use of the min/max values for
> each specific data type. Certainly, these details can be generalized across
> languages/testing frameworks but not without careful treatment.
>
> Moreover, each language implementation still needs to test
> language-specific or internal functions, so having a meta test framework
> will not necessarily get rid of language-specific tests.
>
> ~Eduardo
>
> On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com> wrote:
>
> > With that in mind it seems the somewhat recurring discussion on coming
> > up with a language independent standard for logical query plans
> > (
> > https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
> > )
> > would be relevant. Each test case would then be a triplet of (Input
> > Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
> > effort would be to make progress on both fronts.
> >
> > On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com>
> > wrote:
> > >
> > > Do these any of these compute functions have analogs in other
> > implementations of Arrow (e.g. Rust)?
> > >
> > > I believe that as much as possible of Arrow’s compute functionality
> > should be cross-language. Perhaps there are language-specific differences
> > in how functions are invoked, but the basic functionality is the same.
> > >
> > > If people buy into that premise, then a single suite of tests is a
> > powerful way to make that happen. The tests can be written in a high-level
> > language and can generate tests in each implementation language. (For these
> > purposes, the “high-level language” could be a special text format, could
> > be a data language such as JSON, or could be a programming language such as
> > Python; it doesn’t matter much.)
> > >
> > > For example,
> > >
> > > assertThatCall(“foo(1, 2)”, returns(“3”))
> > >
> > > might actually call foo with arguments 1 and 2, or it might generate a
> > C++ or Rust test that does the same.
> > >
> > >
> > > Julian
> > >
> > >
> > > > On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
> > wrote:
> > > >
> > > >
> > > > Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> > > >> hi folks,
> > > >> As we build more functions (kernels) in the project, I note that the
> > > >> amount of hand-coded C++ code relating to testing function correctness
> > > >> is growing significantly. Many of these tests are quite simple and
> > > >> could be expressed in a text format that can be parsed and evaluated.
> > > >> Thoughts about building something like that to make it easier to write
> > > >> functional correctness tests?
> > > >
> > > > Or perhaps build-up higher level C++ functions if desired?
> > > >
> > > > Or even write some of those tests as part of the PyArrow test suite.
> > > >
> > > > I'm not sure adding a custom (and probably inflexible) text format is
> > really a good use of our time.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > >
> >
>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Eduardo Ponce <ed...@gmail.com>.
Another aspect to keep in mind is that some tests require internal options
to be changed before executing the compute functions (e.g., check overflow,
allow NaN comparisons, change validity bits, etc.). Also, there are tests
that take randomized inputs and others make use of the min/max values for
each specific data type. Certainly, these details can be generalized across
languages/testing frameworks but not without careful treatment.
Moreover, each language implementation still needs to test
language-specific or internal functions, so having a meta test framework
will not necessarily get rid of language-specific tests.
~Eduardo
On Fri, May 14, 2021 at 1:56 PM Weston Pace <we...@gmail.com> wrote:
> With that in mind it seems the somewhat recurring discussion on coming
> up with a language independent standard for logical query plans
> (
> https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E
> )
> would be relevant. Each test case would then be a triplet of (Input
> Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
> effort would be to make progress on both fronts.
>
> On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com>
> wrote:
> >
> > Do these any of these compute functions have analogs in other
> implementations of Arrow (e.g. Rust)?
> >
> > I believe that as much as possible of Arrow’s compute functionality
> should be cross-language. Perhaps there are language-specific differences
> in how functions are invoked, but the basic functionality is the same.
> >
> > If people buy into that premise, then a single suite of tests is a
> powerful way to make that happen. The tests can be written in a high-level
> language and can generate tests in each implementation language. (For these
> purposes, the “high-level language” could be a special text format, could
> be a data language such as JSON, or could be a programming language such as
> Python; it doesn’t matter much.)
> >
> > For example,
> >
> > assertThatCall(“foo(1, 2)”, returns(“3”))
> >
> > might actually call foo with arguments 1 and 2, or it might generate a
> C++ or Rust test that does the same.
> >
> >
> > Julian
> >
> >
> > > On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org>
> wrote:
> > >
> > >
> > > Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> > >> hi folks,
> > >> As we build more functions (kernels) in the project, I note that the
> > >> amount of hand-coded C++ code relating to testing function correctness
> > >> is growing significantly. Many of these tests are quite simple and
> > >> could be expressed in a text format that can be parsed and evaluated.
> > >> Thoughts about building something like that to make it easier to write
> > >> functional correctness tests?
> > >
> > > Or perhaps build-up higher level C++ functions if desired?
> > >
> > > Or even write some of those tests as part of the PyArrow test suite.
> > >
> > > I'm not sure adding a custom (and probably inflexible) text format is
> really a good use of our time.
> > >
> > > Regards
> > >
> > > Antoine.
> >
>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Wes McKinney <we...@gmail.com>.
Regardless of the amount of work involved with writing a test
harness/driver (which could be mostly in Python as one possibility), I
find it hard to believe that writing functional tests that look like
this->CheckUnary("utf8_reverse", "[\"ɑɽⱤæÆ&\", null, \"\", \"bbb\"]",
this->type(),
"[\"&ÆæⱤɽɑ\", null, \"\", \"bbb\"]");
is the best that we can do. There are various issues:
* Adding more tests requires recompilation each time, and the larger
the test suite, the longer our compile times are. Interpreted tests
have no compile-time cost.
* Specifying data inputs as string literals in C++ is more complex
than in JSON or Python, say.
* Tests require interacting with various C++ specific constructs like
data types, and utilizing parametric test structures in Googletest to
run the same tests across data types (like utf8/large_utf8) that we
expect to behave similarly.
* To Julian's point, the tests are C++ specific
There's other C++-specific requirements, like:
* It must not be too difficult to use gdb to set breakpoints when debugging
In an ideal world the interpreted test harness for C++ would not
require building pyarrow (since this would make things more difficult
for people who are just interacting with the C++ project).
On Fri, May 14, 2021 at 12:56 PM Weston Pace <we...@gmail.com> wrote:
>
> With that in mind it seems the somewhat recurring discussion on coming
> up with a language independent standard for logical query plans
> (https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E)
> would be relevant. Each test case would then be a triplet of (Input
> Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
> effort would be to make progress on both fronts.
>
> On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com> wrote:
> >
> > Do these any of these compute functions have analogs in other implementations of Arrow (e.g. Rust)?
> >
> > I believe that as much as possible of Arrow’s compute functionality should be cross-language. Perhaps there are language-specific differences in how functions are invoked, but the basic functionality is the same.
> >
> > If people buy into that premise, then a single suite of tests is a powerful way to make that happen. The tests can be written in a high-level language and can generate tests in each implementation language. (For these purposes, the “high-level language” could be a special text format, could be a data language such as JSON, or could be a programming language such as Python; it doesn’t matter much.)
> >
> > For example,
> >
> > assertThatCall(“foo(1, 2)”, returns(“3”))
> >
> > might actually call foo with arguments 1 and 2, or it might generate a C++ or Rust test that does the same.
> >
> >
> > Julian
> >
> >
> > > On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org> wrote:
> > >
> > >
> > > Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> > >> hi folks,
> > >> As we build more functions (kernels) in the project, I note that the
> > >> amount of hand-coded C++ code relating to testing function correctness
> > >> is growing significantly. Many of these tests are quite simple and
> > >> could be expressed in a text format that can be parsed and evaluated.
> > >> Thoughts about building something like that to make it easier to write
> > >> functional correctness tests?
> > >
> > > Or perhaps build-up higher level C++ functions if desired?
> > >
> > > Or even write some of those tests as part of the PyArrow test suite.
> > >
> > > I'm not sure adding a custom (and probably inflexible) text format is really a good use of our time.
> > >
> > > Regards
> > >
> > > Antoine.
> >
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Weston Pace <we...@gmail.com>.
With that in mind it seems the somewhat recurring discussion on coming
up with a language independent standard for logical query plans
(https://lists.apache.org/thread.html/rfab15e09c97a8fb961d6c5db8b2093824c58d11a51981a40f40cc2c0%40%3Cdev.arrow.apache.org%3E)
would be relevant. Each test case would then be a triplet of (Input
Dataframe, Logical Plan, Output Dataframe). So perhaps tackling this
effort would be to make progress on both fronts.
On Fri, May 14, 2021 at 7:39 AM Julian Hyde <jh...@gmail.com> wrote:
>
> Do these any of these compute functions have analogs in other implementations of Arrow (e.g. Rust)?
>
> I believe that as much as possible of Arrow’s compute functionality should be cross-language. Perhaps there are language-specific differences in how functions are invoked, but the basic functionality is the same.
>
> If people buy into that premise, then a single suite of tests is a powerful way to make that happen. The tests can be written in a high-level language and can generate tests in each implementation language. (For these purposes, the “high-level language” could be a special text format, could be a data language such as JSON, or could be a programming language such as Python; it doesn’t matter much.)
>
> For example,
>
> assertThatCall(“foo(1, 2)”, returns(“3”))
>
> might actually call foo with arguments 1 and 2, or it might generate a C++ or Rust test that does the same.
>
>
> Julian
>
>
> > On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> >> hi folks,
> >> As we build more functions (kernels) in the project, I note that the
> >> amount of hand-coded C++ code relating to testing function correctness
> >> is growing significantly. Many of these tests are quite simple and
> >> could be expressed in a text format that can be parsed and evaluated.
> >> Thoughts about building something like that to make it easier to write
> >> functional correctness tests?
> >
> > Or perhaps build-up higher level C++ functions if desired?
> >
> > Or even write some of those tests as part of the PyArrow test suite.
> >
> > I'm not sure adding a custom (and probably inflexible) text format is really a good use of our time.
> >
> > Regards
> >
> > Antoine.
>
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Julian Hyde <jh...@gmail.com>.
Do these any of these compute functions have analogs in other implementations of Arrow (e.g. Rust)?
I believe that as much as possible of Arrow’s compute functionality should be cross-language. Perhaps there are language-specific differences in how functions are invoked, but the basic functionality is the same.
If people buy into that premise, then a single suite of tests is a powerful way to make that happen. The tests can be written in a high-level language and can generate tests in each implementation language. (For these purposes, the “high-level language” could be a special text format, could be a data language such as JSON, or could be a programming language such as Python; it doesn’t matter much.)
For example,
assertThatCall(“foo(1, 2)”, returns(“3”))
might actually call foo with arguments 1 and 2, or it might generate a C++ or Rust test that does the same.
Julian
> On May 14, 2021, at 8:45 AM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Le 14/05/2021 à 15:30, Wes McKinney a écrit :
>> hi folks,
>> As we build more functions (kernels) in the project, I note that the
>> amount of hand-coded C++ code relating to testing function correctness
>> is growing significantly. Many of these tests are quite simple and
>> could be expressed in a text format that can be parsed and evaluated.
>> Thoughts about building something like that to make it easier to write
>> functional correctness tests?
>
> Or perhaps build-up higher level C++ functions if desired?
>
> Or even write some of those tests as part of the PyArrow test suite.
>
> I'm not sure adding a custom (and probably inflexible) text format is really a good use of our time.
>
> Regards
>
> Antoine.
Re: [C++][DISCUSS] Implementing interpreted (non-compiled) tests for
compute functions
Posted by Antoine Pitrou <an...@python.org>.
Le 14/05/2021 à 15:30, Wes McKinney a écrit :
> hi folks,
>
> As we build more functions (kernels) in the project, I note that the
> amount of hand-coded C++ code relating to testing function correctness
> is growing significantly. Many of these tests are quite simple and
> could be expressed in a text format that can be parsed and evaluated.
>
> Thoughts about building something like that to make it easier to write
> functional correctness tests?
Or perhaps build-up higher level C++ functions if desired?
Or even write some of those tests as part of the PyArrow test suite.
I'm not sure adding a custom (and probably inflexible) text format is
really a good use of our time.
Regards
Antoine.