You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by paddy horan <pa...@hotmail.com> on 2019/08/07 20:29:13 UTC
Options for running the integration tests
Hi All,
I have been away from Arrow for a while due to relocation of family and RSI. I'd like to start working toward getting Rust passing the integration tests. In the last few months a lot of work has been done to "dockerize" many of the build steps in the project, which I'm trying to figure out.
I started out using the 'arrow_integration_xenial_base' image and submitted a PR to allow it to be built from a windows host, but I noticed that there is a page in the pyarrow docs related to integration testing (https://arrow.apache.org/docs/developers/integration.html) that uses docker-compose from the top level of the project. It seems that the 'arrow_integration_xenial_base' image is replaced by this solution?
Is there a way to run the integration tests (integration_test.py) in a reproducible way via docker at this time? If not I plan to add the dependencies for go and java etc to 'arrow_integration_xenial_base' so that I can run integration_test.py in a docker container.
Thanks,
Paddy
Integration Testing — Apache Arrow v0.13.0<https://arrow.apache.org/docs/developers/integration.html>
# Build and run manually docker-compose build cpp docker-compose build python docker-compose run python # Using the makefile with proper image dependency resolution make -f Makefile.docker python
arrow.apache.org
Re: Options for running the integration tests
Posted by paddy horan <pa...@hotmail.com>.
Thanks Krisztián,
I’ll take a look at setting it up.
P
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Krisztián Szűcs <sz...@gmail.com>
Sent: Thursday, August 8, 2019 6:55 PM
To: dev@arrow.apache.org
Subject: Re: Options for running the integration tests
We indeed don't have a docker-compose image for the
"format integration" tests. We can either set it up with
ursabot or the docker-compose, the easiest solution
would be to have the python image as the base image
and install the other language backends, java, rust etc.
then simply run the "format integration" suite.
Because ursabot adoption is under discussion, setting
it up the the docker-compose would require a docker
image like:
```dockerfile
FROM python:3.6
RUN install java
RUN install rust
RUN other backends ...
CMD python arrow/integration/integration_test.py
```
and a corresponding entry in the docker-compose.yml.
On Thu, Aug 8, 2019 at 6:39 PM paddy horan <pa...@hotmail.com> wrote:
> Thanks Antoine,
>
> > Personally I run C++ / Java integration tests locally, without any
> Docker image. But I wouldn't be able to run the other integration tests...
>
> Right this where I started but I figured it's better to use docker as I'm
> not too familiar with other tool chains and the number of languages
> supported is expanding all the time. I'm thinking Krisztian is planning to
> solve this with "ursabot", I just wanted to make sure I wasn't missing
> anything. I'll plug away with the "arrow_integration_xenial_base" image
> for now.
>
> Paddy
>
> ________________________________
> From: Antoine Pitrou <so...@pitrou.net>
> Sent: Thursday, August 8, 2019 10:50 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: Options for running the integration tests
>
> On Wed, 7 Aug 2019 20:29:13 +0000
> paddy horan <pa...@hotmail.com> wrote:
>
> > Hi All,
> >
> > I have been away from Arrow for a while due to relocation of family and
> RSI. I'd like to start working toward getting Rust passing the integration
> tests. In the last few months a lot of work has been done to "dockerize"
> many of the build steps in the project, which I'm trying to figure out.
> >
> > I started out using the 'arrow_integration_xenial_base' image and
> submitted a PR to allow it to be built from a windows host, but I noticed
> that there is a page in the pyarrow docs related to integration testing (
> https://arrow.apache.org/docs/developers/integration.html) that uses
> docker-compose from the top level of the project.
>
> That documentation page may be confusing things. It's entitled
> "integration testing" but it doesn't seem to talk about integration
> tests in the Arrow sense, rather regular unit tests.
>
> > It seems that the 'arrow_integration_xenial_base' image is replaced
> > by this solution?
>
> I have no idea. Perhaps Krisztian knows the answer?
> Personally I run C++ / Java integration tests locally, without any
> Docker image. But I wouldn't be able to run the other integration
> tests...
>
> Regards
>
> Antoine.
>
>
>
Re: Options for running the integration tests
Posted by Krisztián Szűcs <sz...@gmail.com>.
We indeed don't have a docker-compose image for the
"format integration" tests. We can either set it up with
ursabot or the docker-compose, the easiest solution
would be to have the python image as the base image
and install the other language backends, java, rust etc.
then simply run the "format integration" suite.
Because ursabot adoption is under discussion, setting
it up the the docker-compose would require a docker
image like:
```dockerfile
FROM python:3.6
RUN install java
RUN install rust
RUN other backends ...
CMD python arrow/integration/integration_test.py
```
and a corresponding entry in the docker-compose.yml.
On Thu, Aug 8, 2019 at 6:39 PM paddy horan <pa...@hotmail.com> wrote:
> Thanks Antoine,
>
> > Personally I run C++ / Java integration tests locally, without any
> Docker image. But I wouldn't be able to run the other integration tests...
>
> Right this where I started but I figured it's better to use docker as I'm
> not too familiar with other tool chains and the number of languages
> supported is expanding all the time. I'm thinking Krisztian is planning to
> solve this with "ursabot", I just wanted to make sure I wasn't missing
> anything. I'll plug away with the "arrow_integration_xenial_base" image
> for now.
>
> Paddy
>
> ________________________________
> From: Antoine Pitrou <so...@pitrou.net>
> Sent: Thursday, August 8, 2019 10:50 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: Options for running the integration tests
>
> On Wed, 7 Aug 2019 20:29:13 +0000
> paddy horan <pa...@hotmail.com> wrote:
>
> > Hi All,
> >
> > I have been away from Arrow for a while due to relocation of family and
> RSI. I'd like to start working toward getting Rust passing the integration
> tests. In the last few months a lot of work has been done to "dockerize"
> many of the build steps in the project, which I'm trying to figure out.
> >
> > I started out using the 'arrow_integration_xenial_base' image and
> submitted a PR to allow it to be built from a windows host, but I noticed
> that there is a page in the pyarrow docs related to integration testing (
> https://arrow.apache.org/docs/developers/integration.html) that uses
> docker-compose from the top level of the project.
>
> That documentation page may be confusing things. It's entitled
> "integration testing" but it doesn't seem to talk about integration
> tests in the Arrow sense, rather regular unit tests.
>
> > It seems that the 'arrow_integration_xenial_base' image is replaced
> > by this solution?
>
> I have no idea. Perhaps Krisztian knows the answer?
> Personally I run C++ / Java integration tests locally, without any
> Docker image. But I wouldn't be able to run the other integration
> tests...
>
> Regards
>
> Antoine.
>
>
>
Re: Options for running the integration tests
Posted by paddy horan <pa...@hotmail.com>.
Thanks Antoine,
> Personally I run C++ / Java integration tests locally, without any Docker image. But I wouldn't be able to run the other integration tests...
Right this where I started but I figured it's better to use docker as I'm not too familiar with other tool chains and the number of languages supported is expanding all the time. I'm thinking Krisztian is planning to solve this with "ursabot", I just wanted to make sure I wasn't missing anything. I'll plug away with the "arrow_integration_xenial_base" image for now.
Paddy
________________________________
From: Antoine Pitrou <so...@pitrou.net>
Sent: Thursday, August 8, 2019 10:50 AM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Subject: Re: Options for running the integration tests
On Wed, 7 Aug 2019 20:29:13 +0000
paddy horan <pa...@hotmail.com> wrote:
> Hi All,
>
> I have been away from Arrow for a while due to relocation of family and RSI. I'd like to start working toward getting Rust passing the integration tests. In the last few months a lot of work has been done to "dockerize" many of the build steps in the project, which I'm trying to figure out.
>
> I started out using the 'arrow_integration_xenial_base' image and submitted a PR to allow it to be built from a windows host, but I noticed that there is a page in the pyarrow docs related to integration testing (https://arrow.apache.org/docs/developers/integration.html) that uses docker-compose from the top level of the project.
That documentation page may be confusing things. It's entitled
"integration testing" but it doesn't seem to talk about integration
tests in the Arrow sense, rather regular unit tests.
> It seems that the 'arrow_integration_xenial_base' image is replaced
> by this solution?
I have no idea. Perhaps Krisztian knows the answer?
Personally I run C++ / Java integration tests locally, without any
Docker image. But I wouldn't be able to run the other integration
tests...
Regards
Antoine.
Re: Options for running the integration tests
Posted by Antoine Pitrou <so...@pitrou.net>.
On Wed, 7 Aug 2019 20:29:13 +0000
paddy horan <pa...@hotmail.com> wrote:
> Hi All,
>
> I have been away from Arrow for a while due to relocation of family and RSI. I'd like to start working toward getting Rust passing the integration tests. In the last few months a lot of work has been done to "dockerize" many of the build steps in the project, which I'm trying to figure out.
>
> I started out using the 'arrow_integration_xenial_base' image and submitted a PR to allow it to be built from a windows host, but I noticed that there is a page in the pyarrow docs related to integration testing (https://arrow.apache.org/docs/developers/integration.html) that uses docker-compose from the top level of the project.
That documentation page may be confusing things. It's entitled
"integration testing" but it doesn't seem to talk about integration
tests in the Arrow sense, rather regular unit tests.
> It seems that the 'arrow_integration_xenial_base' image is replaced
> by this solution?
I have no idea. Perhaps Krisztian knows the answer?
Personally I run C++ / Java integration tests locally, without any
Docker image. But I wouldn't be able to run the other integration
tests...
Regards
Antoine.