You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Yang Xia <ya...@bitquilltech.com.INVALID> on 2022/05/27 18:36:36 UTC
Re: use of docker-compose in gremlin-go
Hi all,
To follow up on this thread and in light of using docker-compose for
gremlin go testing, I’d like to propose a plan to set up docker-compose for
other non-java GLVs, such that we can use docker-compose with maven for
testing instead of groovy scripts and .glv configurations.
Tasks breakdown:
1. Setup test server base image with no authentication, basic
authentication, and kerberos authentication
2. Add docker-compose test setup for GLVs: gremlin-dotnet, gremlin-go,
gremlin-javascript, gremlin-python, plus gremlin-console (relies on
gremlin-python)
3. Integrate docker-compose into the current mvn workflow and the
current docker system
- Avoid docker-in-docker by extracting the testing of GLVs from the
build set up into a separate docker task
There are a couple of assumptions:
- All subprojects can be built individually using mvn install on the
entire project or the particular subproject.
- Subprojects which require gremlin-server for testing will:
- Use mvn dependencies to build the base gremlin-server docker image,
and docker-compose to build the test server image.
- Use docker-compose to run the test server instead of groovy scripts.
- Can be tested individually using docker-compose.
- All scripts and files used for this Docker testing system will be in
one centralized place for ease of maintenance.
Please let me know if you have any comments, concerns, or suggestions.
Thank you!
Yang
*--*
*Yang Xia*
Software Engineer
Bit Quill Technologies Inc.
yangx@bitquilltech.com
https://www.bitquill.com
This email message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure, or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message. Thank you.
On Tue, Apr 5, 2022 at 9:06 AM Lyndon Bauto
<ly...@bitquilltech.com.invalid> wrote:
> This doesn't actually need to be run in the CI/CD, it was added just as a
> check that the godog translator was working properly and wasn't broken
> between pull requests. I thought that was the minimum command subset to
> generate it, but was wrong on that. The files it generates are not used for
> the godog tests, these have been kept up to date manually ahead of time as
> you suggested.
>
> We are still looking into better automation around the godog tests which
> will hopefully resolve this more completely.
>
> On Tue, Apr 5, 2022 at 8:27 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > gremlin-go is failing in GitHub Actions for my newly tagged 3.5.3. The
> > problem seems obvious (i think), we're not doing a full build of the repo
> > and therefore relying on external builds of whatever version we're
> > currently trying to test:
> >
> >
> >
> https://github.com/apache/tinkerpop/blob/3.5.3/.github/workflows/build-test.yml#L239-L241
> >
> > In this situation, 3.5.3 does not yet exist in any public maven repo so
> > that part of the build fails. I think we should be doing a:
> >
> > mvn clean install -pl
> >
> >
> -:gremlin-javascript,-:gremlin-python,-gremlin-dotnet,-:gremlin-dotnet-source,-:gremlin-dotnet-tests
> > -q -DskipTests -Dci
> >
> > there rather than those three separate commands. I'm also curious why
> that
> > step should be last? shouldn't that happen ahead of running the godog
> tests
> > so that the test files get updated properly if new tests are added to the
> > .feature files?
> >
> >
> >
> > On Mon, Apr 4, 2022 at 2:44 PM Lyndon Bauto
> > <ly...@bitquilltech.com.invalid> wrote:
> >
> > > Thanks for exposing this issue Stephen. Myself or one of the other
> people
> > > who has contributed to gremlin-go will take this item on and make sure
> we
> > > get that working as expected. Agreed on not being a release blocker.
> > >
> > > On Mon, Apr 4, 2022 at 3:33 AM Stephen Mallette <sp...@gmail.com>
> > > wrote:
> > >
> > > > Use of docker-compose seems to have some problems inside of our
> > standard
> > > > docker build, i.e. docker/build.sh doesn't work and fails the whole
> > > build.
> > > > I've disabled the gremlin-go tests until this can be resolved. Not a
> > > > blocker for release imo.
> > > >
> > > > On Fri, Mar 18, 2022 at 4:22 PM Lyndon Bauto
> > > > <ly...@bitquilltech.com.invalid> wrote:
> > > >
> > > > > I think it would be great to use Docker to remove any environment
> > setup
> > > > > requirements, allowing new developers to more easily start with
> > > > TinkerPop,
> > > > > and current developers to test and use a new GLV without worrying
> > about
> > > > > setting up their system to run it.
> > > > >
> > > > > To answer your questions more specifically though:
> > > > >
> > > > > 1. Yes it does reference 3.5 - currently it loads an empty
> graph,
> > > > > however an item we are looking to approach soon will be to load
> > the
> > > > > docker
> > > > > graph with the datasets as well as the authentication built in.
> > > > > 2. Yes, it would be possible for us to have it set up such that
> a
> > > > > developer could build the docker server locally, reference it,
> > then
> > > > > execute
> > > > > tests against it.
> > > > > 3. Yes, that’s the idea. Anyone should be able to run the
> > Gremlin-Go
> > > > > integration tests locally without even installing Go.
> > > > > 1. I think that removing the complicated Maven building for
> the
> > > > > non-Java GLV’s and moving to a docker-compose style build
> > > structure
> > > > > would
> > > > > provide a much simpler build environment for new developers.
> > > > > 2. I’d like to discuss the idea of doing this, and in fact
> was
> > > > > considering starting it after Gremlin-Go is complete.
> > > > >
> > > > > I'd appreciate any feedback from the community on this, for or
> > against.
> > > > >
> > > > > On Fri, Mar 18, 2022 at 9:11 AM Stephen Mallette <
> > spmallette@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Lyndon (or anyone who might have thoughts) - I like the pattern
> > > started
> > > > > for
> > > > > > making better use of docker to solve TinkerPop testing issues. We
> > > need
> > > > > more
> > > > > > smart things to happen with docker to keep our code easy to
> build,
> > > test
> > > > > and
> > > > > > manage. Now that i'm digging into it, I have developed some
> > > > > > thoughts/questions.
> > > > > >
> > > > > > 1. I see the docker-compose.yml file and that it references the
> > > Gremlin
> > > > > > Server image of: tinkerpop/gremlin-server:3.5 but doesn't that
> just
> > > > > start a
> > > > > > Gremlin Server on 8182 which would start it with an empty graph
> by
> > > > > default?
> > > > > > where does the test data for all the different graphs required by
> > > > Gherkin
> > > > > > get loaded?
> > > > > >
> > > > > > 2. Is there a way to get docker-compose to use the local SNAPSHOT
> > of
> > > > > > Gremlin Server rather than pinning to
> tinkerpop/gremlin-server:3.5?
> > > > > >
> > > > > > 3. I think that this model gets rid of the complicated
> > configuration
> > > of
> > > > > the
> > > > > > .glv file which "turns on" the Maven <profile> for GLV building.
> We
> > > can
> > > > > > simply require Maven+docker-compose to do a build and streamline
> > all
> > > of
> > > > > > that sort of stuff. Does that sound right?
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Lyndon Bauto*
> > > > > Team Lead
> > > > > Bit Quill Technologies Inc.
> > > > > lyndonb@bitquilltech.com
> > > > > https://www.bitquilltech.com
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Lyndon Bauto*
> > > Team Lead
> > > Bit Quill Technologies Inc.
> > > lyndonb@bitquilltech.com
> > > https://www.bitquilltech.com
> > >
> >
>
>
> --
> *Lyndon Bauto*
> Team Lead
> Bit Quill Technologies Inc.
> lyndonb@bitquilltech.com
> https://www.bitquilltech.com
>
Re: use of docker-compose in gremlin-go
Posted by Yang Xia <ya...@bitquilltech.com.INVALID>.
Hello all,
A quick update on the plan to set up docker-compose for GLVs. The task
breakdown remains the same as previously described, where we look to
integrate docker-compose into the testing of gremlin-dotnet, gremlin-go,
gremlin-javascript, gremlin-python, plus gremlin-console modules.
There are a few added assumptions:
- Docker compose will be integrated into the testing workflow for
3.5-dev and 3.6-dev.
- Docker will be a system requirement for building and running
integration tests in maven for the modules above.
- The .glv profile activation for building and testing will be removed
in 3.7-dev (breaking).
- Docker set up will be platform agnostic, such that mvn clean install
should continue to work in Windows environment.
- Images on GitHub Actions will be cached to use across GLVs to avoid
building the image each step.
- Building on Windows runners in GitHub Actions may not be supported due
to GHA limitations in running linux containers.
We are looking to get Docker integrated for the modules above by the end of
August, and each GLVs set up will be PRed for review as they are completed.
Please let me know if you have any concerns or questions. Thank you!
Yang
*--*
*Yang Xia*
Software Engineer
Bit Quill Technologies Inc.
yangx@bitquilltech.com
https://www.bitquill.com
This email message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure, or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy
all copies of the original message. Thank you.
On Fri, May 27, 2022 at 11:36 AM Yang Xia <ya...@bitquilltech.com> wrote:
> Hi all,
>
> To follow up on this thread and in light of using docker-compose for
> gremlin go testing, I’d like to propose a plan to set up docker-compose for
> other non-java GLVs, such that we can use docker-compose with maven for
> testing instead of groovy scripts and .glv configurations.
>
> Tasks breakdown:
>
> 1. Setup test server base image with no authentication, basic
> authentication, and kerberos authentication
> 2. Add docker-compose test setup for GLVs: gremlin-dotnet, gremlin-go,
> gremlin-javascript, gremlin-python, plus gremlin-console (relies on
> gremlin-python)
> 3. Integrate docker-compose into the current mvn workflow and the
> current docker system
> - Avoid docker-in-docker by extracting the testing of GLVs from the
> build set up into a separate docker task
>
> There are a couple of assumptions:
>
> - All subprojects can be built individually using mvn install on the
> entire project or the particular subproject.
> - Subprojects which require gremlin-server for testing will:
> - Use mvn dependencies to build the base gremlin-server docker
> image, and docker-compose to build the test server image.
> - Use docker-compose to run the test server instead of groovy
> scripts.
> - Can be tested individually using docker-compose.
> - All scripts and files used for this Docker testing system will be in
> one centralized place for ease of maintenance.
>
> Please let me know if you have any comments, concerns, or suggestions.
> Thank you!
>
> Yang
> *--*
> *Yang Xia*
> Software Engineer
> Bit Quill Technologies Inc.
> yangx@bitquilltech.com
> https://www.bitquill.com
>
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential and privileged information. Any unauthorized
> review, use, disclosure, or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply email and destroy
> all copies of the original message. Thank you.
>
>
> On Tue, Apr 5, 2022 at 9:06 AM Lyndon Bauto
> <ly...@bitquilltech.com.invalid> wrote:
>
>> This doesn't actually need to be run in the CI/CD, it was added just as a
>> check that the godog translator was working properly and wasn't broken
>> between pull requests. I thought that was the minimum command subset to
>> generate it, but was wrong on that. The files it generates are not used
>> for
>> the godog tests, these have been kept up to date manually ahead of time as
>> you suggested.
>>
>> We are still looking into better automation around the godog tests which
>> will hopefully resolve this more completely.
>>
>> On Tue, Apr 5, 2022 at 8:27 AM Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>> > gremlin-go is failing in GitHub Actions for my newly tagged 3.5.3. The
>> > problem seems obvious (i think), we're not doing a full build of the
>> repo
>> > and therefore relying on external builds of whatever version we're
>> > currently trying to test:
>> >
>> >
>> >
>> https://github.com/apache/tinkerpop/blob/3.5.3/.github/workflows/build-test.yml#L239-L241
>> >
>> > In this situation, 3.5.3 does not yet exist in any public maven repo so
>> > that part of the build fails. I think we should be doing a:
>> >
>> > mvn clean install -pl
>> >
>> >
>> -:gremlin-javascript,-:gremlin-python,-gremlin-dotnet,-:gremlin-dotnet-source,-:gremlin-dotnet-tests
>> > -q -DskipTests -Dci
>> >
>> > there rather than those three separate commands. I'm also curious why
>> that
>> > step should be last? shouldn't that happen ahead of running the godog
>> tests
>> > so that the test files get updated properly if new tests are added to
>> the
>> > .feature files?
>> >
>> >
>> >
>> > On Mon, Apr 4, 2022 at 2:44 PM Lyndon Bauto
>> > <ly...@bitquilltech.com.invalid> wrote:
>> >
>> > > Thanks for exposing this issue Stephen. Myself or one of the other
>> people
>> > > who has contributed to gremlin-go will take this item on and make
>> sure we
>> > > get that working as expected. Agreed on not being a release blocker.
>> > >
>> > > On Mon, Apr 4, 2022 at 3:33 AM Stephen Mallette <spmallette@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Use of docker-compose seems to have some problems inside of our
>> > standard
>> > > > docker build, i.e. docker/build.sh doesn't work and fails the whole
>> > > build.
>> > > > I've disabled the gremlin-go tests until this can be resolved. Not a
>> > > > blocker for release imo.
>> > > >
>> > > > On Fri, Mar 18, 2022 at 4:22 PM Lyndon Bauto
>> > > > <ly...@bitquilltech.com.invalid> wrote:
>> > > >
>> > > > > I think it would be great to use Docker to remove any environment
>> > setup
>> > > > > requirements, allowing new developers to more easily start with
>> > > > TinkerPop,
>> > > > > and current developers to test and use a new GLV without worrying
>> > about
>> > > > > setting up their system to run it.
>> > > > >
>> > > > > To answer your questions more specifically though:
>> > > > >
>> > > > > 1. Yes it does reference 3.5 - currently it loads an empty
>> graph,
>> > > > > however an item we are looking to approach soon will be to load
>> > the
>> > > > > docker
>> > > > > graph with the datasets as well as the authentication built in.
>> > > > > 2. Yes, it would be possible for us to have it set up such
>> that a
>> > > > > developer could build the docker server locally, reference it,
>> > then
>> > > > > execute
>> > > > > tests against it.
>> > > > > 3. Yes, that’s the idea. Anyone should be able to run the
>> > Gremlin-Go
>> > > > > integration tests locally without even installing Go.
>> > > > > 1. I think that removing the complicated Maven building for
>> the
>> > > > > non-Java GLV’s and moving to a docker-compose style build
>> > > structure
>> > > > > would
>> > > > > provide a much simpler build environment for new developers.
>> > > > > 2. I’d like to discuss the idea of doing this, and in fact
>> was
>> > > > > considering starting it after Gremlin-Go is complete.
>> > > > >
>> > > > > I'd appreciate any feedback from the community on this, for or
>> > against.
>> > > > >
>> > > > > On Fri, Mar 18, 2022 at 9:11 AM Stephen Mallette <
>> > spmallette@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Lyndon (or anyone who might have thoughts) - I like the pattern
>> > > started
>> > > > > for
>> > > > > > making better use of docker to solve TinkerPop testing issues.
>> We
>> > > need
>> > > > > more
>> > > > > > smart things to happen with docker to keep our code easy to
>> build,
>> > > test
>> > > > > and
>> > > > > > manage. Now that i'm digging into it, I have developed some
>> > > > > > thoughts/questions.
>> > > > > >
>> > > > > > 1. I see the docker-compose.yml file and that it references the
>> > > Gremlin
>> > > > > > Server image of: tinkerpop/gremlin-server:3.5 but doesn't that
>> just
>> > > > > start a
>> > > > > > Gremlin Server on 8182 which would start it with an empty graph
>> by
>> > > > > default?
>> > > > > > where does the test data for all the different graphs required
>> by
>> > > > Gherkin
>> > > > > > get loaded?
>> > > > > >
>> > > > > > 2. Is there a way to get docker-compose to use the local
>> SNAPSHOT
>> > of
>> > > > > > Gremlin Server rather than pinning to
>> tinkerpop/gremlin-server:3.5?
>> > > > > >
>> > > > > > 3. I think that this model gets rid of the complicated
>> > configuration
>> > > of
>> > > > > the
>> > > > > > .glv file which "turns on" the Maven <profile> for GLV
>> building. We
>> > > can
>> > > > > > simply require Maven+docker-compose to do a build and streamline
>> > all
>> > > of
>> > > > > > that sort of stuff. Does that sound right?
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > *Lyndon Bauto*
>> > > > > Team Lead
>> > > > > Bit Quill Technologies Inc.
>> > > > > lyndonb@bitquilltech.com
>> > > > > https://www.bitquilltech.com
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > *Lyndon Bauto*
>> > > Team Lead
>> > > Bit Quill Technologies Inc.
>> > > lyndonb@bitquilltech.com
>> > > https://www.bitquilltech.com
>> > >
>> >
>>
>>
>> --
>> *Lyndon Bauto*
>> Team Lead
>> Bit Quill Technologies Inc.
>> lyndonb@bitquilltech.com
>> https://www.bitquilltech.com
>>
>