docker and GLV development

[Stephen Mallette <sp...@gmail.com>]

Anyone have any idea how to make it so that docker can be useful for
helping to streamline GLV development? I'd love to be able to do something
like:

docker/gremlin-server.sh -test

which would start Gremlin Server with our standard test configuration
(currently built into the maven tool chain, but I think that could be
extracted without too much difficulty). That way, we could easily open up
whatever GLV specific IDE we wanted and run tests directly in there, use
the debugger, etc.

Sound useful? and if so, is that something that is easy/possible? and then
further....any volunteers with the know-how to get that setup (i'm happy to
assist however i can)?

btw, if you have a nicer way to set up an environment for GLV development
and docker is a waste of time, please clue me in. i don't mind using the
maven integration but i feel like i could be more efficient with a more
streamlined setup.

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

Forgot to mention that the gremlin-python version is hardcoded.

Robert Dale


On Wed, Mar 6, 2019 at 2:36 PM Robert Dale <ro...@gmail.com> wrote:

> I just pushed a change that allows python tests to pass against the docker
> image.
>
> This will install the gremlin-python script support on the server so
> that's an extra download to wait for.
>
> I had to update the localhost references to point to the docker IP
> in gremlin-python:   grep -Rl localhost gremlin-python/src/main/jython/ |
> xargs sed -i 's/localhost/172.17.0.2/g'
>
> Then I could `mvn clean install -pl :gremlin-python`
>
> Robert Dale
>
>
> On Wed, Mar 6, 2019 at 9:22 AM Robert Dale <ro...@gmail.com> wrote:
>
>> Florian makes a great point about using the image from gremlin-server.
>> It's much lighter weight.  It also makes it easier for non-java,
>> client-only devs to use since it doesn't require maven if a published
>> version is used. I've pushed my changes.
>>
>> Thus,
>> # Will use the published gremlin-server image with the 'latest' tag.
>> ./docker/gremlin-server.sh
>>
>> # Will use the published gremlin-server image with the specified version
>> ./docker/gremlin-server.sh 3.4.0
>>
>> # To use the working copy, build gremlin-server and the image. e.g.:
>> mvn clean install -pl :gremlin-server -am && mvn -DskipTests install
>> -Pdocker-images -pl :gremlin-server
>>
>> # You still have to know what the working copy version is. Docker will
>> check local images.
>> ./docker/gremlin-server.sh 3.4.1-SNAPSHOT
>>
>> In all cases, the test scripts are still copied from the working copy so
>> be aware of any version conflicts between gremlin-server version used and
>> working copy version test scripts.
>>
>> I've also retained the original image's ability to override the yaml
>> file. However, the version is required even when using 'latest' tag.
>>
>> # Will start latest published gremlin-server with the default config on
>> port 8182
>> ./docker/gremlin-server.sh latest conf/gremlin-server.yaml
>>
>>
>> Robert Dale
>>
>>
>> On Fri, Mar 1, 2019 at 3:13 PM Florian Hockmann <fh...@florian-hockmann.de>
>> wrote:
>>
>>> Wow, for some reason I didn't see your earlier mail and also the linked
>>> PR. So, I didn't know that there is already some automation to try out.
>>>
>>> That made my attempts to help of course a bit pointless :-/ Sorry
>>> Stephen if I waisted your time with that.
>>>
>>> Regarding our initial discussion here: I think that Testcontainers would
>>> be the best solution for the GLVs (gremlin-driver was probably a bad
>>> example), at least in the long term, as it would also enable us to test
>>> modification steps without breaking other tests. Nevertheless, I agree
>>> that a simpler approach might be the better solution for now and it is
>>> definitely an improvement over our current situation.
>>>
>>> I will probably wait for testcontainers-dotnet to support using images
>>> that are only available locally (so we can test with a locally built
>>> Gremlin Server) and then test out how good it works with our build
>>> process.
>>>
>>> Am 01.03.2019 um 20:50 schrieb Robert Dale:
>>> > I force pushed a fix.  Wasn't listening on all nets. Also added a
>>> banner
>>> > for IP address that will appear like so:
>>> > ...
>>> > Successfully built b00dc1f2db35
>>> > #######################
>>> > IP is 172.17.0.2
>>> > #######################
>>> > [INFO] GremlinServer - 3.4.1-SNAPSHOT
>>> > ...
>>> >
>>> > Robert Dale
>>> >
>>> >
>>> > On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <
>>> fh@florian-hockmann.de>
>>> > wrote:
>>> >
>>> >> I don't know what the problem could be, but two things I can think of
>>> >> that could be helpful to find the problem:
>>> >>
>>> >>   * Connect to the container (via docker exec -it [container_id] bash)
>>> >>     and then confirm with netstat (which you have to install,  `apk
>>> >>     update; apk add ospd-netstat` should work) that port 45940 is
>>> >>     actually open and that the foreign address is 0.0.0.0 (`netstat
>>> -tlpn`)
>>> >>   * Check whether you can connect to that port in general without
>>> adding
>>> >>     complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>>> >>     localhost 45940`.
>>> >>
>>> >> Simply restarting Docker could also help. (At least on Windows some
>>> >> strange Docker networks problems are often gone after a restart.)
>>> >>
>>> >> You're probably using Gremlin Console from your host system and not
>>> with
>>> >> our Docker image, right?
>>> >>
>>> >> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
>>> >>> i tried adding that -p thing to the run command in the script, but i
>>> >> still
>>> >>> couldn't connect to it with Gremlin Console
>>> >>>
>>> >>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <
>>> fh@florian-hockmann.de>
>>> >>> wrote:
>>> >>>
>>> >>>> Did you start the container with port mapping (e.g., docker run -p
>>> >>>> 45940:45940 [image-name]) or how are you trying to connect to the
>>> >>>> container?
>>> >>>>
>>> >>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>>> >>>>> The server starts but something seems messed up with the ports.
>>> It's
>>> >>>> like I
>>> >>>>> can't connect to the 45940 port from the host. I tried a couple
>>> things
>>> >> I
>>> >>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's
>>> >> probably
>>> >>>>> something simple...any ideas?
>>> >>>>>
>>> >>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com>
>>> wrote:
>>> >>>>>
>>> >>>>>> I made a first attempt.  Let me know what you think.
>>> >>>>>> https://github.com/apache/tinkerpop/pull/1075
>>> >>>>>>
>>> >>>>>> Robert Dale
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
>>> >> spmallette@gmail.com
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>>>  If we agree on Testcontainers in general, then I can give this
>>> a
>>> >> try
>>> >>>>>> and
>>> >>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>>> >> Server
>>> >>>>>> for
>>> >>>>>>> testing (gremlin-driver maybe?).
>>> >>>>>>>
>>> >>>>>>> The java integration tests for Gremlin Server flip a lot of
>>> >>>> configuration
>>> >>>>>>> bits per test. I don't know how that works with testcontainers
>>> and if
>>> >>>> it
>>> >>>>>>> will make our testing there any better. There are also a number
>>> of
>>> >>>> tests
>>> >>>>>>> that stop/start the server under different conditions - not sure
>>> how
>>> >>>> that
>>> >>>>>>> works with testcontainers either. Finally, wouldn't we expect
>>> adding
>>> >>>>>> docker
>>> >>>>>>> to the mix to slow down our integration test runs for java?
>>> >>>>>>>
>>> >>>>>>> I don't want to seem like i'm completely against testcontainers
>>> btw.
>>> >> I
>>> >>>>>> just
>>> >>>>>>> wonder if a script doesn't give us a 100% better experience than
>>> what
>>> >>>> we
>>> >>>>>>> have now (which is nothing). If so, and the script is easy and
>>> >>>>>> zero-impact
>>> >>>>>>> without new dependencies, then I think I like that approach.
>>> Also, It
>>> >>>>>> would
>>> >>>>>>> be nice to have the script for other sorts of manual testing,
>>> such as
>>> >>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>> >>>>>>> testcontainers for GLVs I suppose that can be done too, but for
>>> my
>>> >>>>>>> purposes, since I was the one who asked for help with this
>>> initially,
>>> >>>>>>> testcontainers doesn't sound like something that is an absolute
>>> >>>>>> requirement
>>> >>>>>>> for my needs.
>>> >>>>>>>
>>> >>>>>>> thanks robert/florian for all the discussion and help -
>>> appreciated.
>>> >>>>>>>
>>> >>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>>> >>>>>> fh@florian-hockmann.de>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> Yes, except for the detail that Testcontainers could make the
>>> script
>>> >>>>>>>> obsolete if it is also used to build the image by itself. That
>>> would
>>> >>>>>> have
>>> >>>>>>>> the advantage that developers wouldn't have to execute the
>>> script to
>>> >>>>>>> build
>>> >>>>>>>> the image. (Executing the script with Testcontainers could also
>>> be a
>>> >>>>>>>> possibility, but I'm not sure how good that works.)
>>> >>>>>>>>
>>> >>>>>>>> If we agree on Testcontainers in general, then I can give this
>>> a try
>>> >>>>>> and
>>> >>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>>> >>>> Server
>>> >>>>>>> for
>>> >>>>>>>> testing (gremlin-driver maybe?).
>>> >>>>>>>> With such a PR it would probably also be easier for others to
>>> see
>>> >> what
>>> >>>>>>>> advantages / disadvantages using Testcontainers would have.
>>> >>>>>>>>
>>> >>>>>>>> I think using a Java project first makes most sense as we can
>>> use
>>> >> all
>>> >>>>>>>> functionality Testcontainers has to offer there. The downside
>>> is of
>>> >>>>>>> course
>>> >>>>>>>> that we might not have all the functionality afterwards for
>>> other
>>> >>>>>>> languages
>>> >>>>>>>> and then we might need to extend the Testcontainers libraries
>>> for
>>> >> the
>>> >>>>>>>> different languages. (My assumption here is that we won't need
>>> that
>>> >>>>>> much
>>> >>>>>>>> functionality so it should be limited to 1 or 2 missing
>>> features.)
>>> >>>>>>>> Alternatively, we could also start with a GLV where the
>>> >> Testcontainers
>>> >>>>>>>> library doesn't have the same functionality which probably means
>>> >> that
>>> >>>>>> we
>>> >>>>>>>> need a workaround to get an image of our current Gremlin Server
>>> >>>> version
>>> >>>>>>> for
>>> >>>>>>>> testing.
>>> >>>>>>>>
>>> >>>>>>>> -----Ursprüngliche Nachricht-----
>>> >>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>> >>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>> >>>>>>>> An: dev@tinkerpop.apache.org
>>> >>>>>>>> Betreff: Re: docker and GLV development
>>> >>>>>>>>
>>> >>>>>>>> I think there are two components here.
>>> >>>>>>>> 1) Script to build the dockerized test server. Includes ability
>>> to
>>> >>>>>> start
>>> >>>>>>>> container manually, externally from tests.
>>> >>>>>>>> 2) Testcontainers integration which uses the docker image from
>>> #1.
>>> >>>>>>>> Testcontainers could also include executing the script in #1 to
>>> >> build
>>> >>>>>> the
>>> >>>>>>>> container.
>>> >>>>>>>>
>>> >>>>>>>> Does this sound right to you?
>>> >>>>>>>>
>>> >>>>>>>> Robert Dale
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>>> >>>>>> fh@florian-hockmann.de
>>> >>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>>> I do like the idea of being able to test against older
>>> versions
>>> >>>>>>> though.
>>> >>>>>>>>> That shouldn't be a problem in general as we could use multiple
>>> >>>>>>>>> different images. See for example this PR that adds
>>> Testcontainers
>>> >> to
>>> >>>>>>>>> JanusGraph to test different Elasticsearch versions:
>>> >>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>> >>>>>>>>>
>>> >>>>>>>>>> Yes, an on-the-fly container would work
>>> >>>>>>>>> We could however also use an on-the-fly container with
>>> >>>>>> Testcontainers.
>>> >>>>>>>>> We would just need some automation to build the image in the
>>> >>>>>>> background.
>>> >>>>>>>>> What if we would build an image during the usual Maven build
>>> for
>>> >>>>>>>>> testing and give it a descriptive tag like "testing". Then, we
>>> >> could
>>> >>>>>>>>> use that tag with Testcontainers in addition to tags for
>>> already
>>> >>>>>>>> released versions.
>>> >>>>>>>>> Testcontainers-Java can apparently even build images by
>>> itself. So,
>>> >>>>>>>>> that could also be a possibility to get an image with the
>>> latest
>>> >>>>>>>>> Gremlin Server with the current (unreleased) changes.
>>> >>>>>>>>>
>>> >>>>>>>>> In general, I would really like to get away from a situation
>>> where
>>> >> we
>>> >>>>>>>>> have to start some script in the background before tests can be
>>> >>>>>>> executed.
>>> >>>>>>>>> @Robert:
>>> >>>>>>>>>> What is the test configuration/data for GLVs?  Could you list
>>> the
>>> >>>>>>>>> paths/files?
>>> >>>>>>>>>
>>> >>>>>>>>> For Gremlin.Net for example, you can find the automation
>>> happening
>>> >> in
>>> >>>>>>>>> the pom.xml of the testing part:
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>
>>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>> >>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>> >>>>>>>>>
>>> >>>>>>>>> It calls the test-server-start.groovy script before the tests
>>> are
>>> >>>>>>>>> executed and afterwards test-server-stop.groovy script to stop
>>> the
>>> >>>>>>>> server again.
>>> >>>>>>>>> This has the obvious downside that the tests can only be
>>> executed
>>> >>>>>> from
>>> >>>>>>>>> the Maven build which isn't the build tool .NET developers
>>> would
>>> >> use
>>> >>>>>>>>> and also not the tool the IDE will use (the situation is
>>> basically
>>> >>>>>> the
>>> >>>>>>>>> same for other GLVs).
>>> >>>>>>>>>
>>> >>>>>>>>> -----Ursprüngliche Nachricht-----
>>> >>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>> >>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>> >>>>>>>>> An: dev@tinkerpop.apache.org
>>> >>>>>>>>> Betreff: Re: docker and GLV development
>>> >>>>>>>>>
>>> >>>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>> >>>>>>>>>
>>> >>>>>>>>> What is the test configuration/data for GLVs?  Could you list
>>> the
>>> >>>>>>>>> paths/files?
>>> >>>>>>>>>
>>> >>>>>>>>> Robert Dale
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>> >>>>>>>>> <sp...@gmail.com>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Yes, an on-the-fly container would work assuming it used the
>>> test
>>> >>>>>>>>>> configuration/data for GLVs. That's why i suggested the
>>> command as
>>> >>>>>> I
>>> >>>>>>>>>> did because it was similar to docker/build.sh.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <
>>> robdale@gmail.com>
>>> >>>>>>> wrote:
>>> >>>>>>>>>>> Do you want to create a docker container on-the-fly using the
>>> >>>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Robert Dale
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>> >>>>>>>>>>> <sp...@gmail.com>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> testcontainers seems neat as it gives some good integration
>>> and
>>> >>>>>>>>>>> automation,
>>> >>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so
>>> great.
>>> >>>>>>>>>>>> There are times when changes are occurring on both client
>>> and
>>> >>>>>>>>>>>> server so a local up-to-the-moment image would be
>>> necessary. I
>>> >>>>>>>>>>>> also think we might want
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures
>>> out
>>> >>>>>>>>>>>> more
>>> >>>>>>>>>>> clearly
>>> >>>>>>>>>>>> how those play into projects (there is a bit of discussion
>>> >>>>>> going
>>> >>>>>>>>>>>> on
>>> >>>>>>>>>> with
>>> >>>>>>>>>>>> that now). I do like the idea of being able to test against
>>> >>>>>>>>>>>> older
>>> >>>>>>>>>>> versions
>>> >>>>>>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>> >>>>>>>>>> fh@florian-hockmann.de
>>> >>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>>> >>>>>> tests
>>> >>>>>>>>>>>>> for Gremlin.Net from my IDE.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>> >>>>>>>>>>>>> problems
>>> >>>>>>>>>>> like
>>> >>>>>>>>>>>>> this:
>>> >>>>>>>>>>>>> https://www.testcontainers.org/
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> The basic idea of Testcontainers is that you define a
>>> Docker
>>> >>>>>>>>>> container
>>> >>>>>>>>>>> in
>>> >>>>>>>>>>>>> your tests for whatever external dependencies you need.
>>> >>>>>>>>>>>>> Databases
>>> >>>>>>>>>> are a
>>> >>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>> >>>>>>>>>>>>> Gremlin
>>> >>>>>>>>>>> Server
>>> >>>>>>>>>>>>> would be such a container.
>>> >>>>>>>>>>>>> Testcontainers then uses the ability of the testing
>>> framework
>>> >>>>>>>> (e.g.
>>> >>>>>>>>>>>> JUnit)
>>> >>>>>>>>>>>>> to start the Docker container for the tests and waits until
>>> >>>>>>>>>>>>> they are
>>> >>>>>>>>>>>> fully
>>> >>>>>>>>>>>>> started (how they are started can be configured for each
>>> >>>>>>>>>>>>> container differently). After the tests have completed, it
>>> >>>>>>>>>>>>> also stops the
>>> >>>>>>>>>>>> containers
>>> >>>>>>>>>>>>> again.
>>> >>>>>>>>>>>>> That way, you don't have to manage the containers manually
>>> >>>>>>>> anymore.
>>> >>>>>>>>>>>>> Everything is done in the background by the IDE / the build
>>> >>>>>>> tool.
>>> >>>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>>> >>>>>>>>>>>>> also have libraries for different languages, including the
>>> >>>>>>>>>>>>> ones for which we
>>> >>>>>>>>>> have
>>> >>>>>>>>>>>>> GLVs:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> testcontainers-python:
>>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>> >>>>>>>>>>>>> testcontainers-dotnet
>>> >>>>>>>>>>>>> <
>>> >>>>>>
>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>> >>>>>>>>>> s-
>>> >>>>>>>>>> dotnet
>>> >>>>>>>>>>>>> :
>>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>> >>>>>>>>>>>>> testcontainers-node
>>> >>>>>>>>>>>>> <
>>> >>>>>>
>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>> >>>>>>>>>> s-
>>> >>>>>>>>>> node
>>> >>>>>>>>>>>>> :
>>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Testcontainers-java is the most advanced library of these
>>> of
>>> >>>>>>>>> course.
>>> >>>>>>>>>>> The
>>> >>>>>>>>>>>>> others still lack some functionality, but I guess that we
>>> >>>>>>>>>>>>> could also
>>> >>>>>>>>>>> just
>>> >>>>>>>>>>>>> contribute to those libraries if an important feature is
>>> >>>>>>>>>>>>> missing for
>>> >>>>>>>>>>> us.
>>> >>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>> >>>>>>>>>>> testcontainers-dotnet
>>> >>>>>>>>>>>>> unfortunately can only use images that are available in
>>> some
>>> >>>>>>>>>> registry.
>>> >>>>>>>>>>>> So,
>>> >>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>> >>>>>>>>>>>>> DockerHub
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>> use
>>> >>>>>>>>>>>>> it in our tests. That means that we could either only test
>>> >>>>>> the
>>> >>>>>>>>>>>>> GLVs
>>> >>>>>>>>>>>> against
>>> >>>>>>>>>>>>> already released versions or we could publish something
>>> like
>>> >>>>>>>>>>>>> nightly
>>> >>>>>>>>>>>> build
>>> >>>>>>>>>>>>> images of Gremlin Server.
>>> >>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>> >>>>>>>>>>>>> able to
>>> >>>>>>>>>> use
>>> >>>>>>>>>>>>> images that are only available locally (testcontainers-java
>>> >>>>>>>>>>>>> already
>>> >>>>>>>>>> has
>>> >>>>>>>>>>>>> that capability), but now that I think more about this, I
>>> >>>>>>>>>>>>> think that
>>> >>>>>>>>>> it
>>> >>>>>>>>>>>>> makes actually more sense to use images that are available
>>> in
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>> registry
>>> >>>>>>>>>>>>> as every developer would otherwise have to build test
>>> images
>>> >>>>>>>>>>>>> first
>>> >>>>>>>>>>>> locally
>>> >>>>>>>>>>>>> in order to use them for tests.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers
>>> to
>>> >>>>>>>>>>>>> see
>>> >>>>>>>>>>> whether
>>> >>>>>>>>>>>>> you agree that it's a good solution to our problem and then
>>> >>>>>> we
>>> >>>>>>>>>>>>> can
>>> >>>>>>>>>> see
>>> >>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>>> >>>>>> or
>>> >>>>>>>>>>>>> how we
>>> >>>>>>>>>>>> want
>>> >>>>>>>>>>>>> to handle that.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>>> >>>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>> >>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>> >>>>>>>>>>>>> An: dev@tinkerpop.apache.org
>>> >>>>>>>>>>>>> Betreff: docker and GLV development
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>> >>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love
>>> to
>>> >>>>>>>>>>>>> be able to do
>>> >>>>>>>>>>>> something
>>> >>>>>>>>>>>>> like:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> docker/gremlin-server.sh -test
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> which would start Gremlin Server with our standard test
>>> >>>>>>>>>>>>> configuration (currently built into the maven tool chain,
>>> but
>>> >>>>>>>>>>>>> I think that could be extracted without too much
>>> difficulty).
>>> >>>>>>>>>>>>> That way, we could easily
>>> >>>>>>>>>> open
>>> >>>>>>>>>>> up
>>> >>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly
>>> in
>>> >>>>>>>>>>>>> there,
>>> >>>>>>>>>>> use
>>> >>>>>>>>>>>>> the debugger, etc.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Sound useful? and if so, is that something that is
>>> >>>>>>>>>>>>> easy/possible? and
>>> >>>>>>>>>>>> then
>>> >>>>>>>>>>>>> further....any volunteers with the know-how to get that
>>> setup
>>> >>>>>>>>>>>>> (i'm
>>> >>>>>>>>>>> happy
>>> >>>>>>>>>>>> to
>>> >>>>>>>>>>>>> assist however i can)?
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for
>>> GLV
>>> >>>>>>>>>>> development
>>> >>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>>> >>>>>> mind
>>> >>>>>>>>>>>>> using
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>>>> maven integration but i feel like i could be more efficient
>>> >>>>>>>>>>>>> with a
>>> >>>>>>>>>> more
>>> >>>>>>>>>>>>> streamlined setup.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>>
>>>

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

I just pushed a change that allows python tests to pass against the docker
image.

This will install the gremlin-python script support on the server so that's
an extra download to wait for.

I had to update the localhost references to point to the docker IP
in gremlin-python:   grep -Rl localhost gremlin-python/src/main/jython/ |
xargs sed -i 's/localhost/172.17.0.2/g'

Then I could `mvn clean install -pl :gremlin-python`

Robert Dale


On Wed, Mar 6, 2019 at 9:22 AM Robert Dale <ro...@gmail.com> wrote:

> Florian makes a great point about using the image from gremlin-server.
> It's much lighter weight.  It also makes it easier for non-java,
> client-only devs to use since it doesn't require maven if a published
> version is used. I've pushed my changes.
>
> Thus,
> # Will use the published gremlin-server image with the 'latest' tag.
> ./docker/gremlin-server.sh
>
> # Will use the published gremlin-server image with the specified version
> ./docker/gremlin-server.sh 3.4.0
>
> # To use the working copy, build gremlin-server and the image. e.g.:
> mvn clean install -pl :gremlin-server -am && mvn -DskipTests install
> -Pdocker-images -pl :gremlin-server
>
> # You still have to know what the working copy version is. Docker will
> check local images.
> ./docker/gremlin-server.sh 3.4.1-SNAPSHOT
>
> In all cases, the test scripts are still copied from the working copy so
> be aware of any version conflicts between gremlin-server version used and
> working copy version test scripts.
>
> I've also retained the original image's ability to override the yaml file.
> However, the version is required even when using 'latest' tag.
>
> # Will start latest published gremlin-server with the default config on
> port 8182
> ./docker/gremlin-server.sh latest conf/gremlin-server.yaml
>
>
> Robert Dale
>
>
> On Fri, Mar 1, 2019 at 3:13 PM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>> Wow, for some reason I didn't see your earlier mail and also the linked
>> PR. So, I didn't know that there is already some automation to try out.
>>
>> That made my attempts to help of course a bit pointless :-/ Sorry
>> Stephen if I waisted your time with that.
>>
>> Regarding our initial discussion here: I think that Testcontainers would
>> be the best solution for the GLVs (gremlin-driver was probably a bad
>> example), at least in the long term, as it would also enable us to test
>> modification steps without breaking other tests. Nevertheless, I agree
>> that a simpler approach might be the better solution for now and it is
>> definitely an improvement over our current situation.
>>
>> I will probably wait for testcontainers-dotnet to support using images
>> that are only available locally (so we can test with a locally built
>> Gremlin Server) and then test out how good it works with our build
>> process.
>>
>> Am 01.03.2019 um 20:50 schrieb Robert Dale:
>> > I force pushed a fix.  Wasn't listening on all nets. Also added a banner
>> > for IP address that will appear like so:
>> > ...
>> > Successfully built b00dc1f2db35
>> > #######################
>> > IP is 172.17.0.2
>> > #######################
>> > [INFO] GremlinServer - 3.4.1-SNAPSHOT
>> > ...
>> >
>> > Robert Dale
>> >
>> >
>> > On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <fh@florian-hockmann.de
>> >
>> > wrote:
>> >
>> >> I don't know what the problem could be, but two things I can think of
>> >> that could be helpful to find the problem:
>> >>
>> >>   * Connect to the container (via docker exec -it [container_id] bash)
>> >>     and then confirm with netstat (which you have to install,  `apk
>> >>     update; apk add ospd-netstat` should work) that port 45940 is
>> >>     actually open and that the foreign address is 0.0.0.0 (`netstat
>> -tlpn`)
>> >>   * Check whether you can connect to that port in general without
>> adding
>> >>     complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>> >>     localhost 45940`.
>> >>
>> >> Simply restarting Docker could also help. (At least on Windows some
>> >> strange Docker networks problems are often gone after a restart.)
>> >>
>> >> You're probably using Gremlin Console from your host system and not
>> with
>> >> our Docker image, right?
>> >>
>> >> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
>> >>> i tried adding that -p thing to the run command in the script, but i
>> >> still
>> >>> couldn't connect to it with Gremlin Console
>> >>>
>> >>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <
>> fh@florian-hockmann.de>
>> >>> wrote:
>> >>>
>> >>>> Did you start the container with port mapping (e.g., docker run -p
>> >>>> 45940:45940 [image-name]) or how are you trying to connect to the
>> >>>> container?
>> >>>>
>> >>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>> >>>>> The server starts but something seems messed up with the ports. It's
>> >>>> like I
>> >>>>> can't connect to the 45940 port from the host. I tried a couple
>> things
>> >> I
>> >>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's
>> >> probably
>> >>>>> something simple...any ideas?
>> >>>>>
>> >>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> I made a first attempt.  Let me know what you think.
>> >>>>>> https://github.com/apache/tinkerpop/pull/1075
>> >>>>>>
>> >>>>>> Robert Dale
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
>> >> spmallette@gmail.com
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>>>  If we agree on Testcontainers in general, then I can give this a
>> >> try
>> >>>>>> and
>> >>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> >> Server
>> >>>>>> for
>> >>>>>>> testing (gremlin-driver maybe?).
>> >>>>>>>
>> >>>>>>> The java integration tests for Gremlin Server flip a lot of
>> >>>> configuration
>> >>>>>>> bits per test. I don't know how that works with testcontainers
>> and if
>> >>>> it
>> >>>>>>> will make our testing there any better. There are also a number of
>> >>>> tests
>> >>>>>>> that stop/start the server under different conditions - not sure
>> how
>> >>>> that
>> >>>>>>> works with testcontainers either. Finally, wouldn't we expect
>> adding
>> >>>>>> docker
>> >>>>>>> to the mix to slow down our integration test runs for java?
>> >>>>>>>
>> >>>>>>> I don't want to seem like i'm completely against testcontainers
>> btw.
>> >> I
>> >>>>>> just
>> >>>>>>> wonder if a script doesn't give us a 100% better experience than
>> what
>> >>>> we
>> >>>>>>> have now (which is nothing). If so, and the script is easy and
>> >>>>>> zero-impact
>> >>>>>>> without new dependencies, then I think I like that approach.
>> Also, It
>> >>>>>> would
>> >>>>>>> be nice to have the script for other sorts of manual testing,
>> such as
>> >>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>> >>>>>>> testcontainers for GLVs I suppose that can be done too, but for my
>> >>>>>>> purposes, since I was the one who asked for help with this
>> initially,
>> >>>>>>> testcontainers doesn't sound like something that is an absolute
>> >>>>>> requirement
>> >>>>>>> for my needs.
>> >>>>>>>
>> >>>>>>> thanks robert/florian for all the discussion and help -
>> appreciated.
>> >>>>>>>
>> >>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>> >>>>>> fh@florian-hockmann.de>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Yes, except for the detail that Testcontainers could make the
>> script
>> >>>>>>>> obsolete if it is also used to build the image by itself. That
>> would
>> >>>>>> have
>> >>>>>>>> the advantage that developers wouldn't have to execute the
>> script to
>> >>>>>>> build
>> >>>>>>>> the image. (Executing the script with Testcontainers could also
>> be a
>> >>>>>>>> possibility, but I'm not sure how good that works.)
>> >>>>>>>>
>> >>>>>>>> If we agree on Testcontainers in general, then I can give this a
>> try
>> >>>>>> and
>> >>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> >>>> Server
>> >>>>>>> for
>> >>>>>>>> testing (gremlin-driver maybe?).
>> >>>>>>>> With such a PR it would probably also be easier for others to see
>> >> what
>> >>>>>>>> advantages / disadvantages using Testcontainers would have.
>> >>>>>>>>
>> >>>>>>>> I think using a Java project first makes most sense as we can use
>> >> all
>> >>>>>>>> functionality Testcontainers has to offer there. The downside is
>> of
>> >>>>>>> course
>> >>>>>>>> that we might not have all the functionality afterwards for other
>> >>>>>>> languages
>> >>>>>>>> and then we might need to extend the Testcontainers libraries for
>> >> the
>> >>>>>>>> different languages. (My assumption here is that we won't need
>> that
>> >>>>>> much
>> >>>>>>>> functionality so it should be limited to 1 or 2 missing
>> features.)
>> >>>>>>>> Alternatively, we could also start with a GLV where the
>> >> Testcontainers
>> >>>>>>>> library doesn't have the same functionality which probably means
>> >> that
>> >>>>>> we
>> >>>>>>>> need a workaround to get an image of our current Gremlin Server
>> >>>> version
>> >>>>>>> for
>> >>>>>>>> testing.
>> >>>>>>>>
>> >>>>>>>> -----Ursprüngliche Nachricht-----
>> >>>>>>>> Von: Robert Dale <ro...@gmail.com>
>> >>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>> >>>>>>>> An: dev@tinkerpop.apache.org
>> >>>>>>>> Betreff: Re: docker and GLV development
>> >>>>>>>>
>> >>>>>>>> I think there are two components here.
>> >>>>>>>> 1) Script to build the dockerized test server. Includes ability
>> to
>> >>>>>> start
>> >>>>>>>> container manually, externally from tests.
>> >>>>>>>> 2) Testcontainers integration which uses the docker image from
>> #1.
>> >>>>>>>> Testcontainers could also include executing the script in #1 to
>> >> build
>> >>>>>> the
>> >>>>>>>> container.
>> >>>>>>>>
>> >>>>>>>> Does this sound right to you?
>> >>>>>>>>
>> >>>>>>>> Robert Dale
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>> >>>>>> fh@florian-hockmann.de
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>>> I do like the idea of being able to test against older versions
>> >>>>>>> though.
>> >>>>>>>>> That shouldn't be a problem in general as we could use multiple
>> >>>>>>>>> different images. See for example this PR that adds
>> Testcontainers
>> >> to
>> >>>>>>>>> JanusGraph to test different Elasticsearch versions:
>> >>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>> >>>>>>>>>
>> >>>>>>>>>> Yes, an on-the-fly container would work
>> >>>>>>>>> We could however also use an on-the-fly container with
>> >>>>>> Testcontainers.
>> >>>>>>>>> We would just need some automation to build the image in the
>> >>>>>>> background.
>> >>>>>>>>> What if we would build an image during the usual Maven build for
>> >>>>>>>>> testing and give it a descriptive tag like "testing". Then, we
>> >> could
>> >>>>>>>>> use that tag with Testcontainers in addition to tags for already
>> >>>>>>>> released versions.
>> >>>>>>>>> Testcontainers-Java can apparently even build images by itself.
>> So,
>> >>>>>>>>> that could also be a possibility to get an image with the latest
>> >>>>>>>>> Gremlin Server with the current (unreleased) changes.
>> >>>>>>>>>
>> >>>>>>>>> In general, I would really like to get away from a situation
>> where
>> >> we
>> >>>>>>>>> have to start some script in the background before tests can be
>> >>>>>>> executed.
>> >>>>>>>>> @Robert:
>> >>>>>>>>>> What is the test configuration/data for GLVs?  Could you list
>> the
>> >>>>>>>>> paths/files?
>> >>>>>>>>>
>> >>>>>>>>> For Gremlin.Net for example, you can find the automation
>> happening
>> >> in
>> >>>>>>>>> the pom.xml of the testing part:
>> >>>>>>>>>
>> >>>>>>>>>
>> >> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>> >>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>> >>>>>>>>>
>> >>>>>>>>> It calls the test-server-start.groovy script before the tests
>> are
>> >>>>>>>>> executed and afterwards test-server-stop.groovy script to stop
>> the
>> >>>>>>>> server again.
>> >>>>>>>>> This has the obvious downside that the tests can only be
>> executed
>> >>>>>> from
>> >>>>>>>>> the Maven build which isn't the build tool .NET developers would
>> >> use
>> >>>>>>>>> and also not the tool the IDE will use (the situation is
>> basically
>> >>>>>> the
>> >>>>>>>>> same for other GLVs).
>> >>>>>>>>>
>> >>>>>>>>> -----Ursprüngliche Nachricht-----
>> >>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>> >>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>> >>>>>>>>> An: dev@tinkerpop.apache.org
>> >>>>>>>>> Betreff: Re: docker and GLV development
>> >>>>>>>>>
>> >>>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>> >>>>>>>>>
>> >>>>>>>>> What is the test configuration/data for GLVs?  Could you list
>> the
>> >>>>>>>>> paths/files?
>> >>>>>>>>>
>> >>>>>>>>> Robert Dale
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>> >>>>>>>>> <sp...@gmail.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Yes, an on-the-fly container would work assuming it used the
>> test
>> >>>>>>>>>> configuration/data for GLVs. That's why i suggested the
>> command as
>> >>>>>> I
>> >>>>>>>>>> did because it was similar to docker/build.sh.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <robdale@gmail.com
>> >
>> >>>>>>> wrote:
>> >>>>>>>>>>> Do you want to create a docker container on-the-fly using the
>> >>>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Robert Dale
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>> >>>>>>>>>>> <sp...@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> testcontainers seems neat as it gives some good integration
>> and
>> >>>>>>>>>>> automation,
>> >>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>> >>>>>>>>>>>> There are times when changes are occurring on both client and
>> >>>>>>>>>>>> server so a local up-to-the-moment image would be necessary.
>> I
>> >>>>>>>>>>>> also think we might want
>> >>>>>>>>>> to
>> >>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>> >>>>>>>>>>>> more
>> >>>>>>>>>>> clearly
>> >>>>>>>>>>>> how those play into projects (there is a bit of discussion
>> >>>>>> going
>> >>>>>>>>>>>> on
>> >>>>>>>>>> with
>> >>>>>>>>>>>> that now). I do like the idea of being able to test against
>> >>>>>>>>>>>> older
>> >>>>>>>>>>> versions
>> >>>>>>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>> >>>>>>>>>> fh@florian-hockmann.de
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>> >>>>>> tests
>> >>>>>>>>>>>>> for Gremlin.Net from my IDE.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>> >>>>>>>>>>>>> problems
>> >>>>>>>>>>> like
>> >>>>>>>>>>>>> this:
>> >>>>>>>>>>>>> https://www.testcontainers.org/
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>> >>>>>>>>>> container
>> >>>>>>>>>>> in
>> >>>>>>>>>>>>> your tests for whatever external dependencies you need.
>> >>>>>>>>>>>>> Databases
>> >>>>>>>>>> are a
>> >>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>> >>>>>>>>>>>>> Gremlin
>> >>>>>>>>>>> Server
>> >>>>>>>>>>>>> would be such a container.
>> >>>>>>>>>>>>> Testcontainers then uses the ability of the testing
>> framework
>> >>>>>>>> (e.g.
>> >>>>>>>>>>>> JUnit)
>> >>>>>>>>>>>>> to start the Docker container for the tests and waits until
>> >>>>>>>>>>>>> they are
>> >>>>>>>>>>>> fully
>> >>>>>>>>>>>>> started (how they are started can be configured for each
>> >>>>>>>>>>>>> container differently). After the tests have completed, it
>> >>>>>>>>>>>>> also stops the
>> >>>>>>>>>>>> containers
>> >>>>>>>>>>>>> again.
>> >>>>>>>>>>>>> That way, you don't have to manage the containers manually
>> >>>>>>>> anymore.
>> >>>>>>>>>>>>> Everything is done in the background by the IDE / the build
>> >>>>>>> tool.
>> >>>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>> >>>>>>>>>>>>> also have libraries for different languages, including the
>> >>>>>>>>>>>>> ones for which we
>> >>>>>>>>>> have
>> >>>>>>>>>>>>> GLVs:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> testcontainers-python:
>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>> >>>>>>>>>>>>> testcontainers-dotnet
>> >>>>>>>>>>>>> <
>> >>>>>>
>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>> >>>>>>>>>> s-
>> >>>>>>>>>> dotnet
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>> >>>>>>>>>>>>> testcontainers-node
>> >>>>>>>>>>>>> <
>> >>>>>>
>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>> >>>>>>>>>> s-
>> >>>>>>>>>> node
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Testcontainers-java is the most advanced library of these of
>> >>>>>>>>> course.
>> >>>>>>>>>>> The
>> >>>>>>>>>>>>> others still lack some functionality, but I guess that we
>> >>>>>>>>>>>>> could also
>> >>>>>>>>>>> just
>> >>>>>>>>>>>>> contribute to those libraries if an important feature is
>> >>>>>>>>>>>>> missing for
>> >>>>>>>>>>> us.
>> >>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>> >>>>>>>>>>> testcontainers-dotnet
>> >>>>>>>>>>>>> unfortunately can only use images that are available in some
>> >>>>>>>>>> registry.
>> >>>>>>>>>>>> So,
>> >>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>> >>>>>>>>>>>>> DockerHub
>> >>>>>>>>>> to
>> >>>>>>>>>>>> use
>> >>>>>>>>>>>>> it in our tests. That means that we could either only test
>> >>>>>> the
>> >>>>>>>>>>>>> GLVs
>> >>>>>>>>>>>> against
>> >>>>>>>>>>>>> already released versions or we could publish something like
>> >>>>>>>>>>>>> nightly
>> >>>>>>>>>>>> build
>> >>>>>>>>>>>>> images of Gremlin Server.
>> >>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>> >>>>>>>>>>>>> able to
>> >>>>>>>>>> use
>> >>>>>>>>>>>>> images that are only available locally (testcontainers-java
>> >>>>>>>>>>>>> already
>> >>>>>>>>>> has
>> >>>>>>>>>>>>> that capability), but now that I think more about this, I
>> >>>>>>>>>>>>> think that
>> >>>>>>>>>> it
>> >>>>>>>>>>>>> makes actually more sense to use images that are available
>> in
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>> registry
>> >>>>>>>>>>>>> as every developer would otherwise have to build test images
>> >>>>>>>>>>>>> first
>> >>>>>>>>>>>> locally
>> >>>>>>>>>>>>> in order to use them for tests.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>> >>>>>>>>>>>>> see
>> >>>>>>>>>>> whether
>> >>>>>>>>>>>>> you agree that it's a good solution to our problem and then
>> >>>>>> we
>> >>>>>>>>>>>>> can
>> >>>>>>>>>> see
>> >>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>> >>>>>> or
>> >>>>>>>>>>>>> how we
>> >>>>>>>>>>>> want
>> >>>>>>>>>>>>> to handle that.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>> >>>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>> >>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>> >>>>>>>>>>>>> An: dev@tinkerpop.apache.org
>> >>>>>>>>>>>>> Betreff: docker and GLV development
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>> >>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love
>> to
>> >>>>>>>>>>>>> be able to do
>> >>>>>>>>>>>> something
>> >>>>>>>>>>>>> like:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> docker/gremlin-server.sh -test
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> which would start Gremlin Server with our standard test
>> >>>>>>>>>>>>> configuration (currently built into the maven tool chain,
>> but
>> >>>>>>>>>>>>> I think that could be extracted without too much
>> difficulty).
>> >>>>>>>>>>>>> That way, we could easily
>> >>>>>>>>>> open
>> >>>>>>>>>>> up
>> >>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly
>> in
>> >>>>>>>>>>>>> there,
>> >>>>>>>>>>> use
>> >>>>>>>>>>>>> the debugger, etc.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Sound useful? and if so, is that something that is
>> >>>>>>>>>>>>> easy/possible? and
>> >>>>>>>>>>>> then
>> >>>>>>>>>>>>> further....any volunteers with the know-how to get that
>> setup
>> >>>>>>>>>>>>> (i'm
>> >>>>>>>>>>> happy
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>>> assist however i can)?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for
>> GLV
>> >>>>>>>>>>> development
>> >>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>> >>>>>> mind
>> >>>>>>>>>>>>> using
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>> maven integration but i feel like i could be more efficient
>> >>>>>>>>>>>>> with a
>> >>>>>>>>>> more
>> >>>>>>>>>>>>> streamlined setup.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>>
>>

Re: docker and GLV development

[Florian Hockmann <fh...@florian-hockmann.de>]

Wow, for some reason I didn't see your earlier mail and also the linked
PR. So, I didn't know that there is already some automation to try out.

That made my attempts to help of course a bit pointless :-/ Sorry
Stephen if I waisted your time with that.

Regarding our initial discussion here: I think that Testcontainers would
be the best solution for the GLVs (gremlin-driver was probably a bad
example), at least in the long term, as it would also enable us to test
modification steps without breaking other tests. Nevertheless, I agree
that a simpler approach might be the better solution for now and it is
definitely an improvement over our current situation.

I will probably wait for testcontainers-dotnet to support using images
that are only available locally (so we can test with a locally built
Gremlin Server) and then test out how good it works with our build process.

Am 01.03.2019 um 20:50 schrieb Robert Dale:
> I force pushed a fix.  Wasn't listening on all nets. Also added a banner
> for IP address that will appear like so:
> ...
> Successfully built b00dc1f2db35
> #######################
> IP is 172.17.0.2
> #######################
> [INFO] GremlinServer - 3.4.1-SNAPSHOT
> ...
>
> Robert Dale
>
>
> On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>> I don't know what the problem could be, but two things I can think of
>> that could be helpful to find the problem:
>>
>>   * Connect to the container (via docker exec -it [container_id] bash)
>>     and then confirm with netstat (which you have to install,  `apk
>>     update; apk add ospd-netstat` should work) that port 45940 is
>>     actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`)
>>   * Check whether you can connect to that port in general without adding
>>     complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>>     localhost 45940`.
>>
>> Simply restarting Docker could also help. (At least on Windows some
>> strange Docker networks problems are often gone after a restart.)
>>
>> You're probably using Gremlin Console from your host system and not with
>> our Docker image, right?
>>
>> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
>>> i tried adding that -p thing to the run command in the script, but i
>> still
>>> couldn't connect to it with Gremlin Console
>>>
>>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
>>> wrote:
>>>
>>>> Did you start the container with port mapping (e.g., docker run -p
>>>> 45940:45940 [image-name]) or how are you trying to connect to the
>>>> container?
>>>>
>>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>>>>> The server starts but something seems messed up with the ports. It's
>>>> like I
>>>>> can't connect to the 45940 port from the host. I tried a couple things
>> I
>>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's
>> probably
>>>>> something simple...any ideas?
>>>>>
>>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
>>>>>
>>>>>> I made a first attempt.  Let me know what you think.
>>>>>> https://github.com/apache/tinkerpop/pull/1075
>>>>>>
>>>>>> Robert Dale
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
>> spmallette@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>>>  If we agree on Testcontainers in general, then I can give this a
>> try
>>>>>> and
>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> Server
>>>>>> for
>>>>>>> testing (gremlin-driver maybe?).
>>>>>>>
>>>>>>> The java integration tests for Gremlin Server flip a lot of
>>>> configuration
>>>>>>> bits per test. I don't know how that works with testcontainers and if
>>>> it
>>>>>>> will make our testing there any better. There are also a number of
>>>> tests
>>>>>>> that stop/start the server under different conditions - not sure how
>>>> that
>>>>>>> works with testcontainers either. Finally, wouldn't we expect adding
>>>>>> docker
>>>>>>> to the mix to slow down our integration test runs for java?
>>>>>>>
>>>>>>> I don't want to seem like i'm completely against testcontainers btw.
>> I
>>>>>> just
>>>>>>> wonder if a script doesn't give us a 100% better experience than what
>>>> we
>>>>>>> have now (which is nothing). If so, and the script is easy and
>>>>>> zero-impact
>>>>>>> without new dependencies, then I think I like that approach. Also, It
>>>>>> would
>>>>>>> be nice to have the script for other sorts of manual testing, such as
>>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>>>>>> testcontainers for GLVs I suppose that can be done too, but for my
>>>>>>> purposes, since I was the one who asked for help with this initially,
>>>>>>> testcontainers doesn't sound like something that is an absolute
>>>>>> requirement
>>>>>>> for my needs.
>>>>>>>
>>>>>>> thanks robert/florian for all the discussion and help - appreciated.
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>>>>>> fh@florian-hockmann.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, except for the detail that Testcontainers could make the script
>>>>>>>> obsolete if it is also used to build the image by itself. That would
>>>>>> have
>>>>>>>> the advantage that developers wouldn't have to execute the script to
>>>>>>> build
>>>>>>>> the image. (Executing the script with Testcontainers could also be a
>>>>>>>> possibility, but I'm not sure how good that works.)
>>>>>>>>
>>>>>>>> If we agree on Testcontainers in general, then I can give this a try
>>>>>> and
>>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>>>> Server
>>>>>>> for
>>>>>>>> testing (gremlin-driver maybe?).
>>>>>>>> With such a PR it would probably also be easier for others to see
>> what
>>>>>>>> advantages / disadvantages using Testcontainers would have.
>>>>>>>>
>>>>>>>> I think using a Java project first makes most sense as we can use
>> all
>>>>>>>> functionality Testcontainers has to offer there. The downside is of
>>>>>>> course
>>>>>>>> that we might not have all the functionality afterwards for other
>>>>>>> languages
>>>>>>>> and then we might need to extend the Testcontainers libraries for
>> the
>>>>>>>> different languages. (My assumption here is that we won't need that
>>>>>> much
>>>>>>>> functionality so it should be limited to 1 or 2 missing features.)
>>>>>>>> Alternatively, we could also start with a GLV where the
>> Testcontainers
>>>>>>>> library doesn't have the same functionality which probably means
>> that
>>>>>> we
>>>>>>>> need a workaround to get an image of our current Gremlin Server
>>>> version
>>>>>>> for
>>>>>>>> testing.
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>> Betreff: Re: docker and GLV development
>>>>>>>>
>>>>>>>> I think there are two components here.
>>>>>>>> 1) Script to build the dockerized test server. Includes ability to
>>>>>> start
>>>>>>>> container manually, externally from tests.
>>>>>>>> 2) Testcontainers integration which uses the docker image from #1.
>>>>>>>> Testcontainers could also include executing the script in #1 to
>> build
>>>>>> the
>>>>>>>> container.
>>>>>>>>
>>>>>>>> Does this sound right to you?
>>>>>>>>
>>>>>>>> Robert Dale
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>>>>>> fh@florian-hockmann.de
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>> I do like the idea of being able to test against older versions
>>>>>>> though.
>>>>>>>>> That shouldn't be a problem in general as we could use multiple
>>>>>>>>> different images. See for example this PR that adds Testcontainers
>> to
>>>>>>>>> JanusGraph to test different Elasticsearch versions:
>>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>>>>>>>>
>>>>>>>>>> Yes, an on-the-fly container would work
>>>>>>>>> We could however also use an on-the-fly container with
>>>>>> Testcontainers.
>>>>>>>>> We would just need some automation to build the image in the
>>>>>>> background.
>>>>>>>>> What if we would build an image during the usual Maven build for
>>>>>>>>> testing and give it a descriptive tag like "testing". Then, we
>> could
>>>>>>>>> use that tag with Testcontainers in addition to tags for already
>>>>>>>> released versions.
>>>>>>>>> Testcontainers-Java can apparently even build images by itself. So,
>>>>>>>>> that could also be a possibility to get an image with the latest
>>>>>>>>> Gremlin Server with the current (unreleased) changes.
>>>>>>>>>
>>>>>>>>> In general, I would really like to get away from a situation where
>> we
>>>>>>>>> have to start some script in the background before tests can be
>>>>>>> executed.
>>>>>>>>> @Robert:
>>>>>>>>>> What is the test configuration/data for GLVs?  Could you list the
>>>>>>>>> paths/files?
>>>>>>>>>
>>>>>>>>> For Gremlin.Net for example, you can find the automation happening
>> in
>>>>>>>>> the pom.xml of the testing part:
>>>>>>>>>
>>>>>>>>>
>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>>>>>>>>
>>>>>>>>> It calls the test-server-start.groovy script before the tests are
>>>>>>>>> executed and afterwards test-server-stop.groovy script to stop the
>>>>>>>> server again.
>>>>>>>>> This has the obvious downside that the tests can only be executed
>>>>>> from
>>>>>>>>> the Maven build which isn't the build tool .NET developers would
>> use
>>>>>>>>> and also not the tool the IDE will use (the situation is basically
>>>>>> the
>>>>>>>>> same for other GLVs).
>>>>>>>>>
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>> Betreff: Re: docker and GLV development
>>>>>>>>>
>>>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>>>>>>>>
>>>>>>>>> What is the test configuration/data for GLVs?  Could you list the
>>>>>>>>> paths/files?
>>>>>>>>>
>>>>>>>>> Robert Dale
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>>>>>>>> <sp...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, an on-the-fly container would work assuming it used the test
>>>>>>>>>> configuration/data for GLVs. That's why i suggested the command as
>>>>>> I
>>>>>>>>>> did because it was similar to docker/build.sh.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>> Do you want to create a docker container on-the-fly using the
>>>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>>>>>>>>>>>
>>>>>>>>>>> Robert Dale
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>>>>>>>>>> <sp...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> testcontainers seems neat as it gives some good integration and
>>>>>>>>>>> automation,
>>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>>>>>>>>>>>> There are times when changes are occurring on both client and
>>>>>>>>>>>> server so a local up-to-the-moment image would be necessary. I
>>>>>>>>>>>> also think we might want
>>>>>>>>>> to
>>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>>>>>>>>>>>> more
>>>>>>>>>>> clearly
>>>>>>>>>>>> how those play into projects (there is a bit of discussion
>>>>>> going
>>>>>>>>>>>> on
>>>>>>>>>> with
>>>>>>>>>>>> that now). I do like the idea of being able to test against
>>>>>>>>>>>> older
>>>>>>>>>>> versions
>>>>>>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>>>>>>>>> fh@florian-hockmann.de
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>>>>>> tests
>>>>>>>>>>>>> for Gremlin.Net from my IDE.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>>>>>>>>>>>> problems
>>>>>>>>>>> like
>>>>>>>>>>>>> this:
>>>>>>>>>>>>> https://www.testcontainers.org/
>>>>>>>>>>>>>
>>>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>>>>>>>>>> container
>>>>>>>>>>> in
>>>>>>>>>>>>> your tests for whatever external dependencies you need.
>>>>>>>>>>>>> Databases
>>>>>>>>>> are a
>>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>>>>>>>>>>>> Gremlin
>>>>>>>>>>> Server
>>>>>>>>>>>>> would be such a container.
>>>>>>>>>>>>> Testcontainers then uses the ability of the testing framework
>>>>>>>> (e.g.
>>>>>>>>>>>> JUnit)
>>>>>>>>>>>>> to start the Docker container for the tests and waits until
>>>>>>>>>>>>> they are
>>>>>>>>>>>> fully
>>>>>>>>>>>>> started (how they are started can be configured for each
>>>>>>>>>>>>> container differently). After the tests have completed, it
>>>>>>>>>>>>> also stops the
>>>>>>>>>>>> containers
>>>>>>>>>>>>> again.
>>>>>>>>>>>>> That way, you don't have to manage the containers manually
>>>>>>>> anymore.
>>>>>>>>>>>>> Everything is done in the background by the IDE / the build
>>>>>>> tool.
>>>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>>>>>>>>>>>>> also have libraries for different languages, including the
>>>>>>>>>>>>> ones for which we
>>>>>>>>>> have
>>>>>>>>>>>>> GLVs:
>>>>>>>>>>>>>
>>>>>>>>>>>>> testcontainers-python:
>>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>>>>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>>>> <
>>>>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>>>>>>>>> s-
>>>>>>>>>> dotnet
>>>>>>>>>>>>> :
>>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>>>>>>>>>>>> testcontainers-node
>>>>>>>>>>>>> <
>>>>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>>>>>>>>> s-
>>>>>>>>>> node
>>>>>>>>>>>>> :
>>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>>>>>>>>>>>>
>>>>>>>>>>>>> Testcontainers-java is the most advanced library of these of
>>>>>>>>> course.
>>>>>>>>>>> The
>>>>>>>>>>>>> others still lack some functionality, but I guess that we
>>>>>>>>>>>>> could also
>>>>>>>>>>> just
>>>>>>>>>>>>> contribute to those libraries if an important feature is
>>>>>>>>>>>>> missing for
>>>>>>>>>>> us.
>>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>>>> unfortunately can only use images that are available in some
>>>>>>>>>> registry.
>>>>>>>>>>>> So,
>>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>>>>>>>>>>>> DockerHub
>>>>>>>>>> to
>>>>>>>>>>>> use
>>>>>>>>>>>>> it in our tests. That means that we could either only test
>>>>>> the
>>>>>>>>>>>>> GLVs
>>>>>>>>>>>> against
>>>>>>>>>>>>> already released versions or we could publish something like
>>>>>>>>>>>>> nightly
>>>>>>>>>>>> build
>>>>>>>>>>>>> images of Gremlin Server.
>>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>>>>>>>>>>>> able to
>>>>>>>>>> use
>>>>>>>>>>>>> images that are only available locally (testcontainers-java
>>>>>>>>>>>>> already
>>>>>>>>>> has
>>>>>>>>>>>>> that capability), but now that I think more about this, I
>>>>>>>>>>>>> think that
>>>>>>>>>> it
>>>>>>>>>>>>> makes actually more sense to use images that are available in
>>>>>>>>>>>>> the
>>>>>>>>>>>> registry
>>>>>>>>>>>>> as every developer would otherwise have to build test images
>>>>>>>>>>>>> first
>>>>>>>>>>>> locally
>>>>>>>>>>>>> in order to use them for tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>>>>>>>>>>>>> see
>>>>>>>>>>> whether
>>>>>>>>>>>>> you agree that it's a good solution to our problem and then
>>>>>> we
>>>>>>>>>>>>> can
>>>>>>>>>> see
>>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>>>>>> or
>>>>>>>>>>>>> how we
>>>>>>>>>>>> want
>>>>>>>>>>>>> to handle that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>>>>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>>>>>> Betreff: docker and GLV development
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love to
>>>>>>>>>>>>> be able to do
>>>>>>>>>>>> something
>>>>>>>>>>>>> like:
>>>>>>>>>>>>>
>>>>>>>>>>>>> docker/gremlin-server.sh -test
>>>>>>>>>>>>>
>>>>>>>>>>>>> which would start Gremlin Server with our standard test
>>>>>>>>>>>>> configuration (currently built into the maven tool chain, but
>>>>>>>>>>>>> I think that could be extracted without too much difficulty).
>>>>>>>>>>>>> That way, we could easily
>>>>>>>>>> open
>>>>>>>>>>> up
>>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
>>>>>>>>>>>>> there,
>>>>>>>>>>> use
>>>>>>>>>>>>> the debugger, etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sound useful? and if so, is that something that is
>>>>>>>>>>>>> easy/possible? and
>>>>>>>>>>>> then
>>>>>>>>>>>>> further....any volunteers with the know-how to get that setup
>>>>>>>>>>>>> (i'm
>>>>>>>>>>> happy
>>>>>>>>>>>> to
>>>>>>>>>>>>> assist however i can)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
>>>>>>>>>>> development
>>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>>>>>> mind
>>>>>>>>>>>>> using
>>>>>>>>>>> the
>>>>>>>>>>>>> maven integration but i feel like i could be more efficient
>>>>>>>>>>>>> with a
>>>>>>>>>> more
>>>>>>>>>>>>> streamlined setup.
>>>>>>>>>>>>>
>>>>>>>>>>>>>

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

I force pushed a fix.  Wasn't listening on all nets. Also added a banner
for IP address that will appear like so:
...
Successfully built b00dc1f2db35
#######################
IP is 172.17.0.2
#######################
[INFO] GremlinServer - 3.4.1-SNAPSHOT
...

Robert Dale


On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> I don't know what the problem could be, but two things I can think of
> that could be helpful to find the problem:
>
>   * Connect to the container (via docker exec -it [container_id] bash)
>     and then confirm with netstat (which you have to install,  `apk
>     update; apk add ospd-netstat` should work) that port 45940 is
>     actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`)
>   * Check whether you can connect to that port in general without adding
>     complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>     localhost 45940`.
>
> Simply restarting Docker could also help. (At least on Windows some
> strange Docker networks problems are often gone after a restart.)
>
> You're probably using Gremlin Console from your host system and not with
> our Docker image, right?
>
> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
> > i tried adding that -p thing to the run command in the script, but i
> still
> > couldn't connect to it with Gremlin Console
> >
> > On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
> > wrote:
> >
> >> Did you start the container with port mapping (e.g., docker run -p
> >> 45940:45940 [image-name]) or how are you trying to connect to the
> >> container?
> >>
> >> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
> >>> The server starts but something seems messed up with the ports. It's
> >> like I
> >>> can't connect to the 45940 port from the host. I tried a couple things
> I
> >>> saw on stackoverflow but nothing i tried seemed to fix it. it's
> probably
> >>> something simple...any ideas?
> >>>
> >>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
> >>>
> >>>> I made a first attempt.  Let me know what you think.
> >>>> https://github.com/apache/tinkerpop/pull/1075
> >>>>
> >>>> Robert Dale
> >>>>
> >>>>
> >>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
> spmallette@gmail.com
> >>>> wrote:
> >>>>
> >>>>>>  If we agree on Testcontainers in general, then I can give this a
> try
> >>>> and
> >>>>> use Testcontainers in one of our Java projects that need Gremlin
> Server
> >>>> for
> >>>>> testing (gremlin-driver maybe?).
> >>>>>
> >>>>> The java integration tests for Gremlin Server flip a lot of
> >> configuration
> >>>>> bits per test. I don't know how that works with testcontainers and if
> >> it
> >>>>> will make our testing there any better. There are also a number of
> >> tests
> >>>>> that stop/start the server under different conditions - not sure how
> >> that
> >>>>> works with testcontainers either. Finally, wouldn't we expect adding
> >>>> docker
> >>>>> to the mix to slow down our integration test runs for java?
> >>>>>
> >>>>> I don't want to seem like i'm completely against testcontainers btw.
> I
> >>>> just
> >>>>> wonder if a script doesn't give us a 100% better experience than what
> >> we
> >>>>> have now (which is nothing). If so, and the script is easy and
> >>>> zero-impact
> >>>>> without new dependencies, then I think I like that approach. Also, It
> >>>> would
> >>>>> be nice to have the script for other sorts of manual testing, such as
> >>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
> >>>>> testcontainers for GLVs I suppose that can be done too, but for my
> >>>>> purposes, since I was the one who asked for help with this initially,
> >>>>> testcontainers doesn't sound like something that is an absolute
> >>>> requirement
> >>>>> for my needs.
> >>>>>
> >>>>> thanks robert/florian for all the discussion and help - appreciated.
> >>>>>
> >>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
> >>>> fh@florian-hockmann.de>
> >>>>> wrote:
> >>>>>
> >>>>>> Yes, except for the detail that Testcontainers could make the script
> >>>>>> obsolete if it is also used to build the image by itself. That would
> >>>> have
> >>>>>> the advantage that developers wouldn't have to execute the script to
> >>>>> build
> >>>>>> the image. (Executing the script with Testcontainers could also be a
> >>>>>> possibility, but I'm not sure how good that works.)
> >>>>>>
> >>>>>> If we agree on Testcontainers in general, then I can give this a try
> >>>> and
> >>>>>> use Testcontainers in one of our Java projects that need Gremlin
> >> Server
> >>>>> for
> >>>>>> testing (gremlin-driver maybe?).
> >>>>>> With such a PR it would probably also be easier for others to see
> what
> >>>>>> advantages / disadvantages using Testcontainers would have.
> >>>>>>
> >>>>>> I think using a Java project first makes most sense as we can use
> all
> >>>>>> functionality Testcontainers has to offer there. The downside is of
> >>>>> course
> >>>>>> that we might not have all the functionality afterwards for other
> >>>>> languages
> >>>>>> and then we might need to extend the Testcontainers libraries for
> the
> >>>>>> different languages. (My assumption here is that we won't need that
> >>>> much
> >>>>>> functionality so it should be limited to 1 or 2 missing features.)
> >>>>>> Alternatively, we could also start with a GLV where the
> Testcontainers
> >>>>>> library doesn't have the same functionality which probably means
> that
> >>>> we
> >>>>>> need a workaround to get an image of our current Gremlin Server
> >> version
> >>>>> for
> >>>>>> testing.
> >>>>>>
> >>>>>> -----Ursprüngliche Nachricht-----
> >>>>>> Von: Robert Dale <ro...@gmail.com>
> >>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
> >>>>>> An: dev@tinkerpop.apache.org
> >>>>>> Betreff: Re: docker and GLV development
> >>>>>>
> >>>>>> I think there are two components here.
> >>>>>> 1) Script to build the dockerized test server. Includes ability to
> >>>> start
> >>>>>> container manually, externally from tests.
> >>>>>> 2) Testcontainers integration which uses the docker image from #1.
> >>>>>> Testcontainers could also include executing the script in #1 to
> build
> >>>> the
> >>>>>> container.
> >>>>>>
> >>>>>> Does this sound right to you?
> >>>>>>
> >>>>>> Robert Dale
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
> >>>> fh@florian-hockmann.de
> >>>>>> wrote:
> >>>>>>
> >>>>>>>> I do like the idea of being able to test against older versions
> >>>>> though.
> >>>>>>> That shouldn't be a problem in general as we could use multiple
> >>>>>>> different images. See for example this PR that adds Testcontainers
> to
> >>>>>>> JanusGraph to test different Elasticsearch versions:
> >>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
> >>>>>>>
> >>>>>>>> Yes, an on-the-fly container would work
> >>>>>>> We could however also use an on-the-fly container with
> >>>> Testcontainers.
> >>>>>>> We would just need some automation to build the image in the
> >>>>> background.
> >>>>>>> What if we would build an image during the usual Maven build for
> >>>>>>> testing and give it a descriptive tag like "testing". Then, we
> could
> >>>>>>> use that tag with Testcontainers in addition to tags for already
> >>>>>> released versions.
> >>>>>>> Testcontainers-Java can apparently even build images by itself. So,
> >>>>>>> that could also be a possibility to get an image with the latest
> >>>>>>> Gremlin Server with the current (unreleased) changes.
> >>>>>>>
> >>>>>>> In general, I would really like to get away from a situation where
> we
> >>>>>>> have to start some script in the background before tests can be
> >>>>> executed.
> >>>>>>> @Robert:
> >>>>>>>> What is the test configuration/data for GLVs?  Could you list the
> >>>>>>> paths/files?
> >>>>>>>
> >>>>>>> For Gremlin.Net for example, you can find the automation happening
> in
> >>>>>>> the pom.xml of the testing part:
> >>>>>>>
> >>>>>>>
> >>>>
> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> >>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> >>>>>>>
> >>>>>>> It calls the test-server-start.groovy script before the tests are
> >>>>>>> executed and afterwards test-server-stop.groovy script to stop the
> >>>>>> server again.
> >>>>>>> This has the obvious downside that the tests can only be executed
> >>>> from
> >>>>>>> the Maven build which isn't the build tool .NET developers would
> use
> >>>>>>> and also not the tool the IDE will use (the situation is basically
> >>>> the
> >>>>>>> same for other GLVs).
> >>>>>>>
> >>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>> Von: Robert Dale <ro...@gmail.com>
> >>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
> >>>>>>> An: dev@tinkerpop.apache.org
> >>>>>>> Betreff: Re: docker and GLV development
> >>>>>>>
> >>>>>>> I'm surprised Daniel doesn't have this done already ;-)
> >>>>>>>
> >>>>>>> What is the test configuration/data for GLVs?  Could you list the
> >>>>>>> paths/files?
> >>>>>>>
> >>>>>>> Robert Dale
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> >>>>>>> <sp...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Yes, an on-the-fly container would work assuming it used the test
> >>>>>>>> configuration/data for GLVs. That's why i suggested the command as
> >>>> I
> >>>>>>>> did because it was similar to docker/build.sh.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> >>>>> wrote:
> >>>>>>>>> Do you want to create a docker container on-the-fly using the
> >>>>>>>>> current working copy kind of like we do for docker/build.sh?
> >>>>>>>>>
> >>>>>>>>> Robert Dale
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> >>>>>>>>> <sp...@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> testcontainers seems neat as it gives some good integration and
> >>>>>>>>> automation,
> >>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
> >>>>>>>>>> There are times when changes are occurring on both client and
> >>>>>>>>>> server so a local up-to-the-moment image would be necessary. I
> >>>>>>>>>> also think we might want
> >>>>>>>> to
> >>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
> >>>>>>>>>> more
> >>>>>>>>> clearly
> >>>>>>>>>> how those play into projects (there is a bit of discussion
> >>>> going
> >>>>>>>>>> on
> >>>>>>>> with
> >>>>>>>>>> that now). I do like the idea of being able to test against
> >>>>>>>>>> older
> >>>>>>>>> versions
> >>>>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> >>>>>>>> fh@florian-hockmann.de
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Fantastic idea, I often missed the ability to just start
> >>>> tests
> >>>>>>>>>>> for Gremlin.Net from my IDE.
> >>>>>>>>>>>
> >>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
> >>>>>>>>>>> problems
> >>>>>>>>> like
> >>>>>>>>>>> this:
> >>>>>>>>>>> https://www.testcontainers.org/
> >>>>>>>>>>>
> >>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
> >>>>>>>> container
> >>>>>>>>> in
> >>>>>>>>>>> your tests for whatever external dependencies you need.
> >>>>>>>>>>> Databases
> >>>>>>>> are a
> >>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
> >>>>>>>>>>> Gremlin
> >>>>>>>>> Server
> >>>>>>>>>>> would be such a container.
> >>>>>>>>>>> Testcontainers then uses the ability of the testing framework
> >>>>>> (e.g.
> >>>>>>>>>> JUnit)
> >>>>>>>>>>> to start the Docker container for the tests and waits until
> >>>>>>>>>>> they are
> >>>>>>>>>> fully
> >>>>>>>>>>> started (how they are started can be configured for each
> >>>>>>>>>>> container differently). After the tests have completed, it
> >>>>>>>>>>> also stops the
> >>>>>>>>>> containers
> >>>>>>>>>>> again.
> >>>>>>>>>>> That way, you don't have to manage the containers manually
> >>>>>> anymore.
> >>>>>>>>>>> Everything is done in the background by the IDE / the build
> >>>>> tool.
> >>>>>>>>>>> Testcontainers was originally a Java library, but they now
> >>>>>>>>>>> also have libraries for different languages, including the
> >>>>>>>>>>> ones for which we
> >>>>>>>> have
> >>>>>>>>>>> GLVs:
> >>>>>>>>>>>
> >>>>>>>>>>> testcontainers-python:
> >>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
> >>>>>>>>>>> testcontainers-dotnet
> >>>>>>>>>>> <
> >>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
> >>>>>>>> s-
> >>>>>>>> dotnet
> >>>>>>>>>>> :
> >>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
> >>>>>>>>>>> testcontainers-node
> >>>>>>>>>>> <
> >>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
> >>>>>>>> s-
> >>>>>>>> node
> >>>>>>>>>>> :
> >>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
> >>>>>>>>>>>
> >>>>>>>>>>> Testcontainers-java is the most advanced library of these of
> >>>>>>> course.
> >>>>>>>>> The
> >>>>>>>>>>> others still lack some functionality, but I guess that we
> >>>>>>>>>>> could also
> >>>>>>>>> just
> >>>>>>>>>>> contribute to those libraries if an important feature is
> >>>>>>>>>>> missing for
> >>>>>>>>> us.
> >>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
> >>>>>>>>> testcontainers-dotnet
> >>>>>>>>>>> unfortunately can only use images that are available in some
> >>>>>>>> registry.
> >>>>>>>>>> So,
> >>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
> >>>>>>>>>>> DockerHub
> >>>>>>>> to
> >>>>>>>>>> use
> >>>>>>>>>>> it in our tests. That means that we could either only test
> >>>> the
> >>>>>>>>>>> GLVs
> >>>>>>>>>> against
> >>>>>>>>>>> already released versions or we could publish something like
> >>>>>>>>>>> nightly
> >>>>>>>>>> build
> >>>>>>>>>>> images of Gremlin Server.
> >>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
> >>>>>>>>>>> able to
> >>>>>>>> use
> >>>>>>>>>>> images that are only available locally (testcontainers-java
> >>>>>>>>>>> already
> >>>>>>>> has
> >>>>>>>>>>> that capability), but now that I think more about this, I
> >>>>>>>>>>> think that
> >>>>>>>> it
> >>>>>>>>>>> makes actually more sense to use images that are available in
> >>>>>>>>>>> the
> >>>>>>>>>> registry
> >>>>>>>>>>> as every developer would otherwise have to build test images
> >>>>>>>>>>> first
> >>>>>>>>>> locally
> >>>>>>>>>>> in order to use them for tests.
> >>>>>>>>>>>
> >>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
> >>>>>>>>>>> see
> >>>>>>>>> whether
> >>>>>>>>>>> you agree that it's a good solution to our problem and then
> >>>> we
> >>>>>>>>>>> can
> >>>>>>>> see
> >>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
> >>>> or
> >>>>>>>>>>> how we
> >>>>>>>>>> want
> >>>>>>>>>>> to handle that.
> >>>>>>>>>>>
> >>>>>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
> >>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
> >>>>>>>>>>> An: dev@tinkerpop.apache.org
> >>>>>>>>>>> Betreff: docker and GLV development
> >>>>>>>>>>>
> >>>>>>>>>>> Anyone have any idea how to make it so that docker can be
> >>>>>>>>>>> useful for helping to streamline GLV development? I'd love to
> >>>>>>>>>>> be able to do
> >>>>>>>>>> something
> >>>>>>>>>>> like:
> >>>>>>>>>>>
> >>>>>>>>>>> docker/gremlin-server.sh -test
> >>>>>>>>>>>
> >>>>>>>>>>> which would start Gremlin Server with our standard test
> >>>>>>>>>>> configuration (currently built into the maven tool chain, but
> >>>>>>>>>>> I think that could be extracted without too much difficulty).
> >>>>>>>>>>> That way, we could easily
> >>>>>>>> open
> >>>>>>>>> up
> >>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
> >>>>>>>>>>> there,
> >>>>>>>>> use
> >>>>>>>>>>> the debugger, etc.
> >>>>>>>>>>>
> >>>>>>>>>>> Sound useful? and if so, is that something that is
> >>>>>>>>>>> easy/possible? and
> >>>>>>>>>> then
> >>>>>>>>>>> further....any volunteers with the know-how to get that setup
> >>>>>>>>>>> (i'm
> >>>>>>>>> happy
> >>>>>>>>>> to
> >>>>>>>>>>> assist however i can)?
> >>>>>>>>>>>
> >>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
> >>>>>>>>> development
> >>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
> >>>> mind
> >>>>>>>>>>> using
> >>>>>>>>> the
> >>>>>>>>>>> maven integration but i feel like i could be more efficient
> >>>>>>>>>>> with a
> >>>>>>>> more
> >>>>>>>>>>> streamlined setup.
> >>>>>>>>>>>
> >>>>>>>>>>>
>

Re: docker and GLV development

[Florian Hockmann <fh...@florian-hockmann.de>]

I don't know what the problem could be, but two things I can think of
that could be helpful to find the problem:

  * Connect to the container (via docker exec -it [container_id] bash)
    and then confirm with netstat (which you have to install,  `apk
    update; apk add ospd-netstat` should work) that port 45940 is
    actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`)
  * Check whether you can connect to that port in general without adding
    complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
    localhost 45940`.

Simply restarting Docker could also help. (At least on Windows some
strange Docker networks problems are often gone after a restart.)

You're probably using Gremlin Console from your host system and not with
our Docker image, right?

Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
> i tried adding that -p thing to the run command in the script, but i still
> couldn't connect to it with Gremlin Console
>
> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>> Did you start the container with port mapping (e.g., docker run -p
>> 45940:45940 [image-name]) or how are you trying to connect to the
>> container?
>>
>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>>> The server starts but something seems messed up with the ports. It's
>> like I
>>> can't connect to the 45940 port from the host. I tried a couple things I
>>> saw on stackoverflow but nothing i tried seemed to fix it. it's probably
>>> something simple...any ideas?
>>>
>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
>>>
>>>> I made a first attempt.  Let me know what you think.
>>>> https://github.com/apache/tinkerpop/pull/1075
>>>>
>>>> Robert Dale
>>>>
>>>>
>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <spmallette@gmail.com
>>>> wrote:
>>>>
>>>>>>  If we agree on Testcontainers in general, then I can give this a try
>>>> and
>>>>> use Testcontainers in one of our Java projects that need Gremlin Server
>>>> for
>>>>> testing (gremlin-driver maybe?).
>>>>>
>>>>> The java integration tests for Gremlin Server flip a lot of
>> configuration
>>>>> bits per test. I don't know how that works with testcontainers and if
>> it
>>>>> will make our testing there any better. There are also a number of
>> tests
>>>>> that stop/start the server under different conditions - not sure how
>> that
>>>>> works with testcontainers either. Finally, wouldn't we expect adding
>>>> docker
>>>>> to the mix to slow down our integration test runs for java?
>>>>>
>>>>> I don't want to seem like i'm completely against testcontainers btw. I
>>>> just
>>>>> wonder if a script doesn't give us a 100% better experience than what
>> we
>>>>> have now (which is nothing). If so, and the script is easy and
>>>> zero-impact
>>>>> without new dependencies, then I think I like that approach. Also, It
>>>> would
>>>>> be nice to have the script for other sorts of manual testing, such as
>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>>>> testcontainers for GLVs I suppose that can be done too, but for my
>>>>> purposes, since I was the one who asked for help with this initially,
>>>>> testcontainers doesn't sound like something that is an absolute
>>>> requirement
>>>>> for my needs.
>>>>>
>>>>> thanks robert/florian for all the discussion and help - appreciated.
>>>>>
>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>>>> fh@florian-hockmann.de>
>>>>> wrote:
>>>>>
>>>>>> Yes, except for the detail that Testcontainers could make the script
>>>>>> obsolete if it is also used to build the image by itself. That would
>>>> have
>>>>>> the advantage that developers wouldn't have to execute the script to
>>>>> build
>>>>>> the image. (Executing the script with Testcontainers could also be a
>>>>>> possibility, but I'm not sure how good that works.)
>>>>>>
>>>>>> If we agree on Testcontainers in general, then I can give this a try
>>>> and
>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> Server
>>>>> for
>>>>>> testing (gremlin-driver maybe?).
>>>>>> With such a PR it would probably also be easier for others to see what
>>>>>> advantages / disadvantages using Testcontainers would have.
>>>>>>
>>>>>> I think using a Java project first makes most sense as we can use all
>>>>>> functionality Testcontainers has to offer there. The downside is of
>>>>> course
>>>>>> that we might not have all the functionality afterwards for other
>>>>> languages
>>>>>> and then we might need to extend the Testcontainers libraries for the
>>>>>> different languages. (My assumption here is that we won't need that
>>>> much
>>>>>> functionality so it should be limited to 1 or 2 missing features.)
>>>>>> Alternatively, we could also start with a GLV where the Testcontainers
>>>>>> library doesn't have the same functionality which probably means that
>>>> we
>>>>>> need a workaround to get an image of our current Gremlin Server
>> version
>>>>> for
>>>>>> testing.
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>>>>> An: dev@tinkerpop.apache.org
>>>>>> Betreff: Re: docker and GLV development
>>>>>>
>>>>>> I think there are two components here.
>>>>>> 1) Script to build the dockerized test server. Includes ability to
>>>> start
>>>>>> container manually, externally from tests.
>>>>>> 2) Testcontainers integration which uses the docker image from #1.
>>>>>> Testcontainers could also include executing the script in #1 to build
>>>> the
>>>>>> container.
>>>>>>
>>>>>> Does this sound right to you?
>>>>>>
>>>>>> Robert Dale
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>>>> fh@florian-hockmann.de
>>>>>> wrote:
>>>>>>
>>>>>>>> I do like the idea of being able to test against older versions
>>>>> though.
>>>>>>> That shouldn't be a problem in general as we could use multiple
>>>>>>> different images. See for example this PR that adds Testcontainers to
>>>>>>> JanusGraph to test different Elasticsearch versions:
>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>>>>>>
>>>>>>>> Yes, an on-the-fly container would work
>>>>>>> We could however also use an on-the-fly container with
>>>> Testcontainers.
>>>>>>> We would just need some automation to build the image in the
>>>>> background.
>>>>>>> What if we would build an image during the usual Maven build for
>>>>>>> testing and give it a descriptive tag like "testing". Then, we could
>>>>>>> use that tag with Testcontainers in addition to tags for already
>>>>>> released versions.
>>>>>>> Testcontainers-Java can apparently even build images by itself. So,
>>>>>>> that could also be a possibility to get an image with the latest
>>>>>>> Gremlin Server with the current (unreleased) changes.
>>>>>>>
>>>>>>> In general, I would really like to get away from a situation where we
>>>>>>> have to start some script in the background before tests can be
>>>>> executed.
>>>>>>> @Robert:
>>>>>>>> What is the test configuration/data for GLVs?  Could you list the
>>>>>>> paths/files?
>>>>>>>
>>>>>>> For Gremlin.Net for example, you can find the automation happening in
>>>>>>> the pom.xml of the testing part:
>>>>>>>
>>>>>>>
>>>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>>>>>>
>>>>>>> It calls the test-server-start.groovy script before the tests are
>>>>>>> executed and afterwards test-server-stop.groovy script to stop the
>>>>>> server again.
>>>>>>> This has the obvious downside that the tests can only be executed
>>>> from
>>>>>>> the Maven build which isn't the build tool .NET developers would use
>>>>>>> and also not the tool the IDE will use (the situation is basically
>>>> the
>>>>>>> same for other GLVs).
>>>>>>>
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>> Betreff: Re: docker and GLV development
>>>>>>>
>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>>>>>>
>>>>>>> What is the test configuration/data for GLVs?  Could you list the
>>>>>>> paths/files?
>>>>>>>
>>>>>>> Robert Dale
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>>>>>> <sp...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, an on-the-fly container would work assuming it used the test
>>>>>>>> configuration/data for GLVs. That's why i suggested the command as
>>>> I
>>>>>>>> did because it was similar to docker/build.sh.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
>>>>> wrote:
>>>>>>>>> Do you want to create a docker container on-the-fly using the
>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>>>>>>>>>
>>>>>>>>> Robert Dale
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>>>>>>>> <sp...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> testcontainers seems neat as it gives some good integration and
>>>>>>>>> automation,
>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>>>>>>>>>> There are times when changes are occurring on both client and
>>>>>>>>>> server so a local up-to-the-moment image would be necessary. I
>>>>>>>>>> also think we might want
>>>>>>>> to
>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>>>>>>>>>> more
>>>>>>>>> clearly
>>>>>>>>>> how those play into projects (there is a bit of discussion
>>>> going
>>>>>>>>>> on
>>>>>>>> with
>>>>>>>>>> that now). I do like the idea of being able to test against
>>>>>>>>>> older
>>>>>>>>> versions
>>>>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>>>>>>> fh@florian-hockmann.de
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>>>> tests
>>>>>>>>>>> for Gremlin.Net from my IDE.
>>>>>>>>>>>
>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>>>>>>>>>> problems
>>>>>>>>> like
>>>>>>>>>>> this:
>>>>>>>>>>> https://www.testcontainers.org/
>>>>>>>>>>>
>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>>>>>>>> container
>>>>>>>>> in
>>>>>>>>>>> your tests for whatever external dependencies you need.
>>>>>>>>>>> Databases
>>>>>>>> are a
>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>>>>>>>>>> Gremlin
>>>>>>>>> Server
>>>>>>>>>>> would be such a container.
>>>>>>>>>>> Testcontainers then uses the ability of the testing framework
>>>>>> (e.g.
>>>>>>>>>> JUnit)
>>>>>>>>>>> to start the Docker container for the tests and waits until
>>>>>>>>>>> they are
>>>>>>>>>> fully
>>>>>>>>>>> started (how they are started can be configured for each
>>>>>>>>>>> container differently). After the tests have completed, it
>>>>>>>>>>> also stops the
>>>>>>>>>> containers
>>>>>>>>>>> again.
>>>>>>>>>>> That way, you don't have to manage the containers manually
>>>>>> anymore.
>>>>>>>>>>> Everything is done in the background by the IDE / the build
>>>>> tool.
>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>>>>>>>>>>> also have libraries for different languages, including the
>>>>>>>>>>> ones for which we
>>>>>>>> have
>>>>>>>>>>> GLVs:
>>>>>>>>>>>
>>>>>>>>>>> testcontainers-python:
>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>> <
>>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>>>>>>> s-
>>>>>>>> dotnet
>>>>>>>>>>> :
>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>>>>>>>>>> testcontainers-node
>>>>>>>>>>> <
>>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>>>>>>> s-
>>>>>>>> node
>>>>>>>>>>> :
>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>>>>>>>>>>
>>>>>>>>>>> Testcontainers-java is the most advanced library of these of
>>>>>>> course.
>>>>>>>>> The
>>>>>>>>>>> others still lack some functionality, but I guess that we
>>>>>>>>>>> could also
>>>>>>>>> just
>>>>>>>>>>> contribute to those libraries if an important feature is
>>>>>>>>>>> missing for
>>>>>>>>> us.
>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>> unfortunately can only use images that are available in some
>>>>>>>> registry.
>>>>>>>>>> So,
>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>>>>>>>>>> DockerHub
>>>>>>>> to
>>>>>>>>>> use
>>>>>>>>>>> it in our tests. That means that we could either only test
>>>> the
>>>>>>>>>>> GLVs
>>>>>>>>>> against
>>>>>>>>>>> already released versions or we could publish something like
>>>>>>>>>>> nightly
>>>>>>>>>> build
>>>>>>>>>>> images of Gremlin Server.
>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>>>>>>>>>> able to
>>>>>>>> use
>>>>>>>>>>> images that are only available locally (testcontainers-java
>>>>>>>>>>> already
>>>>>>>> has
>>>>>>>>>>> that capability), but now that I think more about this, I
>>>>>>>>>>> think that
>>>>>>>> it
>>>>>>>>>>> makes actually more sense to use images that are available in
>>>>>>>>>>> the
>>>>>>>>>> registry
>>>>>>>>>>> as every developer would otherwise have to build test images
>>>>>>>>>>> first
>>>>>>>>>> locally
>>>>>>>>>>> in order to use them for tests.
>>>>>>>>>>>
>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>>>>>>>>>>> see
>>>>>>>>> whether
>>>>>>>>>>> you agree that it's a good solution to our problem and then
>>>> we
>>>>>>>>>>> can
>>>>>>>> see
>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>>>> or
>>>>>>>>>>> how we
>>>>>>>>>> want
>>>>>>>>>>> to handle that.
>>>>>>>>>>>
>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>>>> Betreff: docker and GLV development
>>>>>>>>>>>
>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>>>>>>>>>> useful for helping to streamline GLV development? I'd love to
>>>>>>>>>>> be able to do
>>>>>>>>>> something
>>>>>>>>>>> like:
>>>>>>>>>>>
>>>>>>>>>>> docker/gremlin-server.sh -test
>>>>>>>>>>>
>>>>>>>>>>> which would start Gremlin Server with our standard test
>>>>>>>>>>> configuration (currently built into the maven tool chain, but
>>>>>>>>>>> I think that could be extracted without too much difficulty).
>>>>>>>>>>> That way, we could easily
>>>>>>>> open
>>>>>>>>> up
>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
>>>>>>>>>>> there,
>>>>>>>>> use
>>>>>>>>>>> the debugger, etc.
>>>>>>>>>>>
>>>>>>>>>>> Sound useful? and if so, is that something that is
>>>>>>>>>>> easy/possible? and
>>>>>>>>>> then
>>>>>>>>>>> further....any volunteers with the know-how to get that setup
>>>>>>>>>>> (i'm
>>>>>>>>> happy
>>>>>>>>>> to
>>>>>>>>>>> assist however i can)?
>>>>>>>>>>>
>>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
>>>>>>>>> development
>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>>>> mind
>>>>>>>>>>> using
>>>>>>>>> the
>>>>>>>>>>> maven integration but i feel like i could be more efficient
>>>>>>>>>>> with a
>>>>>>>> more
>>>>>>>>>>> streamlined setup.
>>>>>>>>>>>
>>>>>>>>>>>

Re: docker and GLV development

[Stephen Mallette <sp...@gmail.com>]

i tried adding that -p thing to the run command in the script, but i still
couldn't connect to it with Gremlin Console

On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> Did you start the container with port mapping (e.g., docker run -p
> 45940:45940 [image-name]) or how are you trying to connect to the
> container?
>
> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
> > The server starts but something seems messed up with the ports. It's
> like I
> > can't connect to the 45940 port from the host. I tried a couple things I
> > saw on stackoverflow but nothing i tried seemed to fix it. it's probably
> > something simple...any ideas?
> >
> > On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
> >
> >> I made a first attempt.  Let me know what you think.
> >> https://github.com/apache/tinkerpop/pull/1075
> >>
> >> Robert Dale
> >>
> >>
> >> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <spmallette@gmail.com
> >
> >> wrote:
> >>
> >>>>  If we agree on Testcontainers in general, then I can give this a try
> >> and
> >>> use Testcontainers in one of our Java projects that need Gremlin Server
> >> for
> >>> testing (gremlin-driver maybe?).
> >>>
> >>> The java integration tests for Gremlin Server flip a lot of
> configuration
> >>> bits per test. I don't know how that works with testcontainers and if
> it
> >>> will make our testing there any better. There are also a number of
> tests
> >>> that stop/start the server under different conditions - not sure how
> that
> >>> works with testcontainers either. Finally, wouldn't we expect adding
> >> docker
> >>> to the mix to slow down our integration test runs for java?
> >>>
> >>> I don't want to seem like i'm completely against testcontainers btw. I
> >> just
> >>> wonder if a script doesn't give us a 100% better experience than what
> we
> >>> have now (which is nothing). If so, and the script is easy and
> >> zero-impact
> >>> without new dependencies, then I think I like that approach. Also, It
> >> would
> >>> be nice to have the script for other sorts of manual testing, such as
> >>> testing Gremlin Console with Gremlin Server. If we'd like to do
> >>> testcontainers for GLVs I suppose that can be done too, but for my
> >>> purposes, since I was the one who asked for help with this initially,
> >>> testcontainers doesn't sound like something that is an absolute
> >> requirement
> >>> for my needs.
> >>>
> >>> thanks robert/florian for all the discussion and help - appreciated.
> >>>
> >>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
> >> fh@florian-hockmann.de>
> >>> wrote:
> >>>
> >>>> Yes, except for the detail that Testcontainers could make the script
> >>>> obsolete if it is also used to build the image by itself. That would
> >> have
> >>>> the advantage that developers wouldn't have to execute the script to
> >>> build
> >>>> the image. (Executing the script with Testcontainers could also be a
> >>>> possibility, but I'm not sure how good that works.)
> >>>>
> >>>> If we agree on Testcontainers in general, then I can give this a try
> >> and
> >>>> use Testcontainers in one of our Java projects that need Gremlin
> Server
> >>> for
> >>>> testing (gremlin-driver maybe?).
> >>>> With such a PR it would probably also be easier for others to see what
> >>>> advantages / disadvantages using Testcontainers would have.
> >>>>
> >>>> I think using a Java project first makes most sense as we can use all
> >>>> functionality Testcontainers has to offer there. The downside is of
> >>> course
> >>>> that we might not have all the functionality afterwards for other
> >>> languages
> >>>> and then we might need to extend the Testcontainers libraries for the
> >>>> different languages. (My assumption here is that we won't need that
> >> much
> >>>> functionality so it should be limited to 1 or 2 missing features.)
> >>>> Alternatively, we could also start with a GLV where the Testcontainers
> >>>> library doesn't have the same functionality which probably means that
> >> we
> >>>> need a workaround to get an image of our current Gremlin Server
> version
> >>> for
> >>>> testing.
> >>>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Robert Dale <ro...@gmail.com>
> >>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
> >>>> An: dev@tinkerpop.apache.org
> >>>> Betreff: Re: docker and GLV development
> >>>>
> >>>> I think there are two components here.
> >>>> 1) Script to build the dockerized test server. Includes ability to
> >> start
> >>>> container manually, externally from tests.
> >>>> 2) Testcontainers integration which uses the docker image from #1.
> >>>> Testcontainers could also include executing the script in #1 to build
> >> the
> >>>> container.
> >>>>
> >>>> Does this sound right to you?
> >>>>
> >>>> Robert Dale
> >>>>
> >>>>
> >>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
> >> fh@florian-hockmann.de
> >>>> wrote:
> >>>>
> >>>>>> I do like the idea of being able to test against older versions
> >>> though.
> >>>>> That shouldn't be a problem in general as we could use multiple
> >>>>> different images. See for example this PR that adds Testcontainers to
> >>>>> JanusGraph to test different Elasticsearch versions:
> >>>>> https://github.com/JanusGraph/janusgraph/pull/1409
> >>>>>
> >>>>>> Yes, an on-the-fly container would work
> >>>>> We could however also use an on-the-fly container with
> >> Testcontainers.
> >>>>> We would just need some automation to build the image in the
> >>> background.
> >>>>> What if we would build an image during the usual Maven build for
> >>>>> testing and give it a descriptive tag like "testing". Then, we could
> >>>>> use that tag with Testcontainers in addition to tags for already
> >>>> released versions.
> >>>>> Testcontainers-Java can apparently even build images by itself. So,
> >>>>> that could also be a possibility to get an image with the latest
> >>>>> Gremlin Server with the current (unreleased) changes.
> >>>>>
> >>>>> In general, I would really like to get away from a situation where we
> >>>>> have to start some script in the background before tests can be
> >>> executed.
> >>>>> @Robert:
> >>>>>> What is the test configuration/data for GLVs?  Could you list the
> >>>>> paths/files?
> >>>>>
> >>>>> For Gremlin.Net for example, you can find the automation happening in
> >>>>> the pom.xml of the testing part:
> >>>>>
> >>>>>
> >> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> >>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> >>>>>
> >>>>> It calls the test-server-start.groovy script before the tests are
> >>>>> executed and afterwards test-server-stop.groovy script to stop the
> >>>> server again.
> >>>>> This has the obvious downside that the tests can only be executed
> >> from
> >>>>> the Maven build which isn't the build tool .NET developers would use
> >>>>> and also not the tool the IDE will use (the situation is basically
> >> the
> >>>>> same for other GLVs).
> >>>>>
> >>>>> -----Ursprüngliche Nachricht-----
> >>>>> Von: Robert Dale <ro...@gmail.com>
> >>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
> >>>>> An: dev@tinkerpop.apache.org
> >>>>> Betreff: Re: docker and GLV development
> >>>>>
> >>>>> I'm surprised Daniel doesn't have this done already ;-)
> >>>>>
> >>>>> What is the test configuration/data for GLVs?  Could you list the
> >>>>> paths/files?
> >>>>>
> >>>>> Robert Dale
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> >>>>> <sp...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Yes, an on-the-fly container would work assuming it used the test
> >>>>>> configuration/data for GLVs. That's why i suggested the command as
> >> I
> >>>>>> did because it was similar to docker/build.sh.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> >>> wrote:
> >>>>>>> Do you want to create a docker container on-the-fly using the
> >>>>>>> current working copy kind of like we do for docker/build.sh?
> >>>>>>>
> >>>>>>> Robert Dale
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> >>>>>>> <sp...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> testcontainers seems neat as it gives some good integration and
> >>>>>>> automation,
> >>>>>>>> however reliance on nightly snapshots doesn't sound so great.
> >>>>>>>> There are times when changes are occurring on both client and
> >>>>>>>> server so a local up-to-the-moment image would be necessary. I
> >>>>>>>> also think we might want
> >>>>>> to
> >>>>>>>> avoid nightly snapshots of anything until the ASF figures out
> >>>>>>>> more
> >>>>>>> clearly
> >>>>>>>> how those play into projects (there is a bit of discussion
> >> going
> >>>>>>>> on
> >>>>>> with
> >>>>>>>> that now). I do like the idea of being able to test against
> >>>>>>>> older
> >>>>>>> versions
> >>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> >>>>>> fh@florian-hockmann.de
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Fantastic idea, I often missed the ability to just start
> >> tests
> >>>>>>>>> for Gremlin.Net from my IDE.
> >>>>>>>>>
> >>>>>>>>> Luckily for us, there is a project that is aimed exactly at
> >>>>>>>>> problems
> >>>>>>> like
> >>>>>>>>> this:
> >>>>>>>>> https://www.testcontainers.org/
> >>>>>>>>>
> >>>>>>>>> The basic idea of Testcontainers is that you define a Docker
> >>>>>> container
> >>>>>>> in
> >>>>>>>>> your tests for whatever external dependencies you need.
> >>>>>>>>> Databases
> >>>>>> are a
> >>>>>>>>> frequent example where Testcontainers makes sense. For us,
> >>>>>>>>> Gremlin
> >>>>>>> Server
> >>>>>>>>> would be such a container.
> >>>>>>>>> Testcontainers then uses the ability of the testing framework
> >>>> (e.g.
> >>>>>>>> JUnit)
> >>>>>>>>> to start the Docker container for the tests and waits until
> >>>>>>>>> they are
> >>>>>>>> fully
> >>>>>>>>> started (how they are started can be configured for each
> >>>>>>>>> container differently). After the tests have completed, it
> >>>>>>>>> also stops the
> >>>>>>>> containers
> >>>>>>>>> again.
> >>>>>>>>> That way, you don't have to manage the containers manually
> >>>> anymore.
> >>>>>>>>> Everything is done in the background by the IDE / the build
> >>> tool.
> >>>>>>>>> Testcontainers was originally a Java library, but they now
> >>>>>>>>> also have libraries for different languages, including the
> >>>>>>>>> ones for which we
> >>>>>> have
> >>>>>>>>> GLVs:
> >>>>>>>>>
> >>>>>>>>> testcontainers-python:
> >>>>>>>>> https://github.com/testcontainers/testcontainers-python
> >>>>>>>>> testcontainers-dotnet
> >>>>>>>>> <
> >> https://github.com/testcontainers/testcontainers-pythontestcontainer
> >>>>>> s-
> >>>>>> dotnet
> >>>>>>>>> :
> >>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
> >>>>>>>>> testcontainers-node
> >>>>>>>>> <
> >> https://github.com/testcontainers/testcontainers-dotnettestcontainer
> >>>>>> s-
> >>>>>> node
> >>>>>>>>> :
> >>>>>>>>> https://github.com/testcontainers/testcontainers-node
> >>>>>>>>>
> >>>>>>>>> Testcontainers-java is the most advanced library of these of
> >>>>> course.
> >>>>>>> The
> >>>>>>>>> others still lack some functionality, but I guess that we
> >>>>>>>>> could also
> >>>>>>> just
> >>>>>>>>> contribute to those libraries if an important feature is
> >>>>>>>>> missing for
> >>>>>>> us.
> >>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
> >>>>>>> testcontainers-dotnet
> >>>>>>>>> unfortunately can only use images that are available in some
> >>>>>> registry.
> >>>>>>>> So,
> >>>>>>>>> we would have to push a Docker Image of Gremlin Server to
> >>>>>>>>> DockerHub
> >>>>>> to
> >>>>>>>> use
> >>>>>>>>> it in our tests. That means that we could either only test
> >> the
> >>>>>>>>> GLVs
> >>>>>>>> against
> >>>>>>>>> already released versions or we could publish something like
> >>>>>>>>> nightly
> >>>>>>>> build
> >>>>>>>>> images of Gremlin Server.
> >>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
> >>>>>>>>> able to
> >>>>>> use
> >>>>>>>>> images that are only available locally (testcontainers-java
> >>>>>>>>> already
> >>>>>> has
> >>>>>>>>> that capability), but now that I think more about this, I
> >>>>>>>>> think that
> >>>>>> it
> >>>>>>>>> makes actually more sense to use images that are available in
> >>>>>>>>> the
> >>>>>>>> registry
> >>>>>>>>> as every developer would otherwise have to build test images
> >>>>>>>>> first
> >>>>>>>> locally
> >>>>>>>>> in order to use them for tests.
> >>>>>>>>>
> >>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
> >>>>>>>>> see
> >>>>>>> whether
> >>>>>>>>> you agree that it's a good solution to our problem and then
> >> we
> >>>>>>>>> can
> >>>>>> see
> >>>>>>>>> whether we want to publish nightly builds for the GLV tests
> >> or
> >>>>>>>>> how we
> >>>>>>>> want
> >>>>>>>>> to handle that.
> >>>>>>>>>
> >>>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
> >>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
> >>>>>>>>> An: dev@tinkerpop.apache.org
> >>>>>>>>> Betreff: docker and GLV development
> >>>>>>>>>
> >>>>>>>>> Anyone have any idea how to make it so that docker can be
> >>>>>>>>> useful for helping to streamline GLV development? I'd love to
> >>>>>>>>> be able to do
> >>>>>>>> something
> >>>>>>>>> like:
> >>>>>>>>>
> >>>>>>>>> docker/gremlin-server.sh -test
> >>>>>>>>>
> >>>>>>>>> which would start Gremlin Server with our standard test
> >>>>>>>>> configuration (currently built into the maven tool chain, but
> >>>>>>>>> I think that could be extracted without too much difficulty).
> >>>>>>>>> That way, we could easily
> >>>>>> open
> >>>>>>> up
> >>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
> >>>>>>>>> there,
> >>>>>>> use
> >>>>>>>>> the debugger, etc.
> >>>>>>>>>
> >>>>>>>>> Sound useful? and if so, is that something that is
> >>>>>>>>> easy/possible? and
> >>>>>>>> then
> >>>>>>>>> further....any volunteers with the know-how to get that setup
> >>>>>>>>> (i'm
> >>>>>>> happy
> >>>>>>>> to
> >>>>>>>>> assist however i can)?
> >>>>>>>>>
> >>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
> >>>>>>> development
> >>>>>>>>> and docker is a waste of time, please clue me in. i don't
> >> mind
> >>>>>>>>> using
> >>>>>>> the
> >>>>>>>>> maven integration but i feel like i could be more efficient
> >>>>>>>>> with a
> >>>>>> more
> >>>>>>>>> streamlined setup.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>>
>

Re: docker and GLV development

[Florian Hockmann <fh...@florian-hockmann.de>]

Did you start the container with port mapping (e.g., docker run -p
45940:45940 [image-name]) or how are you trying to connect to the container?

Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
> The server starts but something seems messed up with the ports. It's like I
> can't connect to the 45940 port from the host. I tried a couple things I
> saw on stackoverflow but nothing i tried seemed to fix it. it's probably
> something simple...any ideas?
>
> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
>
>> I made a first attempt.  Let me know what you think.
>> https://github.com/apache/tinkerpop/pull/1075
>>
>> Robert Dale
>>
>>
>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>>>>  If we agree on Testcontainers in general, then I can give this a try
>> and
>>> use Testcontainers in one of our Java projects that need Gremlin Server
>> for
>>> testing (gremlin-driver maybe?).
>>>
>>> The java integration tests for Gremlin Server flip a lot of configuration
>>> bits per test. I don't know how that works with testcontainers and if it
>>> will make our testing there any better. There are also a number of tests
>>> that stop/start the server under different conditions - not sure how that
>>> works with testcontainers either. Finally, wouldn't we expect adding
>> docker
>>> to the mix to slow down our integration test runs for java?
>>>
>>> I don't want to seem like i'm completely against testcontainers btw. I
>> just
>>> wonder if a script doesn't give us a 100% better experience than what we
>>> have now (which is nothing). If so, and the script is easy and
>> zero-impact
>>> without new dependencies, then I think I like that approach. Also, It
>> would
>>> be nice to have the script for other sorts of manual testing, such as
>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>> testcontainers for GLVs I suppose that can be done too, but for my
>>> purposes, since I was the one who asked for help with this initially,
>>> testcontainers doesn't sound like something that is an absolute
>> requirement
>>> for my needs.
>>>
>>> thanks robert/florian for all the discussion and help - appreciated.
>>>
>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>> fh@florian-hockmann.de>
>>> wrote:
>>>
>>>> Yes, except for the detail that Testcontainers could make the script
>>>> obsolete if it is also used to build the image by itself. That would
>> have
>>>> the advantage that developers wouldn't have to execute the script to
>>> build
>>>> the image. (Executing the script with Testcontainers could also be a
>>>> possibility, but I'm not sure how good that works.)
>>>>
>>>> If we agree on Testcontainers in general, then I can give this a try
>> and
>>>> use Testcontainers in one of our Java projects that need Gremlin Server
>>> for
>>>> testing (gremlin-driver maybe?).
>>>> With such a PR it would probably also be easier for others to see what
>>>> advantages / disadvantages using Testcontainers would have.
>>>>
>>>> I think using a Java project first makes most sense as we can use all
>>>> functionality Testcontainers has to offer there. The downside is of
>>> course
>>>> that we might not have all the functionality afterwards for other
>>> languages
>>>> and then we might need to extend the Testcontainers libraries for the
>>>> different languages. (My assumption here is that we won't need that
>> much
>>>> functionality so it should be limited to 1 or 2 missing features.)
>>>> Alternatively, we could also start with a GLV where the Testcontainers
>>>> library doesn't have the same functionality which probably means that
>> we
>>>> need a workaround to get an image of our current Gremlin Server version
>>> for
>>>> testing.
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Robert Dale <ro...@gmail.com>
>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>>> An: dev@tinkerpop.apache.org
>>>> Betreff: Re: docker and GLV development
>>>>
>>>> I think there are two components here.
>>>> 1) Script to build the dockerized test server. Includes ability to
>> start
>>>> container manually, externally from tests.
>>>> 2) Testcontainers integration which uses the docker image from #1.
>>>> Testcontainers could also include executing the script in #1 to build
>> the
>>>> container.
>>>>
>>>> Does this sound right to you?
>>>>
>>>> Robert Dale
>>>>
>>>>
>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>> fh@florian-hockmann.de
>>>> wrote:
>>>>
>>>>>> I do like the idea of being able to test against older versions
>>> though.
>>>>> That shouldn't be a problem in general as we could use multiple
>>>>> different images. See for example this PR that adds Testcontainers to
>>>>> JanusGraph to test different Elasticsearch versions:
>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>>>>
>>>>>> Yes, an on-the-fly container would work
>>>>> We could however also use an on-the-fly container with
>> Testcontainers.
>>>>> We would just need some automation to build the image in the
>>> background.
>>>>> What if we would build an image during the usual Maven build for
>>>>> testing and give it a descriptive tag like "testing". Then, we could
>>>>> use that tag with Testcontainers in addition to tags for already
>>>> released versions.
>>>>> Testcontainers-Java can apparently even build images by itself. So,
>>>>> that could also be a possibility to get an image with the latest
>>>>> Gremlin Server with the current (unreleased) changes.
>>>>>
>>>>> In general, I would really like to get away from a situation where we
>>>>> have to start some script in the background before tests can be
>>> executed.
>>>>> @Robert:
>>>>>> What is the test configuration/data for GLVs?  Could you list the
>>>>> paths/files?
>>>>>
>>>>> For Gremlin.Net for example, you can find the automation happening in
>>>>> the pom.xml of the testing part:
>>>>>
>>>>>
>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>>>>
>>>>> It calls the test-server-start.groovy script before the tests are
>>>>> executed and afterwards test-server-stop.groovy script to stop the
>>>> server again.
>>>>> This has the obvious downside that the tests can only be executed
>> from
>>>>> the Maven build which isn't the build tool .NET developers would use
>>>>> and also not the tool the IDE will use (the situation is basically
>> the
>>>>> same for other GLVs).
>>>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>>>> An: dev@tinkerpop.apache.org
>>>>> Betreff: Re: docker and GLV development
>>>>>
>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>>>>
>>>>> What is the test configuration/data for GLVs?  Could you list the
>>>>> paths/files?
>>>>>
>>>>> Robert Dale
>>>>>
>>>>>
>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>>>> <sp...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, an on-the-fly container would work assuming it used the test
>>>>>> configuration/data for GLVs. That's why i suggested the command as
>> I
>>>>>> did because it was similar to docker/build.sh.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
>>> wrote:
>>>>>>> Do you want to create a docker container on-the-fly using the
>>>>>>> current working copy kind of like we do for docker/build.sh?
>>>>>>>
>>>>>>> Robert Dale
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>>>>>> <sp...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> testcontainers seems neat as it gives some good integration and
>>>>>>> automation,
>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>>>>>>>> There are times when changes are occurring on both client and
>>>>>>>> server so a local up-to-the-moment image would be necessary. I
>>>>>>>> also think we might want
>>>>>> to
>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>>>>>>>> more
>>>>>>> clearly
>>>>>>>> how those play into projects (there is a bit of discussion
>> going
>>>>>>>> on
>>>>>> with
>>>>>>>> that now). I do like the idea of being able to test against
>>>>>>>> older
>>>>>>> versions
>>>>>>>> though.  Ensuring backward compatibility.is a nice thought.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>>>>> fh@florian-hockmann.de
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Fantastic idea, I often missed the ability to just start
>> tests
>>>>>>>>> for Gremlin.Net from my IDE.
>>>>>>>>>
>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>>>>>>>> problems
>>>>>>> like
>>>>>>>>> this:
>>>>>>>>> https://www.testcontainers.org/
>>>>>>>>>
>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>>>>>> container
>>>>>>> in
>>>>>>>>> your tests for whatever external dependencies you need.
>>>>>>>>> Databases
>>>>>> are a
>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>>>>>>>> Gremlin
>>>>>>> Server
>>>>>>>>> would be such a container.
>>>>>>>>> Testcontainers then uses the ability of the testing framework
>>>> (e.g.
>>>>>>>> JUnit)
>>>>>>>>> to start the Docker container for the tests and waits until
>>>>>>>>> they are
>>>>>>>> fully
>>>>>>>>> started (how they are started can be configured for each
>>>>>>>>> container differently). After the tests have completed, it
>>>>>>>>> also stops the
>>>>>>>> containers
>>>>>>>>> again.
>>>>>>>>> That way, you don't have to manage the containers manually
>>>> anymore.
>>>>>>>>> Everything is done in the background by the IDE / the build
>>> tool.
>>>>>>>>> Testcontainers was originally a Java library, but they now
>>>>>>>>> also have libraries for different languages, including the
>>>>>>>>> ones for which we
>>>>>> have
>>>>>>>>> GLVs:
>>>>>>>>>
>>>>>>>>> testcontainers-python:
>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>>>>>>>> testcontainers-dotnet
>>>>>>>>> <
>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>>>>> s-
>>>>>> dotnet
>>>>>>>>> :
>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>>>>>>>> testcontainers-node
>>>>>>>>> <
>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>>>>> s-
>>>>>> node
>>>>>>>>> :
>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>>>>>>>>
>>>>>>>>> Testcontainers-java is the most advanced library of these of
>>>>> course.
>>>>>>> The
>>>>>>>>> others still lack some functionality, but I guess that we
>>>>>>>>> could also
>>>>>>> just
>>>>>>>>> contribute to those libraries if an important feature is
>>>>>>>>> missing for
>>>>>>> us.
>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>>>>>> testcontainers-dotnet
>>>>>>>>> unfortunately can only use images that are available in some
>>>>>> registry.
>>>>>>>> So,
>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>>>>>>>> DockerHub
>>>>>> to
>>>>>>>> use
>>>>>>>>> it in our tests. That means that we could either only test
>> the
>>>>>>>>> GLVs
>>>>>>>> against
>>>>>>>>> already released versions or we could publish something like
>>>>>>>>> nightly
>>>>>>>> build
>>>>>>>>> images of Gremlin Server.
>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>>>>>>>> able to
>>>>>> use
>>>>>>>>> images that are only available locally (testcontainers-java
>>>>>>>>> already
>>>>>> has
>>>>>>>>> that capability), but now that I think more about this, I
>>>>>>>>> think that
>>>>>> it
>>>>>>>>> makes actually more sense to use images that are available in
>>>>>>>>> the
>>>>>>>> registry
>>>>>>>>> as every developer would otherwise have to build test images
>>>>>>>>> first
>>>>>>>> locally
>>>>>>>>> in order to use them for tests.
>>>>>>>>>
>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>>>>>>>>> see
>>>>>>> whether
>>>>>>>>> you agree that it's a good solution to our problem and then
>> we
>>>>>>>>> can
>>>>>> see
>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>> or
>>>>>>>>> how we
>>>>>>>> want
>>>>>>>>> to handle that.
>>>>>>>>>
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>> Betreff: docker and GLV development
>>>>>>>>>
>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>>>>>>>> useful for helping to streamline GLV development? I'd love to
>>>>>>>>> be able to do
>>>>>>>> something
>>>>>>>>> like:
>>>>>>>>>
>>>>>>>>> docker/gremlin-server.sh -test
>>>>>>>>>
>>>>>>>>> which would start Gremlin Server with our standard test
>>>>>>>>> configuration (currently built into the maven tool chain, but
>>>>>>>>> I think that could be extracted without too much difficulty).
>>>>>>>>> That way, we could easily
>>>>>> open
>>>>>>> up
>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
>>>>>>>>> there,
>>>>>>> use
>>>>>>>>> the debugger, etc.
>>>>>>>>>
>>>>>>>>> Sound useful? and if so, is that something that is
>>>>>>>>> easy/possible? and
>>>>>>>> then
>>>>>>>>> further....any volunteers with the know-how to get that setup
>>>>>>>>> (i'm
>>>>>>> happy
>>>>>>>> to
>>>>>>>>> assist however i can)?
>>>>>>>>>
>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
>>>>>>> development
>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>> mind
>>>>>>>>> using
>>>>>>> the
>>>>>>>>> maven integration but i feel like i could be more efficient
>>>>>>>>> with a
>>>>>> more
>>>>>>>>> streamlined setup.
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>

Re: docker and GLV development

[Stephen Mallette <sp...@gmail.com>]

The server starts but something seems messed up with the ports. It's like I
can't connect to the 45940 port from the host. I tried a couple things I
saw on stackoverflow but nothing i tried seemed to fix it. it's probably
something simple...any ideas?

On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:

> I made a first attempt.  Let me know what you think.
> https://github.com/apache/tinkerpop/pull/1075
>
> Robert Dale
>
>
> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > >  If we agree on Testcontainers in general, then I can give this a try
> and
> > use Testcontainers in one of our Java projects that need Gremlin Server
> for
> > testing (gremlin-driver maybe?).
> >
> > The java integration tests for Gremlin Server flip a lot of configuration
> > bits per test. I don't know how that works with testcontainers and if it
> > will make our testing there any better. There are also a number of tests
> > that stop/start the server under different conditions - not sure how that
> > works with testcontainers either. Finally, wouldn't we expect adding
> docker
> > to the mix to slow down our integration test runs for java?
> >
> > I don't want to seem like i'm completely against testcontainers btw. I
> just
> > wonder if a script doesn't give us a 100% better experience than what we
> > have now (which is nothing). If so, and the script is easy and
> zero-impact
> > without new dependencies, then I think I like that approach. Also, It
> would
> > be nice to have the script for other sorts of manual testing, such as
> > testing Gremlin Console with Gremlin Server. If we'd like to do
> > testcontainers for GLVs I suppose that can be done too, but for my
> > purposes, since I was the one who asked for help with this initially,
> > testcontainers doesn't sound like something that is an absolute
> requirement
> > for my needs.
> >
> > thanks robert/florian for all the discussion and help - appreciated.
> >
> > On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
> fh@florian-hockmann.de>
> > wrote:
> >
> > > Yes, except for the detail that Testcontainers could make the script
> > > obsolete if it is also used to build the image by itself. That would
> have
> > > the advantage that developers wouldn't have to execute the script to
> > build
> > > the image. (Executing the script with Testcontainers could also be a
> > > possibility, but I'm not sure how good that works.)
> > >
> > > If we agree on Testcontainers in general, then I can give this a try
> and
> > > use Testcontainers in one of our Java projects that need Gremlin Server
> > for
> > > testing (gremlin-driver maybe?).
> > > With such a PR it would probably also be easier for others to see what
> > > advantages / disadvantages using Testcontainers would have.
> > >
> > > I think using a Java project first makes most sense as we can use all
> > > functionality Testcontainers has to offer there. The downside is of
> > course
> > > that we might not have all the functionality afterwards for other
> > languages
> > > and then we might need to extend the Testcontainers libraries for the
> > > different languages. (My assumption here is that we won't need that
> much
> > > functionality so it should be limited to 1 or 2 missing features.)
> > > Alternatively, we could also start with a GLV where the Testcontainers
> > > library doesn't have the same functionality which probably means that
> we
> > > need a workaround to get an image of our current Gremlin Server version
> > for
> > > testing.
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Robert Dale <ro...@gmail.com>
> > > Gesendet: Mittwoch, 27. Februar 2019 15:49
> > > An: dev@tinkerpop.apache.org
> > > Betreff: Re: docker and GLV development
> > >
> > > I think there are two components here.
> > > 1) Script to build the dockerized test server. Includes ability to
> start
> > > container manually, externally from tests.
> > > 2) Testcontainers integration which uses the docker image from #1.
> > > Testcontainers could also include executing the script in #1 to build
> the
> > > container.
> > >
> > > Does this sound right to you?
> > >
> > > Robert Dale
> > >
> > >
> > > On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
> fh@florian-hockmann.de
> > >
> > > wrote:
> > >
> > > > > I do like the idea of being able to test against older versions
> > though.
> > > >
> > > > That shouldn't be a problem in general as we could use multiple
> > > > different images. See for example this PR that adds Testcontainers to
> > > > JanusGraph to test different Elasticsearch versions:
> > > > https://github.com/JanusGraph/janusgraph/pull/1409
> > > >
> > > > > Yes, an on-the-fly container would work
> > > >
> > > > We could however also use an on-the-fly container with
> Testcontainers.
> > > > We would just need some automation to build the image in the
> > background.
> > > > What if we would build an image during the usual Maven build for
> > > > testing and give it a descriptive tag like "testing". Then, we could
> > > > use that tag with Testcontainers in addition to tags for already
> > > released versions.
> > > > Testcontainers-Java can apparently even build images by itself. So,
> > > > that could also be a possibility to get an image with the latest
> > > > Gremlin Server with the current (unreleased) changes.
> > > >
> > > > In general, I would really like to get away from a situation where we
> > > > have to start some script in the background before tests can be
> > executed.
> > > >
> > > > @Robert:
> > > > > What is the test configuration/data for GLVs?  Could you list the
> > > > paths/files?
> > > >
> > > > For Gremlin.Net for example, you can find the automation happening in
> > > > the pom.xml of the testing part:
> > > >
> > > >
> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> > > > 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> > > >
> > > > It calls the test-server-start.groovy script before the tests are
> > > > executed and afterwards test-server-stop.groovy script to stop the
> > > server again.
> > > > This has the obvious downside that the tests can only be executed
> from
> > > > the Maven build which isn't the build tool .NET developers would use
> > > > and also not the tool the IDE will use (the situation is basically
> the
> > > > same for other GLVs).
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Robert Dale <ro...@gmail.com>
> > > > Gesendet: Mittwoch, 27. Februar 2019 13:52
> > > > An: dev@tinkerpop.apache.org
> > > > Betreff: Re: docker and GLV development
> > > >
> > > > I'm surprised Daniel doesn't have this done already ;-)
> > > >
> > > > What is the test configuration/data for GLVs?  Could you list the
> > > > paths/files?
> > > >
> > > > Robert Dale
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> > > > <sp...@gmail.com>
> > > > wrote:
> > > >
> > > > > Yes, an on-the-fly container would work assuming it used the test
> > > > > configuration/data for GLVs. That's why i suggested the command as
> I
> > > > > did because it was similar to docker/build.sh.
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> > wrote:
> > > > >
> > > > > > Do you want to create a docker container on-the-fly using the
> > > > > > current working copy kind of like we do for docker/build.sh?
> > > > > >
> > > > > > Robert Dale
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> > > > > > <sp...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > testcontainers seems neat as it gives some good integration and
> > > > > > automation,
> > > > > > > however reliance on nightly snapshots doesn't sound so great.
> > > > > > > There are times when changes are occurring on both client and
> > > > > > > server so a local up-to-the-moment image would be necessary. I
> > > > > > > also think we might want
> > > > > to
> > > > > > > avoid nightly snapshots of anything until the ASF figures out
> > > > > > > more
> > > > > > clearly
> > > > > > > how those play into projects (there is a bit of discussion
> going
> > > > > > > on
> > > > > with
> > > > > > > that now). I do like the idea of being able to test against
> > > > > > > older
> > > > > > versions
> > > > > > > though.  Ensuring backward compatibility.is a nice thought.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > > > > fh@florian-hockmann.de
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Fantastic idea, I often missed the ability to just start
> tests
> > > > > > > > for Gremlin.Net from my IDE.
> > > > > > > >
> > > > > > > > Luckily for us, there is a project that is aimed exactly at
> > > > > > > > problems
> > > > > > like
> > > > > > > > this:
> > > > > > > > https://www.testcontainers.org/
> > > > > > > >
> > > > > > > > The basic idea of Testcontainers is that you define a Docker
> > > > > container
> > > > > > in
> > > > > > > > your tests for whatever external dependencies you need.
> > > > > > > > Databases
> > > > > are a
> > > > > > > > frequent example where Testcontainers makes sense. For us,
> > > > > > > > Gremlin
> > > > > > Server
> > > > > > > > would be such a container.
> > > > > > > > Testcontainers then uses the ability of the testing framework
> > > (e.g.
> > > > > > > JUnit)
> > > > > > > > to start the Docker container for the tests and waits until
> > > > > > > > they are
> > > > > > > fully
> > > > > > > > started (how they are started can be configured for each
> > > > > > > > container differently). After the tests have completed, it
> > > > > > > > also stops the
> > > > > > > containers
> > > > > > > > again.
> > > > > > > > That way, you don't have to manage the containers manually
> > > anymore.
> > > > > > > > Everything is done in the background by the IDE / the build
> > tool.
> > > > > > > >
> > > > > > > > Testcontainers was originally a Java library, but they now
> > > > > > > > also have libraries for different languages, including the
> > > > > > > > ones for which we
> > > > > have
> > > > > > > > GLVs:
> > > > > > > >
> > > > > > > > testcontainers-python:
> > > > > > > > https://github.com/testcontainers/testcontainers-python
> > > > > > > > testcontainers-dotnet
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> https://github.com/testcontainers/testcontainers-pythontestcontainer
> > > > > s-
> > > > > dotnet
> > > > > > > >:
> > > > > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > > > > testcontainers-node
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> https://github.com/testcontainers/testcontainers-dotnettestcontainer
> > > > > s-
> > > > > node
> > > > > > > >:
> > > > > > > > https://github.com/testcontainers/testcontainers-node
> > > > > > > >
> > > > > > > > Testcontainers-java is the most advanced library of these of
> > > > course.
> > > > > > The
> > > > > > > > others still lack some functionality, but I guess that we
> > > > > > > > could also
> > > > > > just
> > > > > > > > contribute to those libraries if an important feature is
> > > > > > > > missing for
> > > > > > us.
> > > > > > > >
> > > > > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > > > > testcontainers-dotnet
> > > > > > > > unfortunately can only use images that are available in some
> > > > > registry.
> > > > > > > So,
> > > > > > > > we would have to push a Docker Image of Gremlin Server to
> > > > > > > > DockerHub
> > > > > to
> > > > > > > use
> > > > > > > > it in our tests. That means that we could either only test
> the
> > > > > > > > GLVs
> > > > > > > against
> > > > > > > > already released versions or we could publish something like
> > > > > > > > nightly
> > > > > > > build
> > > > > > > > images of Gremlin Server.
> > > > > > > > I originally wanted to wait until testcontainers-dotnet is
> > > > > > > > able to
> > > > > use
> > > > > > > > images that are only available locally (testcontainers-java
> > > > > > > > already
> > > > > has
> > > > > > > > that capability), but now that I think more about this, I
> > > > > > > > think that
> > > > > it
> > > > > > > > makes actually more sense to use images that are available in
> > > > > > > > the
> > > > > > > registry
> > > > > > > > as every developer would otherwise have to build test images
> > > > > > > > first
> > > > > > > locally
> > > > > > > > in order to use them for tests.
> > > > > > > >
> > > > > > > > So, I suggest that you guys take a look at Testcontainers to
> > > > > > > > see
> > > > > > whether
> > > > > > > > you agree that it's a good solution to our problem and then
> we
> > > > > > > > can
> > > > > see
> > > > > > > > whether we want to publish nightly builds for the GLV tests
> or
> > > > > > > > how we
> > > > > > > want
> > > > > > > > to handle that.
> > > > > > > >
> > > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > > > > An: dev@tinkerpop.apache.org
> > > > > > > > Betreff: docker and GLV development
> > > > > > > >
> > > > > > > > Anyone have any idea how to make it so that docker can be
> > > > > > > > useful for helping to streamline GLV development? I'd love to
> > > > > > > > be able to do
> > > > > > > something
> > > > > > > > like:
> > > > > > > >
> > > > > > > > docker/gremlin-server.sh -test
> > > > > > > >
> > > > > > > > which would start Gremlin Server with our standard test
> > > > > > > > configuration (currently built into the maven tool chain, but
> > > > > > > > I think that could be extracted without too much difficulty).
> > > > > > > > That way, we could easily
> > > > > open
> > > > > > up
> > > > > > > > whatever GLV specific IDE we wanted and run tests directly in
> > > > > > > > there,
> > > > > > use
> > > > > > > > the debugger, etc.
> > > > > > > >
> > > > > > > > Sound useful? and if so, is that something that is
> > > > > > > > easy/possible? and
> > > > > > > then
> > > > > > > > further....any volunteers with the know-how to get that setup
> > > > > > > > (i'm
> > > > > > happy
> > > > > > > to
> > > > > > > > assist however i can)?
> > > > > > > >
> > > > > > > > btw, if you have a nicer way to set up an environment for GLV
> > > > > > development
> > > > > > > > and docker is a waste of time, please clue me in. i don't
> mind
> > > > > > > > using
> > > > > > the
> > > > > > > > maven integration but i feel like i could be more efficient
> > > > > > > > with a
> > > > > more
> > > > > > > > streamlined setup.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
>

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

I made a first attempt.  Let me know what you think.
https://github.com/apache/tinkerpop/pull/1075

Robert Dale


On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <sp...@gmail.com>
wrote:

> >  If we agree on Testcontainers in general, then I can give this a try and
> use Testcontainers in one of our Java projects that need Gremlin Server for
> testing (gremlin-driver maybe?).
>
> The java integration tests for Gremlin Server flip a lot of configuration
> bits per test. I don't know how that works with testcontainers and if it
> will make our testing there any better. There are also a number of tests
> that stop/start the server under different conditions - not sure how that
> works with testcontainers either. Finally, wouldn't we expect adding docker
> to the mix to slow down our integration test runs for java?
>
> I don't want to seem like i'm completely against testcontainers btw. I just
> wonder if a script doesn't give us a 100% better experience than what we
> have now (which is nothing). If so, and the script is easy and zero-impact
> without new dependencies, then I think I like that approach. Also, It would
> be nice to have the script for other sorts of manual testing, such as
> testing Gremlin Console with Gremlin Server. If we'd like to do
> testcontainers for GLVs I suppose that can be done too, but for my
> purposes, since I was the one who asked for help with this initially,
> testcontainers doesn't sound like something that is an absolute requirement
> for my needs.
>
> thanks robert/florian for all the discussion and help - appreciated.
>
> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
> > Yes, except for the detail that Testcontainers could make the script
> > obsolete if it is also used to build the image by itself. That would have
> > the advantage that developers wouldn't have to execute the script to
> build
> > the image. (Executing the script with Testcontainers could also be a
> > possibility, but I'm not sure how good that works.)
> >
> > If we agree on Testcontainers in general, then I can give this a try and
> > use Testcontainers in one of our Java projects that need Gremlin Server
> for
> > testing (gremlin-driver maybe?).
> > With such a PR it would probably also be easier for others to see what
> > advantages / disadvantages using Testcontainers would have.
> >
> > I think using a Java project first makes most sense as we can use all
> > functionality Testcontainers has to offer there. The downside is of
> course
> > that we might not have all the functionality afterwards for other
> languages
> > and then we might need to extend the Testcontainers libraries for the
> > different languages. (My assumption here is that we won't need that much
> > functionality so it should be limited to 1 or 2 missing features.)
> > Alternatively, we could also start with a GLV where the Testcontainers
> > library doesn't have the same functionality which probably means that we
> > need a workaround to get an image of our current Gremlin Server version
> for
> > testing.
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Robert Dale <ro...@gmail.com>
> > Gesendet: Mittwoch, 27. Februar 2019 15:49
> > An: dev@tinkerpop.apache.org
> > Betreff: Re: docker and GLV development
> >
> > I think there are two components here.
> > 1) Script to build the dockerized test server. Includes ability to start
> > container manually, externally from tests.
> > 2) Testcontainers integration which uses the docker image from #1.
> > Testcontainers could also include executing the script in #1 to build the
> > container.
> >
> > Does this sound right to you?
> >
> > Robert Dale
> >
> >
> > On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <fh@florian-hockmann.de
> >
> > wrote:
> >
> > > > I do like the idea of being able to test against older versions
> though.
> > >
> > > That shouldn't be a problem in general as we could use multiple
> > > different images. See for example this PR that adds Testcontainers to
> > > JanusGraph to test different Elasticsearch versions:
> > > https://github.com/JanusGraph/janusgraph/pull/1409
> > >
> > > > Yes, an on-the-fly container would work
> > >
> > > We could however also use an on-the-fly container with Testcontainers.
> > > We would just need some automation to build the image in the
> background.
> > > What if we would build an image during the usual Maven build for
> > > testing and give it a descriptive tag like "testing". Then, we could
> > > use that tag with Testcontainers in addition to tags for already
> > released versions.
> > > Testcontainers-Java can apparently even build images by itself. So,
> > > that could also be a possibility to get an image with the latest
> > > Gremlin Server with the current (unreleased) changes.
> > >
> > > In general, I would really like to get away from a situation where we
> > > have to start some script in the background before tests can be
> executed.
> > >
> > > @Robert:
> > > > What is the test configuration/data for GLVs?  Could you list the
> > > paths/files?
> > >
> > > For Gremlin.Net for example, you can find the automation happening in
> > > the pom.xml of the testing part:
> > >
> > > https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> > > 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> > >
> > > It calls the test-server-start.groovy script before the tests are
> > > executed and afterwards test-server-stop.groovy script to stop the
> > server again.
> > > This has the obvious downside that the tests can only be executed from
> > > the Maven build which isn't the build tool .NET developers would use
> > > and also not the tool the IDE will use (the situation is basically the
> > > same for other GLVs).
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Robert Dale <ro...@gmail.com>
> > > Gesendet: Mittwoch, 27. Februar 2019 13:52
> > > An: dev@tinkerpop.apache.org
> > > Betreff: Re: docker and GLV development
> > >
> > > I'm surprised Daniel doesn't have this done already ;-)
> > >
> > > What is the test configuration/data for GLVs?  Could you list the
> > > paths/files?
> > >
> > > Robert Dale
> > >
> > >
> > > On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> > > <sp...@gmail.com>
> > > wrote:
> > >
> > > > Yes, an on-the-fly container would work assuming it used the test
> > > > configuration/data for GLVs. That's why i suggested the command as I
> > > > did because it was similar to docker/build.sh.
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> wrote:
> > > >
> > > > > Do you want to create a docker container on-the-fly using the
> > > > > current working copy kind of like we do for docker/build.sh?
> > > > >
> > > > > Robert Dale
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> > > > > <sp...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > testcontainers seems neat as it gives some good integration and
> > > > > automation,
> > > > > > however reliance on nightly snapshots doesn't sound so great.
> > > > > > There are times when changes are occurring on both client and
> > > > > > server so a local up-to-the-moment image would be necessary. I
> > > > > > also think we might want
> > > > to
> > > > > > avoid nightly snapshots of anything until the ASF figures out
> > > > > > more
> > > > > clearly
> > > > > > how those play into projects (there is a bit of discussion going
> > > > > > on
> > > > with
> > > > > > that now). I do like the idea of being able to test against
> > > > > > older
> > > > > versions
> > > > > > though.  Ensuring backward compatibility.is a nice thought.
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > > > fh@florian-hockmann.de
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Fantastic idea, I often missed the ability to just start tests
> > > > > > > for Gremlin.Net from my IDE.
> > > > > > >
> > > > > > > Luckily for us, there is a project that is aimed exactly at
> > > > > > > problems
> > > > > like
> > > > > > > this:
> > > > > > > https://www.testcontainers.org/
> > > > > > >
> > > > > > > The basic idea of Testcontainers is that you define a Docker
> > > > container
> > > > > in
> > > > > > > your tests for whatever external dependencies you need.
> > > > > > > Databases
> > > > are a
> > > > > > > frequent example where Testcontainers makes sense. For us,
> > > > > > > Gremlin
> > > > > Server
> > > > > > > would be such a container.
> > > > > > > Testcontainers then uses the ability of the testing framework
> > (e.g.
> > > > > > JUnit)
> > > > > > > to start the Docker container for the tests and waits until
> > > > > > > they are
> > > > > > fully
> > > > > > > started (how they are started can be configured for each
> > > > > > > container differently). After the tests have completed, it
> > > > > > > also stops the
> > > > > > containers
> > > > > > > again.
> > > > > > > That way, you don't have to manage the containers manually
> > anymore.
> > > > > > > Everything is done in the background by the IDE / the build
> tool.
> > > > > > >
> > > > > > > Testcontainers was originally a Java library, but they now
> > > > > > > also have libraries for different languages, including the
> > > > > > > ones for which we
> > > > have
> > > > > > > GLVs:
> > > > > > >
> > > > > > > testcontainers-python:
> > > > > > > https://github.com/testcontainers/testcontainers-python
> > > > > > > testcontainers-dotnet
> > > > > > > <
> > > > > >
> > > > >
> > > > https://github.com/testcontainers/testcontainers-pythontestcontainer
> > > > s-
> > > > dotnet
> > > > > > >:
> > > > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > > > testcontainers-node
> > > > > > > <
> > > > > >
> > > > >
> > > > https://github.com/testcontainers/testcontainers-dotnettestcontainer
> > > > s-
> > > > node
> > > > > > >:
> > > > > > > https://github.com/testcontainers/testcontainers-node
> > > > > > >
> > > > > > > Testcontainers-java is the most advanced library of these of
> > > course.
> > > > > The
> > > > > > > others still lack some functionality, but I guess that we
> > > > > > > could also
> > > > > just
> > > > > > > contribute to those libraries if an important feature is
> > > > > > > missing for
> > > > > us.
> > > > > > >
> > > > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > > > testcontainers-dotnet
> > > > > > > unfortunately can only use images that are available in some
> > > > registry.
> > > > > > So,
> > > > > > > we would have to push a Docker Image of Gremlin Server to
> > > > > > > DockerHub
> > > > to
> > > > > > use
> > > > > > > it in our tests. That means that we could either only test the
> > > > > > > GLVs
> > > > > > against
> > > > > > > already released versions or we could publish something like
> > > > > > > nightly
> > > > > > build
> > > > > > > images of Gremlin Server.
> > > > > > > I originally wanted to wait until testcontainers-dotnet is
> > > > > > > able to
> > > > use
> > > > > > > images that are only available locally (testcontainers-java
> > > > > > > already
> > > > has
> > > > > > > that capability), but now that I think more about this, I
> > > > > > > think that
> > > > it
> > > > > > > makes actually more sense to use images that are available in
> > > > > > > the
> > > > > > registry
> > > > > > > as every developer would otherwise have to build test images
> > > > > > > first
> > > > > > locally
> > > > > > > in order to use them for tests.
> > > > > > >
> > > > > > > So, I suggest that you guys take a look at Testcontainers to
> > > > > > > see
> > > > > whether
> > > > > > > you agree that it's a good solution to our problem and then we
> > > > > > > can
> > > > see
> > > > > > > whether we want to publish nightly builds for the GLV tests or
> > > > > > > how we
> > > > > > want
> > > > > > > to handle that.
> > > > > > >
> > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > > > An: dev@tinkerpop.apache.org
> > > > > > > Betreff: docker and GLV development
> > > > > > >
> > > > > > > Anyone have any idea how to make it so that docker can be
> > > > > > > useful for helping to streamline GLV development? I'd love to
> > > > > > > be able to do
> > > > > > something
> > > > > > > like:
> > > > > > >
> > > > > > > docker/gremlin-server.sh -test
> > > > > > >
> > > > > > > which would start Gremlin Server with our standard test
> > > > > > > configuration (currently built into the maven tool chain, but
> > > > > > > I think that could be extracted without too much difficulty).
> > > > > > > That way, we could easily
> > > > open
> > > > > up
> > > > > > > whatever GLV specific IDE we wanted and run tests directly in
> > > > > > > there,
> > > > > use
> > > > > > > the debugger, etc.
> > > > > > >
> > > > > > > Sound useful? and if so, is that something that is
> > > > > > > easy/possible? and
> > > > > > then
> > > > > > > further....any volunteers with the know-how to get that setup
> > > > > > > (i'm
> > > > > happy
> > > > > > to
> > > > > > > assist however i can)?
> > > > > > >
> > > > > > > btw, if you have a nicer way to set up an environment for GLV
> > > > > development
> > > > > > > and docker is a waste of time, please clue me in. i don't mind
> > > > > > > using
> > > > > the
> > > > > > > maven integration but i feel like i could be more efficient
> > > > > > > with a
> > > > more
> > > > > > > streamlined setup.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
>

Re: docker and GLV development

[Stephen Mallette <sp...@gmail.com>]

>  If we agree on Testcontainers in general, then I can give this a try and
use Testcontainers in one of our Java projects that need Gremlin Server for
testing (gremlin-driver maybe?).

The java integration tests for Gremlin Server flip a lot of configuration
bits per test. I don't know how that works with testcontainers and if it
will make our testing there any better. There are also a number of tests
that stop/start the server under different conditions - not sure how that
works with testcontainers either. Finally, wouldn't we expect adding docker
to the mix to slow down our integration test runs for java?

I don't want to seem like i'm completely against testcontainers btw. I just
wonder if a script doesn't give us a 100% better experience than what we
have now (which is nothing). If so, and the script is easy and zero-impact
without new dependencies, then I think I like that approach. Also, It would
be nice to have the script for other sorts of manual testing, such as
testing Gremlin Console with Gremlin Server. If we'd like to do
testcontainers for GLVs I suppose that can be done too, but for my
purposes, since I was the one who asked for help with this initially,
testcontainers doesn't sound like something that is an absolute requirement
for my needs.

thanks robert/florian for all the discussion and help - appreciated.

On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> Yes, except for the detail that Testcontainers could make the script
> obsolete if it is also used to build the image by itself. That would have
> the advantage that developers wouldn't have to execute the script to build
> the image. (Executing the script with Testcontainers could also be a
> possibility, but I'm not sure how good that works.)
>
> If we agree on Testcontainers in general, then I can give this a try and
> use Testcontainers in one of our Java projects that need Gremlin Server for
> testing (gremlin-driver maybe?).
> With such a PR it would probably also be easier for others to see what
> advantages / disadvantages using Testcontainers would have.
>
> I think using a Java project first makes most sense as we can use all
> functionality Testcontainers has to offer there. The downside is of course
> that we might not have all the functionality afterwards for other languages
> and then we might need to extend the Testcontainers libraries for the
> different languages. (My assumption here is that we won't need that much
> functionality so it should be limited to 1 or 2 missing features.)
> Alternatively, we could also start with a GLV where the Testcontainers
> library doesn't have the same functionality which probably means that we
> need a workaround to get an image of our current Gremlin Server version for
> testing.
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Dale <ro...@gmail.com>
> Gesendet: Mittwoch, 27. Februar 2019 15:49
> An: dev@tinkerpop.apache.org
> Betreff: Re: docker and GLV development
>
> I think there are two components here.
> 1) Script to build the dockerized test server. Includes ability to start
> container manually, externally from tests.
> 2) Testcontainers integration which uses the docker image from #1.
> Testcontainers could also include executing the script in #1 to build the
> container.
>
> Does this sound right to you?
>
> Robert Dale
>
>
> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
> > > I do like the idea of being able to test against older versions though.
> >
> > That shouldn't be a problem in general as we could use multiple
> > different images. See for example this PR that adds Testcontainers to
> > JanusGraph to test different Elasticsearch versions:
> > https://github.com/JanusGraph/janusgraph/pull/1409
> >
> > > Yes, an on-the-fly container would work
> >
> > We could however also use an on-the-fly container with Testcontainers.
> > We would just need some automation to build the image in the background.
> > What if we would build an image during the usual Maven build for
> > testing and give it a descriptive tag like "testing". Then, we could
> > use that tag with Testcontainers in addition to tags for already
> released versions.
> > Testcontainers-Java can apparently even build images by itself. So,
> > that could also be a possibility to get an image with the latest
> > Gremlin Server with the current (unreleased) changes.
> >
> > In general, I would really like to get away from a situation where we
> > have to start some script in the background before tests can be executed.
> >
> > @Robert:
> > > What is the test configuration/data for GLVs?  Could you list the
> > paths/files?
> >
> > For Gremlin.Net for example, you can find the automation happening in
> > the pom.xml of the testing part:
> >
> > https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> > 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> >
> > It calls the test-server-start.groovy script before the tests are
> > executed and afterwards test-server-stop.groovy script to stop the
> server again.
> > This has the obvious downside that the tests can only be executed from
> > the Maven build which isn't the build tool .NET developers would use
> > and also not the tool the IDE will use (the situation is basically the
> > same for other GLVs).
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Robert Dale <ro...@gmail.com>
> > Gesendet: Mittwoch, 27. Februar 2019 13:52
> > An: dev@tinkerpop.apache.org
> > Betreff: Re: docker and GLV development
> >
> > I'm surprised Daniel doesn't have this done already ;-)
> >
> > What is the test configuration/data for GLVs?  Could you list the
> > paths/files?
> >
> > Robert Dale
> >
> >
> > On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> > <sp...@gmail.com>
> > wrote:
> >
> > > Yes, an on-the-fly container would work assuming it used the test
> > > configuration/data for GLVs. That's why i suggested the command as I
> > > did because it was similar to docker/build.sh.
> > >
> > >
> > > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com> wrote:
> > >
> > > > Do you want to create a docker container on-the-fly using the
> > > > current working copy kind of like we do for docker/build.sh?
> > > >
> > > > Robert Dale
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> > > > <sp...@gmail.com>
> > > > wrote:
> > > >
> > > > > testcontainers seems neat as it gives some good integration and
> > > > automation,
> > > > > however reliance on nightly snapshots doesn't sound so great.
> > > > > There are times when changes are occurring on both client and
> > > > > server so a local up-to-the-moment image would be necessary. I
> > > > > also think we might want
> > > to
> > > > > avoid nightly snapshots of anything until the ASF figures out
> > > > > more
> > > > clearly
> > > > > how those play into projects (there is a bit of discussion going
> > > > > on
> > > with
> > > > > that now). I do like the idea of being able to test against
> > > > > older
> > > > versions
> > > > > though.  Ensuring backward compatibility.is a nice thought.
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > > fh@florian-hockmann.de
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Fantastic idea, I often missed the ability to just start tests
> > > > > > for Gremlin.Net from my IDE.
> > > > > >
> > > > > > Luckily for us, there is a project that is aimed exactly at
> > > > > > problems
> > > > like
> > > > > > this:
> > > > > > https://www.testcontainers.org/
> > > > > >
> > > > > > The basic idea of Testcontainers is that you define a Docker
> > > container
> > > > in
> > > > > > your tests for whatever external dependencies you need.
> > > > > > Databases
> > > are a
> > > > > > frequent example where Testcontainers makes sense. For us,
> > > > > > Gremlin
> > > > Server
> > > > > > would be such a container.
> > > > > > Testcontainers then uses the ability of the testing framework
> (e.g.
> > > > > JUnit)
> > > > > > to start the Docker container for the tests and waits until
> > > > > > they are
> > > > > fully
> > > > > > started (how they are started can be configured for each
> > > > > > container differently). After the tests have completed, it
> > > > > > also stops the
> > > > > containers
> > > > > > again.
> > > > > > That way, you don't have to manage the containers manually
> anymore.
> > > > > > Everything is done in the background by the IDE / the build tool.
> > > > > >
> > > > > > Testcontainers was originally a Java library, but they now
> > > > > > also have libraries for different languages, including the
> > > > > > ones for which we
> > > have
> > > > > > GLVs:
> > > > > >
> > > > > > testcontainers-python:
> > > > > > https://github.com/testcontainers/testcontainers-python
> > > > > > testcontainers-dotnet
> > > > > > <
> > > > >
> > > >
> > > https://github.com/testcontainers/testcontainers-pythontestcontainer
> > > s-
> > > dotnet
> > > > > >:
> > > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > > testcontainers-node
> > > > > > <
> > > > >
> > > >
> > > https://github.com/testcontainers/testcontainers-dotnettestcontainer
> > > s-
> > > node
> > > > > >:
> > > > > > https://github.com/testcontainers/testcontainers-node
> > > > > >
> > > > > > Testcontainers-java is the most advanced library of these of
> > course.
> > > > The
> > > > > > others still lack some functionality, but I guess that we
> > > > > > could also
> > > > just
> > > > > > contribute to those libraries if an important feature is
> > > > > > missing for
> > > > us.
> > > > > >
> > > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > > testcontainers-dotnet
> > > > > > unfortunately can only use images that are available in some
> > > registry.
> > > > > So,
> > > > > > we would have to push a Docker Image of Gremlin Server to
> > > > > > DockerHub
> > > to
> > > > > use
> > > > > > it in our tests. That means that we could either only test the
> > > > > > GLVs
> > > > > against
> > > > > > already released versions or we could publish something like
> > > > > > nightly
> > > > > build
> > > > > > images of Gremlin Server.
> > > > > > I originally wanted to wait until testcontainers-dotnet is
> > > > > > able to
> > > use
> > > > > > images that are only available locally (testcontainers-java
> > > > > > already
> > > has
> > > > > > that capability), but now that I think more about this, I
> > > > > > think that
> > > it
> > > > > > makes actually more sense to use images that are available in
> > > > > > the
> > > > > registry
> > > > > > as every developer would otherwise have to build test images
> > > > > > first
> > > > > locally
> > > > > > in order to use them for tests.
> > > > > >
> > > > > > So, I suggest that you guys take a look at Testcontainers to
> > > > > > see
> > > > whether
> > > > > > you agree that it's a good solution to our problem and then we
> > > > > > can
> > > see
> > > > > > whether we want to publish nightly builds for the GLV tests or
> > > > > > how we
> > > > > want
> > > > > > to handle that.
> > > > > >
> > > > > > -----Ursprüngliche Nachricht-----
> > > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > > An: dev@tinkerpop.apache.org
> > > > > > Betreff: docker and GLV development
> > > > > >
> > > > > > Anyone have any idea how to make it so that docker can be
> > > > > > useful for helping to streamline GLV development? I'd love to
> > > > > > be able to do
> > > > > something
> > > > > > like:
> > > > > >
> > > > > > docker/gremlin-server.sh -test
> > > > > >
> > > > > > which would start Gremlin Server with our standard test
> > > > > > configuration (currently built into the maven tool chain, but
> > > > > > I think that could be extracted without too much difficulty).
> > > > > > That way, we could easily
> > > open
> > > > up
> > > > > > whatever GLV specific IDE we wanted and run tests directly in
> > > > > > there,
> > > > use
> > > > > > the debugger, etc.
> > > > > >
> > > > > > Sound useful? and if so, is that something that is
> > > > > > easy/possible? and
> > > > > then
> > > > > > further....any volunteers with the know-how to get that setup
> > > > > > (i'm
> > > > happy
> > > > > to
> > > > > > assist however i can)?
> > > > > >
> > > > > > btw, if you have a nicer way to set up an environment for GLV
> > > > development
> > > > > > and docker is a waste of time, please clue me in. i don't mind
> > > > > > using
> > > > the
> > > > > > maven integration but i feel like i could be more efficient
> > > > > > with a
> > > more
> > > > > > streamlined setup.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>
>

AW: docker and GLV development

[Florian Hockmann <fh...@florian-hockmann.de>]

Yes, except for the detail that Testcontainers could make the script obsolete if it is also used to build the image by itself. That would have the advantage that developers wouldn't have to execute the script to build the image. (Executing the script with Testcontainers could also be a possibility, but I'm not sure how good that works.)

If we agree on Testcontainers in general, then I can give this a try and use Testcontainers in one of our Java projects that need Gremlin Server for testing (gremlin-driver maybe?).
With such a PR it would probably also be easier for others to see what advantages / disadvantages using Testcontainers would have.

I think using a Java project first makes most sense as we can use all functionality Testcontainers has to offer there. The downside is of course that we might not have all the functionality afterwards for other languages and then we might need to extend the Testcontainers libraries for the different languages. (My assumption here is that we won't need that much functionality so it should be limited to 1 or 2 missing features.)
Alternatively, we could also start with a GLV where the Testcontainers library doesn't have the same functionality which probably means that we need a workaround to get an image of our current Gremlin Server version for testing.

-----Ursprüngliche Nachricht-----
Von: Robert Dale <ro...@gmail.com> 
Gesendet: Mittwoch, 27. Februar 2019 15:49
An: dev@tinkerpop.apache.org
Betreff: Re: docker and GLV development

I think there are two components here.
1) Script to build the dockerized test server. Includes ability to start container manually, externally from tests.
2) Testcontainers integration which uses the docker image from #1.
Testcontainers could also include executing the script in #1 to build the container.

Does this sound right to you?

Robert Dale


On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> > I do like the idea of being able to test against older versions though.
>
> That shouldn't be a problem in general as we could use multiple 
> different images. See for example this PR that adds Testcontainers to 
> JanusGraph to test different Elasticsearch versions:
> https://github.com/JanusGraph/janusgraph/pull/1409
>
> > Yes, an on-the-fly container would work
>
> We could however also use an on-the-fly container with Testcontainers. 
> We would just need some automation to build the image in the background.
> What if we would build an image during the usual Maven build for 
> testing and give it a descriptive tag like "testing". Then, we could 
> use that tag with Testcontainers in addition to tags for already released versions.
> Testcontainers-Java can apparently even build images by itself. So, 
> that could also be a possibility to get an image with the latest 
> Gremlin Server with the current (unreleased) changes.
>
> In general, I would really like to get away from a situation where we 
> have to start some script in the background before tests can be executed.
>
> @Robert:
> > What is the test configuration/data for GLVs?  Could you list the
> paths/files?
>
> For Gremlin.Net for example, you can find the automation happening in 
> the pom.xml of the testing part:
>
> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>
> It calls the test-server-start.groovy script before the tests are 
> executed and afterwards test-server-stop.groovy script to stop the server again.
> This has the obvious downside that the tests can only be executed from 
> the Maven build which isn't the build tool .NET developers would use 
> and also not the tool the IDE will use (the situation is basically the 
> same for other GLVs).
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Dale <ro...@gmail.com>
> Gesendet: Mittwoch, 27. Februar 2019 13:52
> An: dev@tinkerpop.apache.org
> Betreff: Re: docker and GLV development
>
> I'm surprised Daniel doesn't have this done already ;-)
>
> What is the test configuration/data for GLVs?  Could you list the 
> paths/files?
>
> Robert Dale
>
>
> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette 
> <sp...@gmail.com>
> wrote:
>
> > Yes, an on-the-fly container would work assuming it used the test 
> > configuration/data for GLVs. That's why i suggested the command as I 
> > did because it was similar to docker/build.sh.
> >
> >
> > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com> wrote:
> >
> > > Do you want to create a docker container on-the-fly using the 
> > > current working copy kind of like we do for docker/build.sh?
> > >
> > > Robert Dale
> > >
> > >
> > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette 
> > > <sp...@gmail.com>
> > > wrote:
> > >
> > > > testcontainers seems neat as it gives some good integration and
> > > automation,
> > > > however reliance on nightly snapshots doesn't sound so great.
> > > > There are times when changes are occurring on both client and 
> > > > server so a local up-to-the-moment image would be necessary. I 
> > > > also think we might want
> > to
> > > > avoid nightly snapshots of anything until the ASF figures out 
> > > > more
> > > clearly
> > > > how those play into projects (there is a bit of discussion going 
> > > > on
> > with
> > > > that now). I do like the idea of being able to test against 
> > > > older
> > > versions
> > > > though.  Ensuring backward compatibility.is a nice thought.
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > fh@florian-hockmann.de
> > > >
> > > > wrote:
> > > >
> > > > > Fantastic idea, I often missed the ability to just start tests 
> > > > > for Gremlin.Net from my IDE.
> > > > >
> > > > > Luckily for us, there is a project that is aimed exactly at 
> > > > > problems
> > > like
> > > > > this:
> > > > > https://www.testcontainers.org/
> > > > >
> > > > > The basic idea of Testcontainers is that you define a Docker
> > container
> > > in
> > > > > your tests for whatever external dependencies you need.
> > > > > Databases
> > are a
> > > > > frequent example where Testcontainers makes sense. For us, 
> > > > > Gremlin
> > > Server
> > > > > would be such a container.
> > > > > Testcontainers then uses the ability of the testing framework (e.g.
> > > > JUnit)
> > > > > to start the Docker container for the tests and waits until 
> > > > > they are
> > > > fully
> > > > > started (how they are started can be configured for each 
> > > > > container differently). After the tests have completed, it 
> > > > > also stops the
> > > > containers
> > > > > again.
> > > > > That way, you don't have to manage the containers manually anymore.
> > > > > Everything is done in the background by the IDE / the build tool.
> > > > >
> > > > > Testcontainers was originally a Java library, but they now 
> > > > > also have libraries for different languages, including the 
> > > > > ones for which we
> > have
> > > > > GLVs:
> > > > >
> > > > > testcontainers-python:
> > > > > https://github.com/testcontainers/testcontainers-python
> > > > > testcontainers-dotnet
> > > > > <
> > > >
> > >
> > https://github.com/testcontainers/testcontainers-pythontestcontainer
> > s-
> > dotnet
> > > > >:
> > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > testcontainers-node
> > > > > <
> > > >
> > >
> > https://github.com/testcontainers/testcontainers-dotnettestcontainer
> > s-
> > node
> > > > >:
> > > > > https://github.com/testcontainers/testcontainers-node
> > > > >
> > > > > Testcontainers-java is the most advanced library of these of
> course.
> > > The
> > > > > others still lack some functionality, but I guess that we 
> > > > > could also
> > > just
> > > > > contribute to those libraries if an important feature is 
> > > > > missing for
> > > us.
> > > > >
> > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > testcontainers-dotnet
> > > > > unfortunately can only use images that are available in some
> > registry.
> > > > So,
> > > > > we would have to push a Docker Image of Gremlin Server to 
> > > > > DockerHub
> > to
> > > > use
> > > > > it in our tests. That means that we could either only test the 
> > > > > GLVs
> > > > against
> > > > > already released versions or we could publish something like 
> > > > > nightly
> > > > build
> > > > > images of Gremlin Server.
> > > > > I originally wanted to wait until testcontainers-dotnet is 
> > > > > able to
> > use
> > > > > images that are only available locally (testcontainers-java 
> > > > > already
> > has
> > > > > that capability), but now that I think more about this, I 
> > > > > think that
> > it
> > > > > makes actually more sense to use images that are available in 
> > > > > the
> > > > registry
> > > > > as every developer would otherwise have to build test images 
> > > > > first
> > > > locally
> > > > > in order to use them for tests.
> > > > >
> > > > > So, I suggest that you guys take a look at Testcontainers to 
> > > > > see
> > > whether
> > > > > you agree that it's a good solution to our problem and then we 
> > > > > can
> > see
> > > > > whether we want to publish nightly builds for the GLV tests or 
> > > > > how we
> > > > want
> > > > > to handle that.
> > > > >
> > > > > -----Ursprüngliche Nachricht-----
> > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > An: dev@tinkerpop.apache.org
> > > > > Betreff: docker and GLV development
> > > > >
> > > > > Anyone have any idea how to make it so that docker can be 
> > > > > useful for helping to streamline GLV development? I'd love to 
> > > > > be able to do
> > > > something
> > > > > like:
> > > > >
> > > > > docker/gremlin-server.sh -test
> > > > >
> > > > > which would start Gremlin Server with our standard test 
> > > > > configuration (currently built into the maven tool chain, but 
> > > > > I think that could be extracted without too much difficulty). 
> > > > > That way, we could easily
> > open
> > > up
> > > > > whatever GLV specific IDE we wanted and run tests directly in 
> > > > > there,
> > > use
> > > > > the debugger, etc.
> > > > >
> > > > > Sound useful? and if so, is that something that is 
> > > > > easy/possible? and
> > > > then
> > > > > further....any volunteers with the know-how to get that setup 
> > > > > (i'm
> > > happy
> > > > to
> > > > > assist however i can)?
> > > > >
> > > > > btw, if you have a nicer way to set up an environment for GLV
> > > development
> > > > > and docker is a waste of time, please clue me in. i don't mind 
> > > > > using
> > > the
> > > > > maven integration but i feel like i could be more efficient 
> > > > > with a
> > more
> > > > > streamlined setup.
> > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

I think there are two components here.
1) Script to build the dockerized test server. Includes ability to start
container manually, externally from tests.
2) Testcontainers integration which uses the docker image from #1.
Testcontainers could also include executing the script in #1 to build the
container.

Does this sound right to you?

Robert Dale


On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> > I do like the idea of being able to test against older versions though.
>
> That shouldn't be a problem in general as we could use multiple different
> images. See for example this PR that adds Testcontainers to JanusGraph to
> test different Elasticsearch versions:
> https://github.com/JanusGraph/janusgraph/pull/1409
>
> > Yes, an on-the-fly container would work
>
> We could however also use an on-the-fly container with Testcontainers. We
> would just need some automation to build the image in the background.
> What if we would build an image during the usual Maven build for testing
> and give it a descriptive tag like "testing". Then, we could use that tag
> with Testcontainers in addition to tags for already released versions.
> Testcontainers-Java can apparently even build images by itself. So, that
> could also be a possibility to get an image with the latest Gremlin Server
> with the current (unreleased) changes.
>
> In general, I would really like to get away from a situation where we have
> to start some script in the background before tests can be executed.
>
> @Robert:
> > What is the test configuration/data for GLVs?  Could you list the
> paths/files?
>
> For Gremlin.Net for example, you can find the automation happening in the
> pom.xml of the testing part:
>
> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d09988282ea5/gremlin-dotnet/test/pom.xml#L179
>
> It calls the test-server-start.groovy script before the tests are executed
> and afterwards test-server-stop.groovy script to stop the server again.
> This has the obvious downside that the tests can only be executed from the
> Maven build which isn't the build tool .NET developers would use and also
> not the tool the IDE will use (the situation is basically the same for
> other GLVs).
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Dale <ro...@gmail.com>
> Gesendet: Mittwoch, 27. Februar 2019 13:52
> An: dev@tinkerpop.apache.org
> Betreff: Re: docker and GLV development
>
> I'm surprised Daniel doesn't have this done already ;-)
>
> What is the test configuration/data for GLVs?  Could you list the
> paths/files?
>
> Robert Dale
>
>
> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > Yes, an on-the-fly container would work assuming it used the test
> > configuration/data for GLVs. That's why i suggested the command as I
> > did because it was similar to docker/build.sh.
> >
> >
> > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com> wrote:
> >
> > > Do you want to create a docker container on-the-fly using the
> > > current working copy kind of like we do for docker/build.sh?
> > >
> > > Robert Dale
> > >
> > >
> > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> > > <sp...@gmail.com>
> > > wrote:
> > >
> > > > testcontainers seems neat as it gives some good integration and
> > > automation,
> > > > however reliance on nightly snapshots doesn't sound so great.
> > > > There are times when changes are occurring on both client and
> > > > server so a local up-to-the-moment image would be necessary. I
> > > > also think we might want
> > to
> > > > avoid nightly snapshots of anything until the ASF figures out more
> > > clearly
> > > > how those play into projects (there is a bit of discussion going
> > > > on
> > with
> > > > that now). I do like the idea of being able to test against older
> > > versions
> > > > though.  Ensuring backward compatibility.is a nice thought.
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > fh@florian-hockmann.de
> > > >
> > > > wrote:
> > > >
> > > > > Fantastic idea, I often missed the ability to just start tests
> > > > > for Gremlin.Net from my IDE.
> > > > >
> > > > > Luckily for us, there is a project that is aimed exactly at
> > > > > problems
> > > like
> > > > > this:
> > > > > https://www.testcontainers.org/
> > > > >
> > > > > The basic idea of Testcontainers is that you define a Docker
> > container
> > > in
> > > > > your tests for whatever external dependencies you need.
> > > > > Databases
> > are a
> > > > > frequent example where Testcontainers makes sense. For us,
> > > > > Gremlin
> > > Server
> > > > > would be such a container.
> > > > > Testcontainers then uses the ability of the testing framework (e.g.
> > > > JUnit)
> > > > > to start the Docker container for the tests and waits until they
> > > > > are
> > > > fully
> > > > > started (how they are started can be configured for each
> > > > > container differently). After the tests have completed, it also
> > > > > stops the
> > > > containers
> > > > > again.
> > > > > That way, you don't have to manage the containers manually anymore.
> > > > > Everything is done in the background by the IDE / the build tool.
> > > > >
> > > > > Testcontainers was originally a Java library, but they now also
> > > > > have libraries for different languages, including the ones for
> > > > > which we
> > have
> > > > > GLVs:
> > > > >
> > > > > testcontainers-python:
> > > > > https://github.com/testcontainers/testcontainers-python
> > > > > testcontainers-dotnet
> > > > > <
> > > >
> > >
> > https://github.com/testcontainers/testcontainers-pythontestcontainers-
> > dotnet
> > > > >:
> > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > testcontainers-node
> > > > > <
> > > >
> > >
> > https://github.com/testcontainers/testcontainers-dotnettestcontainers-
> > node
> > > > >:
> > > > > https://github.com/testcontainers/testcontainers-node
> > > > >
> > > > > Testcontainers-java is the most advanced library of these of
> course.
> > > The
> > > > > others still lack some functionality, but I guess that we could
> > > > > also
> > > just
> > > > > contribute to those libraries if an important feature is missing
> > > > > for
> > > us.
> > > > >
> > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > testcontainers-dotnet
> > > > > unfortunately can only use images that are available in some
> > registry.
> > > > So,
> > > > > we would have to push a Docker Image of Gremlin Server to
> > > > > DockerHub
> > to
> > > > use
> > > > > it in our tests. That means that we could either only test the
> > > > > GLVs
> > > > against
> > > > > already released versions or we could publish something like
> > > > > nightly
> > > > build
> > > > > images of Gremlin Server.
> > > > > I originally wanted to wait until testcontainers-dotnet is able
> > > > > to
> > use
> > > > > images that are only available locally (testcontainers-java
> > > > > already
> > has
> > > > > that capability), but now that I think more about this, I think
> > > > > that
> > it
> > > > > makes actually more sense to use images that are available in
> > > > > the
> > > > registry
> > > > > as every developer would otherwise have to build test images
> > > > > first
> > > > locally
> > > > > in order to use them for tests.
> > > > >
> > > > > So, I suggest that you guys take a look at Testcontainers to see
> > > whether
> > > > > you agree that it's a good solution to our problem and then we
> > > > > can
> > see
> > > > > whether we want to publish nightly builds for the GLV tests or
> > > > > how we
> > > > want
> > > > > to handle that.
> > > > >
> > > > > -----Ursprüngliche Nachricht-----
> > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > An: dev@tinkerpop.apache.org
> > > > > Betreff: docker and GLV development
> > > > >
> > > > > Anyone have any idea how to make it so that docker can be useful
> > > > > for helping to streamline GLV development? I'd love to be able
> > > > > to do
> > > > something
> > > > > like:
> > > > >
> > > > > docker/gremlin-server.sh -test
> > > > >
> > > > > which would start Gremlin Server with our standard test
> > > > > configuration (currently built into the maven tool chain, but I
> > > > > think that could be extracted without too much difficulty). That
> > > > > way, we could easily
> > open
> > > up
> > > > > whatever GLV specific IDE we wanted and run tests directly in
> > > > > there,
> > > use
> > > > > the debugger, etc.
> > > > >
> > > > > Sound useful? and if so, is that something that is
> > > > > easy/possible? and
> > > > then
> > > > > further....any volunteers with the know-how to get that setup
> > > > > (i'm
> > > happy
> > > > to
> > > > > assist however i can)?
> > > > >
> > > > > btw, if you have a nicer way to set up an environment for GLV
> > > development
> > > > > and docker is a waste of time, please clue me in. i don't mind
> > > > > using
> > > the
> > > > > maven integration but i feel like i could be more efficient with
> > > > > a
> > more
> > > > > streamlined setup.
> > > > >
> > > > >
> > > >
> > >
> >
>
>

AW: docker and GLV development

[Florian Hockmann <fh...@florian-hockmann.de>]

> I do like the idea of being able to test against older versions though.  

That shouldn't be a problem in general as we could use multiple different images. See for example this PR that adds Testcontainers to JanusGraph to test different Elasticsearch versions:
https://github.com/JanusGraph/janusgraph/pull/1409

> Yes, an on-the-fly container would work

We could however also use an on-the-fly container with Testcontainers. We would just need some automation to build the image in the background.
What if we would build an image during the usual Maven build for testing and give it a descriptive tag like "testing". Then, we could use that tag with Testcontainers in addition to tags for already released versions.
Testcontainers-Java can apparently even build images by itself. So, that could also be a possibility to get an image with the latest Gremlin Server with the current (unreleased) changes.

In general, I would really like to get away from a situation where we have to start some script in the background before tests can be executed.

@Robert:
> What is the test configuration/data for GLVs?  Could you list the paths/files?

For Gremlin.Net for example, you can find the automation happening in the pom.xml of the testing part:
https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d09988282ea5/gremlin-dotnet/test/pom.xml#L179

It calls the test-server-start.groovy script before the tests are executed and afterwards test-server-stop.groovy script to stop the server again. This has the obvious downside that the tests can only be executed from the Maven build which isn't the build tool .NET developers would use and also not the tool the IDE will use (the situation is basically the same for other GLVs).

-----Ursprüngliche Nachricht-----
Von: Robert Dale <ro...@gmail.com> 
Gesendet: Mittwoch, 27. Februar 2019 13:52
An: dev@tinkerpop.apache.org
Betreff: Re: docker and GLV development

I'm surprised Daniel doesn't have this done already ;-)

What is the test configuration/data for GLVs?  Could you list the paths/files?

Robert Dale


On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette <sp...@gmail.com>
wrote:

> Yes, an on-the-fly container would work assuming it used the test 
> configuration/data for GLVs. That's why i suggested the command as I 
> did because it was similar to docker/build.sh.
>
>
> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com> wrote:
>
> > Do you want to create a docker container on-the-fly using the 
> > current working copy kind of like we do for docker/build.sh?
> >
> > Robert Dale
> >
> >
> > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette 
> > <sp...@gmail.com>
> > wrote:
> >
> > > testcontainers seems neat as it gives some good integration and
> > automation,
> > > however reliance on nightly snapshots doesn't sound so great. 
> > > There are times when changes are occurring on both client and 
> > > server so a local up-to-the-moment image would be necessary. I 
> > > also think we might want
> to
> > > avoid nightly snapshots of anything until the ASF figures out more
> > clearly
> > > how those play into projects (there is a bit of discussion going 
> > > on
> with
> > > that now). I do like the idea of being able to test against older
> > versions
> > > though.  Ensuring backward compatibility.is a nice thought.
> > >
> > >
> > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> fh@florian-hockmann.de
> > >
> > > wrote:
> > >
> > > > Fantastic idea, I often missed the ability to just start tests 
> > > > for Gremlin.Net from my IDE.
> > > >
> > > > Luckily for us, there is a project that is aimed exactly at 
> > > > problems
> > like
> > > > this:
> > > > https://www.testcontainers.org/
> > > >
> > > > The basic idea of Testcontainers is that you define a Docker
> container
> > in
> > > > your tests for whatever external dependencies you need. 
> > > > Databases
> are a
> > > > frequent example where Testcontainers makes sense. For us, 
> > > > Gremlin
> > Server
> > > > would be such a container.
> > > > Testcontainers then uses the ability of the testing framework (e.g.
> > > JUnit)
> > > > to start the Docker container for the tests and waits until they 
> > > > are
> > > fully
> > > > started (how they are started can be configured for each 
> > > > container differently). After the tests have completed, it also 
> > > > stops the
> > > containers
> > > > again.
> > > > That way, you don't have to manage the containers manually anymore.
> > > > Everything is done in the background by the IDE / the build tool.
> > > >
> > > > Testcontainers was originally a Java library, but they now also 
> > > > have libraries for different languages, including the ones for 
> > > > which we
> have
> > > > GLVs:
> > > >
> > > > testcontainers-python:
> > > > https://github.com/testcontainers/testcontainers-python
> > > > testcontainers-dotnet
> > > > <
> > >
> >
> https://github.com/testcontainers/testcontainers-pythontestcontainers-
> dotnet
> > > >:
> > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > testcontainers-node
> > > > <
> > >
> >
> https://github.com/testcontainers/testcontainers-dotnettestcontainers-
> node
> > > >:
> > > > https://github.com/testcontainers/testcontainers-node
> > > >
> > > > Testcontainers-java is the most advanced library of these of course.
> > The
> > > > others still lack some functionality, but I guess that we could 
> > > > also
> > just
> > > > contribute to those libraries if an important feature is missing 
> > > > for
> > us.
> > > >
> > > > I wanted to use Testcontainers for Gremlin.Net, but
> > testcontainers-dotnet
> > > > unfortunately can only use images that are available in some
> registry.
> > > So,
> > > > we would have to push a Docker Image of Gremlin Server to 
> > > > DockerHub
> to
> > > use
> > > > it in our tests. That means that we could either only test the 
> > > > GLVs
> > > against
> > > > already released versions or we could publish something like 
> > > > nightly
> > > build
> > > > images of Gremlin Server.
> > > > I originally wanted to wait until testcontainers-dotnet is able 
> > > > to
> use
> > > > images that are only available locally (testcontainers-java 
> > > > already
> has
> > > > that capability), but now that I think more about this, I think 
> > > > that
> it
> > > > makes actually more sense to use images that are available in 
> > > > the
> > > registry
> > > > as every developer would otherwise have to build test images 
> > > > first
> > > locally
> > > > in order to use them for tests.
> > > >
> > > > So, I suggest that you guys take a look at Testcontainers to see
> > whether
> > > > you agree that it's a good solution to our problem and then we 
> > > > can
> see
> > > > whether we want to publish nightly builds for the GLV tests or 
> > > > how we
> > > want
> > > > to handle that.
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > An: dev@tinkerpop.apache.org
> > > > Betreff: docker and GLV development
> > > >
> > > > Anyone have any idea how to make it so that docker can be useful 
> > > > for helping to streamline GLV development? I'd love to be able 
> > > > to do
> > > something
> > > > like:
> > > >
> > > > docker/gremlin-server.sh -test
> > > >
> > > > which would start Gremlin Server with our standard test 
> > > > configuration (currently built into the maven tool chain, but I 
> > > > think that could be extracted without too much difficulty). That 
> > > > way, we could easily
> open
> > up
> > > > whatever GLV specific IDE we wanted and run tests directly in 
> > > > there,
> > use
> > > > the debugger, etc.
> > > >
> > > > Sound useful? and if so, is that something that is 
> > > > easy/possible? and
> > > then
> > > > further....any volunteers with the know-how to get that setup 
> > > > (i'm
> > happy
> > > to
> > > > assist however i can)?
> > > >
> > > > btw, if you have a nicer way to set up an environment for GLV
> > development
> > > > and docker is a waste of time, please clue me in. i don't mind 
> > > > using
> > the
> > > > maven integration but i feel like i could be more efficient with 
> > > > a
> more
> > > > streamlined setup.
> > > >
> > > >
> > >
> >
>

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

I'm surprised Daniel doesn't have this done already ;-)

What is the test configuration/data for GLVs?  Could you list the
paths/files?

Robert Dale


On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette <sp...@gmail.com>
wrote:

> Yes, an on-the-fly container would work assuming it used the test
> configuration/data for GLVs. That's why i suggested the command as I did
> because it was similar to docker/build.sh.
>
>
> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com> wrote:
>
> > Do you want to create a docker container on-the-fly using the current
> > working copy kind of like we do for docker/build.sh?
> >
> > Robert Dale
> >
> >
> > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > testcontainers seems neat as it gives some good integration and
> > automation,
> > > however reliance on nightly snapshots doesn't sound so great. There are
> > > times when changes are occurring on both client and server so a local
> > > up-to-the-moment image would be necessary. I also think we might want
> to
> > > avoid nightly snapshots of anything until the ASF figures out more
> > clearly
> > > how those play into projects (there is a bit of discussion going on
> with
> > > that now). I do like the idea of being able to test against older
> > versions
> > > though.  Ensuring backward compatibility.is a nice thought.
> > >
> > >
> > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> fh@florian-hockmann.de
> > >
> > > wrote:
> > >
> > > > Fantastic idea, I often missed the ability to just start tests for
> > > > Gremlin.Net from my IDE.
> > > >
> > > > Luckily for us, there is a project that is aimed exactly at problems
> > like
> > > > this:
> > > > https://www.testcontainers.org/
> > > >
> > > > The basic idea of Testcontainers is that you define a Docker
> container
> > in
> > > > your tests for whatever external dependencies you need. Databases
> are a
> > > > frequent example where Testcontainers makes sense. For us, Gremlin
> > Server
> > > > would be such a container.
> > > > Testcontainers then uses the ability of the testing framework (e.g.
> > > JUnit)
> > > > to start the Docker container for the tests and waits until they are
> > > fully
> > > > started (how they are started can be configured for each container
> > > > differently). After the tests have completed, it also stops the
> > > containers
> > > > again.
> > > > That way, you don't have to manage the containers manually anymore.
> > > > Everything is done in the background by the IDE / the build tool.
> > > >
> > > > Testcontainers was originally a Java library, but they now also have
> > > > libraries for different languages, including the ones for which we
> have
> > > > GLVs:
> > > >
> > > > testcontainers-python:
> > > > https://github.com/testcontainers/testcontainers-python
> > > > testcontainers-dotnet
> > > > <
> > >
> >
> https://github.com/testcontainers/testcontainers-pythontestcontainers-dotnet
> > > >:
> > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > testcontainers-node
> > > > <
> > >
> >
> https://github.com/testcontainers/testcontainers-dotnettestcontainers-node
> > > >:
> > > > https://github.com/testcontainers/testcontainers-node
> > > >
> > > > Testcontainers-java is the most advanced library of these of course.
> > The
> > > > others still lack some functionality, but I guess that we could also
> > just
> > > > contribute to those libraries if an important feature is missing for
> > us.
> > > >
> > > > I wanted to use Testcontainers for Gremlin.Net, but
> > testcontainers-dotnet
> > > > unfortunately can only use images that are available in some
> registry.
> > > So,
> > > > we would have to push a Docker Image of Gremlin Server to DockerHub
> to
> > > use
> > > > it in our tests. That means that we could either only test the GLVs
> > > against
> > > > already released versions or we could publish something like nightly
> > > build
> > > > images of Gremlin Server.
> > > > I originally wanted to wait until testcontainers-dotnet is able to
> use
> > > > images that are only available locally (testcontainers-java already
> has
> > > > that capability), but now that I think more about this, I think that
> it
> > > > makes actually more sense to use images that are available in the
> > > registry
> > > > as every developer would otherwise have to build test images first
> > > locally
> > > > in order to use them for tests.
> > > >
> > > > So, I suggest that you guys take a look at Testcontainers to see
> > whether
> > > > you agree that it's a good solution to our problem and then we can
> see
> > > > whether we want to publish nightly builds for the GLV tests or how we
> > > want
> > > > to handle that.
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > An: dev@tinkerpop.apache.org
> > > > Betreff: docker and GLV development
> > > >
> > > > Anyone have any idea how to make it so that docker can be useful for
> > > > helping to streamline GLV development? I'd love to be able to do
> > > something
> > > > like:
> > > >
> > > > docker/gremlin-server.sh -test
> > > >
> > > > which would start Gremlin Server with our standard test configuration
> > > > (currently built into the maven tool chain, but I think that could be
> > > > extracted without too much difficulty). That way, we could easily
> open
> > up
> > > > whatever GLV specific IDE we wanted and run tests directly in there,
> > use
> > > > the debugger, etc.
> > > >
> > > > Sound useful? and if so, is that something that is easy/possible? and
> > > then
> > > > further....any volunteers with the know-how to get that setup (i'm
> > happy
> > > to
> > > > assist however i can)?
> > > >
> > > > btw, if you have a nicer way to set up an environment for GLV
> > development
> > > > and docker is a waste of time, please clue me in. i don't mind using
> > the
> > > > maven integration but i feel like i could be more efficient with a
> more
> > > > streamlined setup.
> > > >
> > > >
> > >
> >
>

Re: docker and GLV development

[Stephen Mallette <sp...@gmail.com>]

Yes, an on-the-fly container would work assuming it used the test
configuration/data for GLVs. That's why i suggested the command as I did
because it was similar to docker/build.sh.


On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com> wrote:

> Do you want to create a docker container on-the-fly using the current
> working copy kind of like we do for docker/build.sh?
>
> Robert Dale
>
>
> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > testcontainers seems neat as it gives some good integration and
> automation,
> > however reliance on nightly snapshots doesn't sound so great. There are
> > times when changes are occurring on both client and server so a local
> > up-to-the-moment image would be necessary. I also think we might want to
> > avoid nightly snapshots of anything until the ASF figures out more
> clearly
> > how those play into projects (there is a bit of discussion going on with
> > that now). I do like the idea of being able to test against older
> versions
> > though.  Ensuring backward compatibility.is a nice thought.
> >
> >
> > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <fh@florian-hockmann.de
> >
> > wrote:
> >
> > > Fantastic idea, I often missed the ability to just start tests for
> > > Gremlin.Net from my IDE.
> > >
> > > Luckily for us, there is a project that is aimed exactly at problems
> like
> > > this:
> > > https://www.testcontainers.org/
> > >
> > > The basic idea of Testcontainers is that you define a Docker container
> in
> > > your tests for whatever external dependencies you need. Databases are a
> > > frequent example where Testcontainers makes sense. For us, Gremlin
> Server
> > > would be such a container.
> > > Testcontainers then uses the ability of the testing framework (e.g.
> > JUnit)
> > > to start the Docker container for the tests and waits until they are
> > fully
> > > started (how they are started can be configured for each container
> > > differently). After the tests have completed, it also stops the
> > containers
> > > again.
> > > That way, you don't have to manage the containers manually anymore.
> > > Everything is done in the background by the IDE / the build tool.
> > >
> > > Testcontainers was originally a Java library, but they now also have
> > > libraries for different languages, including the ones for which we have
> > > GLVs:
> > >
> > > testcontainers-python:
> > > https://github.com/testcontainers/testcontainers-python
> > > testcontainers-dotnet
> > > <
> >
> https://github.com/testcontainers/testcontainers-pythontestcontainers-dotnet
> > >:
> > > https://github.com/testcontainers/testcontainers-dotnet
> > > testcontainers-node
> > > <
> >
> https://github.com/testcontainers/testcontainers-dotnettestcontainers-node
> > >:
> > > https://github.com/testcontainers/testcontainers-node
> > >
> > > Testcontainers-java is the most advanced library of these of course.
> The
> > > others still lack some functionality, but I guess that we could also
> just
> > > contribute to those libraries if an important feature is missing for
> us.
> > >
> > > I wanted to use Testcontainers for Gremlin.Net, but
> testcontainers-dotnet
> > > unfortunately can only use images that are available in some registry.
> > So,
> > > we would have to push a Docker Image of Gremlin Server to DockerHub to
> > use
> > > it in our tests. That means that we could either only test the GLVs
> > against
> > > already released versions or we could publish something like nightly
> > build
> > > images of Gremlin Server.
> > > I originally wanted to wait until testcontainers-dotnet is able to use
> > > images that are only available locally (testcontainers-java already has
> > > that capability), but now that I think more about this, I think that it
> > > makes actually more sense to use images that are available in the
> > registry
> > > as every developer would otherwise have to build test images first
> > locally
> > > in order to use them for tests.
> > >
> > > So, I suggest that you guys take a look at Testcontainers to see
> whether
> > > you agree that it's a good solution to our problem and then we can see
> > > whether we want to publish nightly builds for the GLV tests or how we
> > want
> > > to handle that.
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Stephen Mallette <sp...@gmail.com>
> > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > An: dev@tinkerpop.apache.org
> > > Betreff: docker and GLV development
> > >
> > > Anyone have any idea how to make it so that docker can be useful for
> > > helping to streamline GLV development? I'd love to be able to do
> > something
> > > like:
> > >
> > > docker/gremlin-server.sh -test
> > >
> > > which would start Gremlin Server with our standard test configuration
> > > (currently built into the maven tool chain, but I think that could be
> > > extracted without too much difficulty). That way, we could easily open
> up
> > > whatever GLV specific IDE we wanted and run tests directly in there,
> use
> > > the debugger, etc.
> > >
> > > Sound useful? and if so, is that something that is easy/possible? and
> > then
> > > further....any volunteers with the know-how to get that setup (i'm
> happy
> > to
> > > assist however i can)?
> > >
> > > btw, if you have a nicer way to set up an environment for GLV
> development
> > > and docker is a waste of time, please clue me in. i don't mind using
> the
> > > maven integration but i feel like i could be more efficient with a more
> > > streamlined setup.
> > >
> > >
> >
>

Re: docker and GLV development

[Robert Dale <ro...@gmail.com>]

Do you want to create a docker container on-the-fly using the current
working copy kind of like we do for docker/build.sh?

Robert Dale


On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette <sp...@gmail.com>
wrote:

> testcontainers seems neat as it gives some good integration and automation,
> however reliance on nightly snapshots doesn't sound so great. There are
> times when changes are occurring on both client and server so a local
> up-to-the-moment image would be necessary. I also think we might want to
> avoid nightly snapshots of anything until the ASF figures out more clearly
> how those play into projects (there is a bit of discussion going on with
> that now). I do like the idea of being able to test against older versions
> though.  Ensuring backward compatibility.is a nice thought.
>
>
> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
> > Fantastic idea, I often missed the ability to just start tests for
> > Gremlin.Net from my IDE.
> >
> > Luckily for us, there is a project that is aimed exactly at problems like
> > this:
> > https://www.testcontainers.org/
> >
> > The basic idea of Testcontainers is that you define a Docker container in
> > your tests for whatever external dependencies you need. Databases are a
> > frequent example where Testcontainers makes sense. For us, Gremlin Server
> > would be such a container.
> > Testcontainers then uses the ability of the testing framework (e.g.
> JUnit)
> > to start the Docker container for the tests and waits until they are
> fully
> > started (how they are started can be configured for each container
> > differently). After the tests have completed, it also stops the
> containers
> > again.
> > That way, you don't have to manage the containers manually anymore.
> > Everything is done in the background by the IDE / the build tool.
> >
> > Testcontainers was originally a Java library, but they now also have
> > libraries for different languages, including the ones for which we have
> > GLVs:
> >
> > testcontainers-python:
> > https://github.com/testcontainers/testcontainers-python
> > testcontainers-dotnet
> > <
> https://github.com/testcontainers/testcontainers-pythontestcontainers-dotnet
> >:
> > https://github.com/testcontainers/testcontainers-dotnet
> > testcontainers-node
> > <
> https://github.com/testcontainers/testcontainers-dotnettestcontainers-node
> >:
> > https://github.com/testcontainers/testcontainers-node
> >
> > Testcontainers-java is the most advanced library of these of course. The
> > others still lack some functionality, but I guess that we could also just
> > contribute to those libraries if an important feature is missing for us.
> >
> > I wanted to use Testcontainers for Gremlin.Net, but testcontainers-dotnet
> > unfortunately can only use images that are available in some registry.
> So,
> > we would have to push a Docker Image of Gremlin Server to DockerHub to
> use
> > it in our tests. That means that we could either only test the GLVs
> against
> > already released versions or we could publish something like nightly
> build
> > images of Gremlin Server.
> > I originally wanted to wait until testcontainers-dotnet is able to use
> > images that are only available locally (testcontainers-java already has
> > that capability), but now that I think more about this, I think that it
> > makes actually more sense to use images that are available in the
> registry
> > as every developer would otherwise have to build test images first
> locally
> > in order to use them for tests.
> >
> > So, I suggest that you guys take a look at Testcontainers to see whether
> > you agree that it's a good solution to our problem and then we can see
> > whether we want to publish nightly builds for the GLV tests or how we
> want
> > to handle that.
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Stephen Mallette <sp...@gmail.com>
> > Gesendet: Dienstag, 26. Februar 2019 22:24
> > An: dev@tinkerpop.apache.org
> > Betreff: docker and GLV development
> >
> > Anyone have any idea how to make it so that docker can be useful for
> > helping to streamline GLV development? I'd love to be able to do
> something
> > like:
> >
> > docker/gremlin-server.sh -test
> >
> > which would start Gremlin Server with our standard test configuration
> > (currently built into the maven tool chain, but I think that could be
> > extracted without too much difficulty). That way, we could easily open up
> > whatever GLV specific IDE we wanted and run tests directly in there, use
> > the debugger, etc.
> >
> > Sound useful? and if so, is that something that is easy/possible? and
> then
> > further....any volunteers with the know-how to get that setup (i'm happy
> to
> > assist however i can)?
> >
> > btw, if you have a nicer way to set up an environment for GLV development
> > and docker is a waste of time, please clue me in. i don't mind using the
> > maven integration but i feel like i could be more efficient with a more
> > streamlined setup.
> >
> >
>

Re: docker and GLV development

[Stephen Mallette <sp...@gmail.com>]

testcontainers seems neat as it gives some good integration and automation,
however reliance on nightly snapshots doesn't sound so great. There are
times when changes are occurring on both client and server so a local
up-to-the-moment image would be necessary. I also think we might want to
avoid nightly snapshots of anything until the ASF figures out more clearly
how those play into projects (there is a bit of discussion going on with
that now). I do like the idea of being able to test against older versions
though.  Ensuring backward compatibility.is a nice thought.


On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <fh...@florian-hockmann.de>
wrote:

> Fantastic idea, I often missed the ability to just start tests for
> Gremlin.Net from my IDE.
>
> Luckily for us, there is a project that is aimed exactly at problems like
> this:
> https://www.testcontainers.org/
>
> The basic idea of Testcontainers is that you define a Docker container in
> your tests for whatever external dependencies you need. Databases are a
> frequent example where Testcontainers makes sense. For us, Gremlin Server
> would be such a container.
> Testcontainers then uses the ability of the testing framework (e.g. JUnit)
> to start the Docker container for the tests and waits until they are fully
> started (how they are started can be configured for each container
> differently). After the tests have completed, it also stops the containers
> again.
> That way, you don't have to manage the containers manually anymore.
> Everything is done in the background by the IDE / the build tool.
>
> Testcontainers was originally a Java library, but they now also have
> libraries for different languages, including the ones for which we have
> GLVs:
>
> testcontainers-python:
> https://github.com/testcontainers/testcontainers-python
> testcontainers-dotnet
> <https://github.com/testcontainers/testcontainers-pythontestcontainers-dotnet>:
> https://github.com/testcontainers/testcontainers-dotnet
> testcontainers-node
> <https://github.com/testcontainers/testcontainers-dotnettestcontainers-node>:
> https://github.com/testcontainers/testcontainers-node
>
> Testcontainers-java is the most advanced library of these of course. The
> others still lack some functionality, but I guess that we could also just
> contribute to those libraries if an important feature is missing for us.
>
> I wanted to use Testcontainers for Gremlin.Net, but testcontainers-dotnet
> unfortunately can only use images that are available in some registry. So,
> we would have to push a Docker Image of Gremlin Server to DockerHub to use
> it in our tests. That means that we could either only test the GLVs against
> already released versions or we could publish something like nightly build
> images of Gremlin Server.
> I originally wanted to wait until testcontainers-dotnet is able to use
> images that are only available locally (testcontainers-java already has
> that capability), but now that I think more about this, I think that it
> makes actually more sense to use images that are available in the registry
> as every developer would otherwise have to build test images first locally
> in order to use them for tests.
>
> So, I suggest that you guys take a look at Testcontainers to see whether
> you agree that it's a good solution to our problem and then we can see
> whether we want to publish nightly builds for the GLV tests or how we want
> to handle that.
>
> -----Ursprüngliche Nachricht-----
> Von: Stephen Mallette <sp...@gmail.com>
> Gesendet: Dienstag, 26. Februar 2019 22:24
> An: dev@tinkerpop.apache.org
> Betreff: docker and GLV development
>
> Anyone have any idea how to make it so that docker can be useful for
> helping to streamline GLV development? I'd love to be able to do something
> like:
>
> docker/gremlin-server.sh -test
>
> which would start Gremlin Server with our standard test configuration
> (currently built into the maven tool chain, but I think that could be
> extracted without too much difficulty). That way, we could easily open up
> whatever GLV specific IDE we wanted and run tests directly in there, use
> the debugger, etc.
>
> Sound useful? and if so, is that something that is easy/possible? and then
> further....any volunteers with the know-how to get that setup (i'm happy to
> assist however i can)?
>
> btw, if you have a nicer way to set up an environment for GLV development
> and docker is a waste of time, please clue me in. i don't mind using the
> maven integration but i feel like i could be more efficient with a more
> streamlined setup.
>
>

AW: docker and GLV development

[Florian Hockmann <fh...@florian-hockmann.de>]

Fantastic idea, I often missed the ability to just start tests for Gremlin.Net from my IDE.

Luckily for us, there is a project that is aimed exactly at problems like this:
https://www.testcontainers.org/

The basic idea of Testcontainers is that you define a Docker container in your tests for whatever external dependencies you need. Databases are a frequent example where Testcontainers makes sense. For us, Gremlin Server would be such a container.
Testcontainers then uses the ability of the testing framework (e.g. JUnit) to start the Docker container for the tests and waits until they are fully started (how they are started can be configured for each container differently). After the tests have completed, it also stops the containers again.
That way, you don't have to manage the containers manually anymore. Everything is done in the background by the IDE / the build tool.

Testcontainers was originally a Java library, but they now also have libraries for different languages, including the ones for which we have GLVs:

testcontainers-python: https://github.com/testcontainers/testcontainers-python
testcontainers-dotnet: https://github.com/testcontainers/testcontainers-dotnet
testcontainers-node: https://github.com/testcontainers/testcontainers-node

Testcontainers-java is the most advanced library of these of course. The others still lack some functionality, but I guess that we could also just contribute to those libraries if an important feature is missing for us.

I wanted to use Testcontainers for Gremlin.Net, but testcontainers-dotnet unfortunately can only use images that are available in some registry. So, we would have to push a Docker Image of Gremlin Server to DockerHub to use it in our tests. That means that we could either only test the GLVs against already released versions or we could publish something like nightly build images of Gremlin Server.
I originally wanted to wait until testcontainers-dotnet is able to use images that are only available locally (testcontainers-java already has that capability), but now that I think more about this, I think that it makes actually more sense to use images that are available in the registry as every developer would otherwise have to build test images first locally in order to use them for tests.

So, I suggest that you guys take a look at Testcontainers to see whether you agree that it's a good solution to our problem and then we can see whether we want to publish nightly builds for the GLV tests or how we want to handle that.

-----Ursprüngliche Nachricht-----
Von: Stephen Mallette <sp...@gmail.com> 
Gesendet: Dienstag, 26. Februar 2019 22:24
An: dev@tinkerpop.apache.org
Betreff: docker and GLV development

Anyone have any idea how to make it so that docker can be useful for helping to streamline GLV development? I'd love to be able to do something
like:

docker/gremlin-server.sh -test

which would start Gremlin Server with our standard test configuration (currently built into the maven tool chain, but I think that could be extracted without too much difficulty). That way, we could easily open up whatever GLV specific IDE we wanted and run tests directly in there, use the debugger, etc.

Sound useful? and if so, is that something that is easy/possible? and then further....any volunteers with the know-how to get that setup (i'm happy to assist however i can)?

btw, if you have a nicer way to set up an environment for GLV development and docker is a waste of time, please clue me in. i don't mind using the maven integration but i feel like i could be more efficient with a more streamlined setup.