You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Robert Dale <ro...@gmail.com> on 2019/03/01 15:05:21 UTC
Re: docker and GLV development
I made a first attempt. Let me know what you think.
https://github.com/apache/tinkerpop/pull/1075
Robert Dale
On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <sp...@gmail.com>
wrote:
> > If we agree on Testcontainers in general, then I can give this a try and
> use Testcontainers in one of our Java projects that need Gremlin Server for
> testing (gremlin-driver maybe?).
>
> The java integration tests for Gremlin Server flip a lot of configuration
> bits per test. I don't know how that works with testcontainers and if it
> will make our testing there any better. There are also a number of tests
> that stop/start the server under different conditions - not sure how that
> works with testcontainers either. Finally, wouldn't we expect adding docker
> to the mix to slow down our integration test runs for java?
>
> I don't want to seem like i'm completely against testcontainers btw. I just
> wonder if a script doesn't give us a 100% better experience than what we
> have now (which is nothing). If so, and the script is easy and zero-impact
> without new dependencies, then I think I like that approach. Also, It would
> be nice to have the script for other sorts of manual testing, such as
> testing Gremlin Console with Gremlin Server. If we'd like to do
> testcontainers for GLVs I suppose that can be done too, but for my
> purposes, since I was the one who asked for help with this initially,
> testcontainers doesn't sound like something that is an absolute requirement
> for my needs.
>
> thanks robert/florian for all the discussion and help - appreciated.
>
> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
> > Yes, except for the detail that Testcontainers could make the script
> > obsolete if it is also used to build the image by itself. That would have
> > the advantage that developers wouldn't have to execute the script to
> build
> > the image. (Executing the script with Testcontainers could also be a
> > possibility, but I'm not sure how good that works.)
> >
> > If we agree on Testcontainers in general, then I can give this a try and
> > use Testcontainers in one of our Java projects that need Gremlin Server
> for
> > testing (gremlin-driver maybe?).
> > With such a PR it would probably also be easier for others to see what
> > advantages / disadvantages using Testcontainers would have.
> >
> > I think using a Java project first makes most sense as we can use all
> > functionality Testcontainers has to offer there. The downside is of
> course
> > that we might not have all the functionality afterwards for other
> languages
> > and then we might need to extend the Testcontainers libraries for the
> > different languages. (My assumption here is that we won't need that much
> > functionality so it should be limited to 1 or 2 missing features.)
> > Alternatively, we could also start with a GLV where the Testcontainers
> > library doesn't have the same functionality which probably means that we
> > need a workaround to get an image of our current Gremlin Server version
> for
> > testing.
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Robert Dale <ro...@gmail.com>
> > Gesendet: Mittwoch, 27. Februar 2019 15:49
> > An: dev@tinkerpop.apache.org
> > Betreff: Re: docker and GLV development
> >
> > I think there are two components here.
> > 1) Script to build the dockerized test server. Includes ability to start
> > container manually, externally from tests.
> > 2) Testcontainers integration which uses the docker image from #1.
> > Testcontainers could also include executing the script in #1 to build the
> > container.
> >
> > Does this sound right to you?
> >
> > Robert Dale
> >
> >
> > On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <fh@florian-hockmann.de
> >
> > wrote:
> >
> > > > I do like the idea of being able to test against older versions
> though.
> > >
> > > That shouldn't be a problem in general as we could use multiple
> > > different images. See for example this PR that adds Testcontainers to
> > > JanusGraph to test different Elasticsearch versions:
> > > https://github.com/JanusGraph/janusgraph/pull/1409
> > >
> > > > Yes, an on-the-fly container would work
> > >
> > > We could however also use an on-the-fly container with Testcontainers.
> > > We would just need some automation to build the image in the
> background.
> > > What if we would build an image during the usual Maven build for
> > > testing and give it a descriptive tag like "testing". Then, we could
> > > use that tag with Testcontainers in addition to tags for already
> > released versions.
> > > Testcontainers-Java can apparently even build images by itself. So,
> > > that could also be a possibility to get an image with the latest
> > > Gremlin Server with the current (unreleased) changes.
> > >
> > > In general, I would really like to get away from a situation where we
> > > have to start some script in the background before tests can be
> executed.
> > >
> > > @Robert:
> > > > What is the test configuration/data for GLVs? Could you list the
> > > paths/files?
> > >
> > > For Gremlin.Net for example, you can find the automation happening in
> > > the pom.xml of the testing part:
> > >
> > > https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> > > 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> > >
> > > It calls the test-server-start.groovy script before the tests are
> > > executed and afterwards test-server-stop.groovy script to stop the
> > server again.
> > > This has the obvious downside that the tests can only be executed from
> > > the Maven build which isn't the build tool .NET developers would use
> > > and also not the tool the IDE will use (the situation is basically the
> > > same for other GLVs).
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Robert Dale <ro...@gmail.com>
> > > Gesendet: Mittwoch, 27. Februar 2019 13:52
> > > An: dev@tinkerpop.apache.org
> > > Betreff: Re: docker and GLV development
> > >
> > > I'm surprised Daniel doesn't have this done already ;-)
> > >
> > > What is the test configuration/data for GLVs? Could you list the
> > > paths/files?
> > >
> > > Robert Dale
> > >
> > >
> > > On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> > > <sp...@gmail.com>
> > > wrote:
> > >
> > > > Yes, an on-the-fly container would work assuming it used the test
> > > > configuration/data for GLVs. That's why i suggested the command as I
> > > > did because it was similar to docker/build.sh.
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> wrote:
> > > >
> > > > > Do you want to create a docker container on-the-fly using the
> > > > > current working copy kind of like we do for docker/build.sh?
> > > > >
> > > > > Robert Dale
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> > > > > <sp...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > testcontainers seems neat as it gives some good integration and
> > > > > automation,
> > > > > > however reliance on nightly snapshots doesn't sound so great.
> > > > > > There are times when changes are occurring on both client and
> > > > > > server so a local up-to-the-moment image would be necessary. I
> > > > > > also think we might want
> > > > to
> > > > > > avoid nightly snapshots of anything until the ASF figures out
> > > > > > more
> > > > > clearly
> > > > > > how those play into projects (there is a bit of discussion going
> > > > > > on
> > > > with
> > > > > > that now). I do like the idea of being able to test against
> > > > > > older
> > > > > versions
> > > > > > though. Ensuring backward compatibility.is a nice thought.
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > > > fh@florian-hockmann.de
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Fantastic idea, I often missed the ability to just start tests
> > > > > > > for Gremlin.Net from my IDE.
> > > > > > >
> > > > > > > Luckily for us, there is a project that is aimed exactly at
> > > > > > > problems
> > > > > like
> > > > > > > this:
> > > > > > > https://www.testcontainers.org/
> > > > > > >
> > > > > > > The basic idea of Testcontainers is that you define a Docker
> > > > container
> > > > > in
> > > > > > > your tests for whatever external dependencies you need.
> > > > > > > Databases
> > > > are a
> > > > > > > frequent example where Testcontainers makes sense. For us,
> > > > > > > Gremlin
> > > > > Server
> > > > > > > would be such a container.
> > > > > > > Testcontainers then uses the ability of the testing framework
> > (e.g.
> > > > > > JUnit)
> > > > > > > to start the Docker container for the tests and waits until
> > > > > > > they are
> > > > > > fully
> > > > > > > started (how they are started can be configured for each
> > > > > > > container differently). After the tests have completed, it
> > > > > > > also stops the
> > > > > > containers
> > > > > > > again.
> > > > > > > That way, you don't have to manage the containers manually
> > anymore.
> > > > > > > Everything is done in the background by the IDE / the build
> tool.
> > > > > > >
> > > > > > > Testcontainers was originally a Java library, but they now
> > > > > > > also have libraries for different languages, including the
> > > > > > > ones for which we
> > > > have
> > > > > > > GLVs:
> > > > > > >
> > > > > > > testcontainers-python:
> > > > > > > https://github.com/testcontainers/testcontainers-python
> > > > > > > testcontainers-dotnet
> > > > > > > <
> > > > > >
> > > > >
> > > > https://github.com/testcontainers/testcontainers-pythontestcontainer
> > > > s-
> > > > dotnet
> > > > > > >:
> > > > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > > > testcontainers-node
> > > > > > > <
> > > > > >
> > > > >
> > > > https://github.com/testcontainers/testcontainers-dotnettestcontainer
> > > > s-
> > > > node
> > > > > > >:
> > > > > > > https://github.com/testcontainers/testcontainers-node
> > > > > > >
> > > > > > > Testcontainers-java is the most advanced library of these of
> > > course.
> > > > > The
> > > > > > > others still lack some functionality, but I guess that we
> > > > > > > could also
> > > > > just
> > > > > > > contribute to those libraries if an important feature is
> > > > > > > missing for
> > > > > us.
> > > > > > >
> > > > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > > > testcontainers-dotnet
> > > > > > > unfortunately can only use images that are available in some
> > > > registry.
> > > > > > So,
> > > > > > > we would have to push a Docker Image of Gremlin Server to
> > > > > > > DockerHub
> > > > to
> > > > > > use
> > > > > > > it in our tests. That means that we could either only test the
> > > > > > > GLVs
> > > > > > against
> > > > > > > already released versions or we could publish something like
> > > > > > > nightly
> > > > > > build
> > > > > > > images of Gremlin Server.
> > > > > > > I originally wanted to wait until testcontainers-dotnet is
> > > > > > > able to
> > > > use
> > > > > > > images that are only available locally (testcontainers-java
> > > > > > > already
> > > > has
> > > > > > > that capability), but now that I think more about this, I
> > > > > > > think that
> > > > it
> > > > > > > makes actually more sense to use images that are available in
> > > > > > > the
> > > > > > registry
> > > > > > > as every developer would otherwise have to build test images
> > > > > > > first
> > > > > > locally
> > > > > > > in order to use them for tests.
> > > > > > >
> > > > > > > So, I suggest that you guys take a look at Testcontainers to
> > > > > > > see
> > > > > whether
> > > > > > > you agree that it's a good solution to our problem and then we
> > > > > > > can
> > > > see
> > > > > > > whether we want to publish nightly builds for the GLV tests or
> > > > > > > how we
> > > > > > want
> > > > > > > to handle that.
> > > > > > >
> > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > > > An: dev@tinkerpop.apache.org
> > > > > > > Betreff: docker and GLV development
> > > > > > >
> > > > > > > Anyone have any idea how to make it so that docker can be
> > > > > > > useful for helping to streamline GLV development? I'd love to
> > > > > > > be able to do
> > > > > > something
> > > > > > > like:
> > > > > > >
> > > > > > > docker/gremlin-server.sh -test
> > > > > > >
> > > > > > > which would start Gremlin Server with our standard test
> > > > > > > configuration (currently built into the maven tool chain, but
> > > > > > > I think that could be extracted without too much difficulty).
> > > > > > > That way, we could easily
> > > > open
> > > > > up
> > > > > > > whatever GLV specific IDE we wanted and run tests directly in
> > > > > > > there,
> > > > > use
> > > > > > > the debugger, etc.
> > > > > > >
> > > > > > > Sound useful? and if so, is that something that is
> > > > > > > easy/possible? and
> > > > > > then
> > > > > > > further....any volunteers with the know-how to get that setup
> > > > > > > (i'm
> > > > > happy
> > > > > > to
> > > > > > > assist however i can)?
> > > > > > >
> > > > > > > btw, if you have a nicer way to set up an environment for GLV
> > > > > development
> > > > > > > and docker is a waste of time, please clue me in. i don't mind
> > > > > > > using
> > > > > the
> > > > > > > maven integration but i feel like i could be more efficient
> > > > > > > with a
> > > > more
> > > > > > > streamlined setup.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
> >
>
Re: docker and GLV development
Posted by Robert Dale <ro...@gmail.com>.
Forgot to mention that the gremlin-python version is hardcoded.
Robert Dale
On Wed, Mar 6, 2019 at 2:36 PM Robert Dale <ro...@gmail.com> wrote:
> I just pushed a change that allows python tests to pass against the docker
> image.
>
> This will install the gremlin-python script support on the server so
> that's an extra download to wait for.
>
> I had to update the localhost references to point to the docker IP
> in gremlin-python: grep -Rl localhost gremlin-python/src/main/jython/ |
> xargs sed -i 's/localhost/172.17.0.2/g'
>
> Then I could `mvn clean install -pl :gremlin-python`
>
> Robert Dale
>
>
> On Wed, Mar 6, 2019 at 9:22 AM Robert Dale <ro...@gmail.com> wrote:
>
>> Florian makes a great point about using the image from gremlin-server.
>> It's much lighter weight. It also makes it easier for non-java,
>> client-only devs to use since it doesn't require maven if a published
>> version is used. I've pushed my changes.
>>
>> Thus,
>> # Will use the published gremlin-server image with the 'latest' tag.
>> ./docker/gremlin-server.sh
>>
>> # Will use the published gremlin-server image with the specified version
>> ./docker/gremlin-server.sh 3.4.0
>>
>> # To use the working copy, build gremlin-server and the image. e.g.:
>> mvn clean install -pl :gremlin-server -am && mvn -DskipTests install
>> -Pdocker-images -pl :gremlin-server
>>
>> # You still have to know what the working copy version is. Docker will
>> check local images.
>> ./docker/gremlin-server.sh 3.4.1-SNAPSHOT
>>
>> In all cases, the test scripts are still copied from the working copy so
>> be aware of any version conflicts between gremlin-server version used and
>> working copy version test scripts.
>>
>> I've also retained the original image's ability to override the yaml
>> file. However, the version is required even when using 'latest' tag.
>>
>> # Will start latest published gremlin-server with the default config on
>> port 8182
>> ./docker/gremlin-server.sh latest conf/gremlin-server.yaml
>>
>>
>> Robert Dale
>>
>>
>> On Fri, Mar 1, 2019 at 3:13 PM Florian Hockmann <fh...@florian-hockmann.de>
>> wrote:
>>
>>> Wow, for some reason I didn't see your earlier mail and also the linked
>>> PR. So, I didn't know that there is already some automation to try out.
>>>
>>> That made my attempts to help of course a bit pointless :-/ Sorry
>>> Stephen if I waisted your time with that.
>>>
>>> Regarding our initial discussion here: I think that Testcontainers would
>>> be the best solution for the GLVs (gremlin-driver was probably a bad
>>> example), at least in the long term, as it would also enable us to test
>>> modification steps without breaking other tests. Nevertheless, I agree
>>> that a simpler approach might be the better solution for now and it is
>>> definitely an improvement over our current situation.
>>>
>>> I will probably wait for testcontainers-dotnet to support using images
>>> that are only available locally (so we can test with a locally built
>>> Gremlin Server) and then test out how good it works with our build
>>> process.
>>>
>>> Am 01.03.2019 um 20:50 schrieb Robert Dale:
>>> > I force pushed a fix. Wasn't listening on all nets. Also added a
>>> banner
>>> > for IP address that will appear like so:
>>> > ...
>>> > Successfully built b00dc1f2db35
>>> > #######################
>>> > IP is 172.17.0.2
>>> > #######################
>>> > [INFO] GremlinServer - 3.4.1-SNAPSHOT
>>> > ...
>>> >
>>> > Robert Dale
>>> >
>>> >
>>> > On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <
>>> fh@florian-hockmann.de>
>>> > wrote:
>>> >
>>> >> I don't know what the problem could be, but two things I can think of
>>> >> that could be helpful to find the problem:
>>> >>
>>> >> * Connect to the container (via docker exec -it [container_id] bash)
>>> >> and then confirm with netstat (which you have to install, `apk
>>> >> update; apk add ospd-netstat` should work) that port 45940 is
>>> >> actually open and that the foreign address is 0.0.0.0 (`netstat
>>> -tlpn`)
>>> >> * Check whether you can connect to that port in general without
>>> adding
>>> >> complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>>> >> localhost 45940`.
>>> >>
>>> >> Simply restarting Docker could also help. (At least on Windows some
>>> >> strange Docker networks problems are often gone after a restart.)
>>> >>
>>> >> You're probably using Gremlin Console from your host system and not
>>> with
>>> >> our Docker image, right?
>>> >>
>>> >> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
>>> >>> i tried adding that -p thing to the run command in the script, but i
>>> >> still
>>> >>> couldn't connect to it with Gremlin Console
>>> >>>
>>> >>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <
>>> fh@florian-hockmann.de>
>>> >>> wrote:
>>> >>>
>>> >>>> Did you start the container with port mapping (e.g., docker run -p
>>> >>>> 45940:45940 [image-name]) or how are you trying to connect to the
>>> >>>> container?
>>> >>>>
>>> >>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>>> >>>>> The server starts but something seems messed up with the ports.
>>> It's
>>> >>>> like I
>>> >>>>> can't connect to the 45940 port from the host. I tried a couple
>>> things
>>> >> I
>>> >>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's
>>> >> probably
>>> >>>>> something simple...any ideas?
>>> >>>>>
>>> >>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com>
>>> wrote:
>>> >>>>>
>>> >>>>>> I made a first attempt. Let me know what you think.
>>> >>>>>> https://github.com/apache/tinkerpop/pull/1075
>>> >>>>>>
>>> >>>>>> Robert Dale
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
>>> >> spmallette@gmail.com
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>>> If we agree on Testcontainers in general, then I can give this
>>> a
>>> >> try
>>> >>>>>> and
>>> >>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>>> >> Server
>>> >>>>>> for
>>> >>>>>>> testing (gremlin-driver maybe?).
>>> >>>>>>>
>>> >>>>>>> The java integration tests for Gremlin Server flip a lot of
>>> >>>> configuration
>>> >>>>>>> bits per test. I don't know how that works with testcontainers
>>> and if
>>> >>>> it
>>> >>>>>>> will make our testing there any better. There are also a number
>>> of
>>> >>>> tests
>>> >>>>>>> that stop/start the server under different conditions - not sure
>>> how
>>> >>>> that
>>> >>>>>>> works with testcontainers either. Finally, wouldn't we expect
>>> adding
>>> >>>>>> docker
>>> >>>>>>> to the mix to slow down our integration test runs for java?
>>> >>>>>>>
>>> >>>>>>> I don't want to seem like i'm completely against testcontainers
>>> btw.
>>> >> I
>>> >>>>>> just
>>> >>>>>>> wonder if a script doesn't give us a 100% better experience than
>>> what
>>> >>>> we
>>> >>>>>>> have now (which is nothing). If so, and the script is easy and
>>> >>>>>> zero-impact
>>> >>>>>>> without new dependencies, then I think I like that approach.
>>> Also, It
>>> >>>>>> would
>>> >>>>>>> be nice to have the script for other sorts of manual testing,
>>> such as
>>> >>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>> >>>>>>> testcontainers for GLVs I suppose that can be done too, but for
>>> my
>>> >>>>>>> purposes, since I was the one who asked for help with this
>>> initially,
>>> >>>>>>> testcontainers doesn't sound like something that is an absolute
>>> >>>>>> requirement
>>> >>>>>>> for my needs.
>>> >>>>>>>
>>> >>>>>>> thanks robert/florian for all the discussion and help -
>>> appreciated.
>>> >>>>>>>
>>> >>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>>> >>>>>> fh@florian-hockmann.de>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> Yes, except for the detail that Testcontainers could make the
>>> script
>>> >>>>>>>> obsolete if it is also used to build the image by itself. That
>>> would
>>> >>>>>> have
>>> >>>>>>>> the advantage that developers wouldn't have to execute the
>>> script to
>>> >>>>>>> build
>>> >>>>>>>> the image. (Executing the script with Testcontainers could also
>>> be a
>>> >>>>>>>> possibility, but I'm not sure how good that works.)
>>> >>>>>>>>
>>> >>>>>>>> If we agree on Testcontainers in general, then I can give this
>>> a try
>>> >>>>>> and
>>> >>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>>> >>>> Server
>>> >>>>>>> for
>>> >>>>>>>> testing (gremlin-driver maybe?).
>>> >>>>>>>> With such a PR it would probably also be easier for others to
>>> see
>>> >> what
>>> >>>>>>>> advantages / disadvantages using Testcontainers would have.
>>> >>>>>>>>
>>> >>>>>>>> I think using a Java project first makes most sense as we can
>>> use
>>> >> all
>>> >>>>>>>> functionality Testcontainers has to offer there. The downside
>>> is of
>>> >>>>>>> course
>>> >>>>>>>> that we might not have all the functionality afterwards for
>>> other
>>> >>>>>>> languages
>>> >>>>>>>> and then we might need to extend the Testcontainers libraries
>>> for
>>> >> the
>>> >>>>>>>> different languages. (My assumption here is that we won't need
>>> that
>>> >>>>>> much
>>> >>>>>>>> functionality so it should be limited to 1 or 2 missing
>>> features.)
>>> >>>>>>>> Alternatively, we could also start with a GLV where the
>>> >> Testcontainers
>>> >>>>>>>> library doesn't have the same functionality which probably means
>>> >> that
>>> >>>>>> we
>>> >>>>>>>> need a workaround to get an image of our current Gremlin Server
>>> >>>> version
>>> >>>>>>> for
>>> >>>>>>>> testing.
>>> >>>>>>>>
>>> >>>>>>>> -----Ursprüngliche Nachricht-----
>>> >>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>> >>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>> >>>>>>>> An: dev@tinkerpop.apache.org
>>> >>>>>>>> Betreff: Re: docker and GLV development
>>> >>>>>>>>
>>> >>>>>>>> I think there are two components here.
>>> >>>>>>>> 1) Script to build the dockerized test server. Includes ability
>>> to
>>> >>>>>> start
>>> >>>>>>>> container manually, externally from tests.
>>> >>>>>>>> 2) Testcontainers integration which uses the docker image from
>>> #1.
>>> >>>>>>>> Testcontainers could also include executing the script in #1 to
>>> >> build
>>> >>>>>> the
>>> >>>>>>>> container.
>>> >>>>>>>>
>>> >>>>>>>> Does this sound right to you?
>>> >>>>>>>>
>>> >>>>>>>> Robert Dale
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>>> >>>>>> fh@florian-hockmann.de
>>> >>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>>> I do like the idea of being able to test against older
>>> versions
>>> >>>>>>> though.
>>> >>>>>>>>> That shouldn't be a problem in general as we could use multiple
>>> >>>>>>>>> different images. See for example this PR that adds
>>> Testcontainers
>>> >> to
>>> >>>>>>>>> JanusGraph to test different Elasticsearch versions:
>>> >>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>> >>>>>>>>>
>>> >>>>>>>>>> Yes, an on-the-fly container would work
>>> >>>>>>>>> We could however also use an on-the-fly container with
>>> >>>>>> Testcontainers.
>>> >>>>>>>>> We would just need some automation to build the image in the
>>> >>>>>>> background.
>>> >>>>>>>>> What if we would build an image during the usual Maven build
>>> for
>>> >>>>>>>>> testing and give it a descriptive tag like "testing". Then, we
>>> >> could
>>> >>>>>>>>> use that tag with Testcontainers in addition to tags for
>>> already
>>> >>>>>>>> released versions.
>>> >>>>>>>>> Testcontainers-Java can apparently even build images by
>>> itself. So,
>>> >>>>>>>>> that could also be a possibility to get an image with the
>>> latest
>>> >>>>>>>>> Gremlin Server with the current (unreleased) changes.
>>> >>>>>>>>>
>>> >>>>>>>>> In general, I would really like to get away from a situation
>>> where
>>> >> we
>>> >>>>>>>>> have to start some script in the background before tests can be
>>> >>>>>>> executed.
>>> >>>>>>>>> @Robert:
>>> >>>>>>>>>> What is the test configuration/data for GLVs? Could you list
>>> the
>>> >>>>>>>>> paths/files?
>>> >>>>>>>>>
>>> >>>>>>>>> For Gremlin.Net for example, you can find the automation
>>> happening
>>> >> in
>>> >>>>>>>>> the pom.xml of the testing part:
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>
>>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>> >>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>> >>>>>>>>>
>>> >>>>>>>>> It calls the test-server-start.groovy script before the tests
>>> are
>>> >>>>>>>>> executed and afterwards test-server-stop.groovy script to stop
>>> the
>>> >>>>>>>> server again.
>>> >>>>>>>>> This has the obvious downside that the tests can only be
>>> executed
>>> >>>>>> from
>>> >>>>>>>>> the Maven build which isn't the build tool .NET developers
>>> would
>>> >> use
>>> >>>>>>>>> and also not the tool the IDE will use (the situation is
>>> basically
>>> >>>>>> the
>>> >>>>>>>>> same for other GLVs).
>>> >>>>>>>>>
>>> >>>>>>>>> -----Ursprüngliche Nachricht-----
>>> >>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>> >>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>> >>>>>>>>> An: dev@tinkerpop.apache.org
>>> >>>>>>>>> Betreff: Re: docker and GLV development
>>> >>>>>>>>>
>>> >>>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>> >>>>>>>>>
>>> >>>>>>>>> What is the test configuration/data for GLVs? Could you list
>>> the
>>> >>>>>>>>> paths/files?
>>> >>>>>>>>>
>>> >>>>>>>>> Robert Dale
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>> >>>>>>>>> <sp...@gmail.com>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> Yes, an on-the-fly container would work assuming it used the
>>> test
>>> >>>>>>>>>> configuration/data for GLVs. That's why i suggested the
>>> command as
>>> >>>>>> I
>>> >>>>>>>>>> did because it was similar to docker/build.sh.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <
>>> robdale@gmail.com>
>>> >>>>>>> wrote:
>>> >>>>>>>>>>> Do you want to create a docker container on-the-fly using the
>>> >>>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Robert Dale
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>> >>>>>>>>>>> <sp...@gmail.com>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> testcontainers seems neat as it gives some good integration
>>> and
>>> >>>>>>>>>>> automation,
>>> >>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so
>>> great.
>>> >>>>>>>>>>>> There are times when changes are occurring on both client
>>> and
>>> >>>>>>>>>>>> server so a local up-to-the-moment image would be
>>> necessary. I
>>> >>>>>>>>>>>> also think we might want
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures
>>> out
>>> >>>>>>>>>>>> more
>>> >>>>>>>>>>> clearly
>>> >>>>>>>>>>>> how those play into projects (there is a bit of discussion
>>> >>>>>> going
>>> >>>>>>>>>>>> on
>>> >>>>>>>>>> with
>>> >>>>>>>>>>>> that now). I do like the idea of being able to test against
>>> >>>>>>>>>>>> older
>>> >>>>>>>>>>> versions
>>> >>>>>>>>>>>> though. Ensuring backward compatibility.is a nice thought.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>> >>>>>>>>>> fh@florian-hockmann.de
>>> >>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>>> >>>>>> tests
>>> >>>>>>>>>>>>> for Gremlin.Net from my IDE.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>> >>>>>>>>>>>>> problems
>>> >>>>>>>>>>> like
>>> >>>>>>>>>>>>> this:
>>> >>>>>>>>>>>>> https://www.testcontainers.org/
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> The basic idea of Testcontainers is that you define a
>>> Docker
>>> >>>>>>>>>> container
>>> >>>>>>>>>>> in
>>> >>>>>>>>>>>>> your tests for whatever external dependencies you need.
>>> >>>>>>>>>>>>> Databases
>>> >>>>>>>>>> are a
>>> >>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>> >>>>>>>>>>>>> Gremlin
>>> >>>>>>>>>>> Server
>>> >>>>>>>>>>>>> would be such a container.
>>> >>>>>>>>>>>>> Testcontainers then uses the ability of the testing
>>> framework
>>> >>>>>>>> (e.g.
>>> >>>>>>>>>>>> JUnit)
>>> >>>>>>>>>>>>> to start the Docker container for the tests and waits until
>>> >>>>>>>>>>>>> they are
>>> >>>>>>>>>>>> fully
>>> >>>>>>>>>>>>> started (how they are started can be configured for each
>>> >>>>>>>>>>>>> container differently). After the tests have completed, it
>>> >>>>>>>>>>>>> also stops the
>>> >>>>>>>>>>>> containers
>>> >>>>>>>>>>>>> again.
>>> >>>>>>>>>>>>> That way, you don't have to manage the containers manually
>>> >>>>>>>> anymore.
>>> >>>>>>>>>>>>> Everything is done in the background by the IDE / the build
>>> >>>>>>> tool.
>>> >>>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>>> >>>>>>>>>>>>> also have libraries for different languages, including the
>>> >>>>>>>>>>>>> ones for which we
>>> >>>>>>>>>> have
>>> >>>>>>>>>>>>> GLVs:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> testcontainers-python:
>>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>> >>>>>>>>>>>>> testcontainers-dotnet
>>> >>>>>>>>>>>>> <
>>> >>>>>>
>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>> >>>>>>>>>> s-
>>> >>>>>>>>>> dotnet
>>> >>>>>>>>>>>>> :
>>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>> >>>>>>>>>>>>> testcontainers-node
>>> >>>>>>>>>>>>> <
>>> >>>>>>
>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>> >>>>>>>>>> s-
>>> >>>>>>>>>> node
>>> >>>>>>>>>>>>> :
>>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Testcontainers-java is the most advanced library of these
>>> of
>>> >>>>>>>>> course.
>>> >>>>>>>>>>> The
>>> >>>>>>>>>>>>> others still lack some functionality, but I guess that we
>>> >>>>>>>>>>>>> could also
>>> >>>>>>>>>>> just
>>> >>>>>>>>>>>>> contribute to those libraries if an important feature is
>>> >>>>>>>>>>>>> missing for
>>> >>>>>>>>>>> us.
>>> >>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>> >>>>>>>>>>> testcontainers-dotnet
>>> >>>>>>>>>>>>> unfortunately can only use images that are available in
>>> some
>>> >>>>>>>>>> registry.
>>> >>>>>>>>>>>> So,
>>> >>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>> >>>>>>>>>>>>> DockerHub
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>> use
>>> >>>>>>>>>>>>> it in our tests. That means that we could either only test
>>> >>>>>> the
>>> >>>>>>>>>>>>> GLVs
>>> >>>>>>>>>>>> against
>>> >>>>>>>>>>>>> already released versions or we could publish something
>>> like
>>> >>>>>>>>>>>>> nightly
>>> >>>>>>>>>>>> build
>>> >>>>>>>>>>>>> images of Gremlin Server.
>>> >>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>> >>>>>>>>>>>>> able to
>>> >>>>>>>>>> use
>>> >>>>>>>>>>>>> images that are only available locally (testcontainers-java
>>> >>>>>>>>>>>>> already
>>> >>>>>>>>>> has
>>> >>>>>>>>>>>>> that capability), but now that I think more about this, I
>>> >>>>>>>>>>>>> think that
>>> >>>>>>>>>> it
>>> >>>>>>>>>>>>> makes actually more sense to use images that are available
>>> in
>>> >>>>>>>>>>>>> the
>>> >>>>>>>>>>>> registry
>>> >>>>>>>>>>>>> as every developer would otherwise have to build test
>>> images
>>> >>>>>>>>>>>>> first
>>> >>>>>>>>>>>> locally
>>> >>>>>>>>>>>>> in order to use them for tests.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers
>>> to
>>> >>>>>>>>>>>>> see
>>> >>>>>>>>>>> whether
>>> >>>>>>>>>>>>> you agree that it's a good solution to our problem and then
>>> >>>>>> we
>>> >>>>>>>>>>>>> can
>>> >>>>>>>>>> see
>>> >>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>>> >>>>>> or
>>> >>>>>>>>>>>>> how we
>>> >>>>>>>>>>>> want
>>> >>>>>>>>>>>>> to handle that.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>>> >>>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>> >>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>> >>>>>>>>>>>>> An: dev@tinkerpop.apache.org
>>> >>>>>>>>>>>>> Betreff: docker and GLV development
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>> >>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love
>>> to
>>> >>>>>>>>>>>>> be able to do
>>> >>>>>>>>>>>> something
>>> >>>>>>>>>>>>> like:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> docker/gremlin-server.sh -test
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> which would start Gremlin Server with our standard test
>>> >>>>>>>>>>>>> configuration (currently built into the maven tool chain,
>>> but
>>> >>>>>>>>>>>>> I think that could be extracted without too much
>>> difficulty).
>>> >>>>>>>>>>>>> That way, we could easily
>>> >>>>>>>>>> open
>>> >>>>>>>>>>> up
>>> >>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly
>>> in
>>> >>>>>>>>>>>>> there,
>>> >>>>>>>>>>> use
>>> >>>>>>>>>>>>> the debugger, etc.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Sound useful? and if so, is that something that is
>>> >>>>>>>>>>>>> easy/possible? and
>>> >>>>>>>>>>>> then
>>> >>>>>>>>>>>>> further....any volunteers with the know-how to get that
>>> setup
>>> >>>>>>>>>>>>> (i'm
>>> >>>>>>>>>>> happy
>>> >>>>>>>>>>>> to
>>> >>>>>>>>>>>>> assist however i can)?
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for
>>> GLV
>>> >>>>>>>>>>> development
>>> >>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>>> >>>>>> mind
>>> >>>>>>>>>>>>> using
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>>>> maven integration but i feel like i could be more efficient
>>> >>>>>>>>>>>>> with a
>>> >>>>>>>>>> more
>>> >>>>>>>>>>>>> streamlined setup.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>>
>>>
Re: docker and GLV development
Posted by Robert Dale <ro...@gmail.com>.
I just pushed a change that allows python tests to pass against the docker
image.
This will install the gremlin-python script support on the server so that's
an extra download to wait for.
I had to update the localhost references to point to the docker IP
in gremlin-python: grep -Rl localhost gremlin-python/src/main/jython/ |
xargs sed -i 's/localhost/172.17.0.2/g'
Then I could `mvn clean install -pl :gremlin-python`
Robert Dale
On Wed, Mar 6, 2019 at 9:22 AM Robert Dale <ro...@gmail.com> wrote:
> Florian makes a great point about using the image from gremlin-server.
> It's much lighter weight. It also makes it easier for non-java,
> client-only devs to use since it doesn't require maven if a published
> version is used. I've pushed my changes.
>
> Thus,
> # Will use the published gremlin-server image with the 'latest' tag.
> ./docker/gremlin-server.sh
>
> # Will use the published gremlin-server image with the specified version
> ./docker/gremlin-server.sh 3.4.0
>
> # To use the working copy, build gremlin-server and the image. e.g.:
> mvn clean install -pl :gremlin-server -am && mvn -DskipTests install
> -Pdocker-images -pl :gremlin-server
>
> # You still have to know what the working copy version is. Docker will
> check local images.
> ./docker/gremlin-server.sh 3.4.1-SNAPSHOT
>
> In all cases, the test scripts are still copied from the working copy so
> be aware of any version conflicts between gremlin-server version used and
> working copy version test scripts.
>
> I've also retained the original image's ability to override the yaml file.
> However, the version is required even when using 'latest' tag.
>
> # Will start latest published gremlin-server with the default config on
> port 8182
> ./docker/gremlin-server.sh latest conf/gremlin-server.yaml
>
>
> Robert Dale
>
>
> On Fri, Mar 1, 2019 at 3:13 PM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>> Wow, for some reason I didn't see your earlier mail and also the linked
>> PR. So, I didn't know that there is already some automation to try out.
>>
>> That made my attempts to help of course a bit pointless :-/ Sorry
>> Stephen if I waisted your time with that.
>>
>> Regarding our initial discussion here: I think that Testcontainers would
>> be the best solution for the GLVs (gremlin-driver was probably a bad
>> example), at least in the long term, as it would also enable us to test
>> modification steps without breaking other tests. Nevertheless, I agree
>> that a simpler approach might be the better solution for now and it is
>> definitely an improvement over our current situation.
>>
>> I will probably wait for testcontainers-dotnet to support using images
>> that are only available locally (so we can test with a locally built
>> Gremlin Server) and then test out how good it works with our build
>> process.
>>
>> Am 01.03.2019 um 20:50 schrieb Robert Dale:
>> > I force pushed a fix. Wasn't listening on all nets. Also added a banner
>> > for IP address that will appear like so:
>> > ...
>> > Successfully built b00dc1f2db35
>> > #######################
>> > IP is 172.17.0.2
>> > #######################
>> > [INFO] GremlinServer - 3.4.1-SNAPSHOT
>> > ...
>> >
>> > Robert Dale
>> >
>> >
>> > On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <fh@florian-hockmann.de
>> >
>> > wrote:
>> >
>> >> I don't know what the problem could be, but two things I can think of
>> >> that could be helpful to find the problem:
>> >>
>> >> * Connect to the container (via docker exec -it [container_id] bash)
>> >> and then confirm with netstat (which you have to install, `apk
>> >> update; apk add ospd-netstat` should work) that port 45940 is
>> >> actually open and that the foreign address is 0.0.0.0 (`netstat
>> -tlpn`)
>> >> * Check whether you can connect to that port in general without
>> adding
>> >> complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>> >> localhost 45940`.
>> >>
>> >> Simply restarting Docker could also help. (At least on Windows some
>> >> strange Docker networks problems are often gone after a restart.)
>> >>
>> >> You're probably using Gremlin Console from your host system and not
>> with
>> >> our Docker image, right?
>> >>
>> >> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
>> >>> i tried adding that -p thing to the run command in the script, but i
>> >> still
>> >>> couldn't connect to it with Gremlin Console
>> >>>
>> >>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <
>> fh@florian-hockmann.de>
>> >>> wrote:
>> >>>
>> >>>> Did you start the container with port mapping (e.g., docker run -p
>> >>>> 45940:45940 [image-name]) or how are you trying to connect to the
>> >>>> container?
>> >>>>
>> >>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>> >>>>> The server starts but something seems messed up with the ports. It's
>> >>>> like I
>> >>>>> can't connect to the 45940 port from the host. I tried a couple
>> things
>> >> I
>> >>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's
>> >> probably
>> >>>>> something simple...any ideas?
>> >>>>>
>> >>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> I made a first attempt. Let me know what you think.
>> >>>>>> https://github.com/apache/tinkerpop/pull/1075
>> >>>>>>
>> >>>>>> Robert Dale
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
>> >> spmallette@gmail.com
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>>> If we agree on Testcontainers in general, then I can give this a
>> >> try
>> >>>>>> and
>> >>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> >> Server
>> >>>>>> for
>> >>>>>>> testing (gremlin-driver maybe?).
>> >>>>>>>
>> >>>>>>> The java integration tests for Gremlin Server flip a lot of
>> >>>> configuration
>> >>>>>>> bits per test. I don't know how that works with testcontainers
>> and if
>> >>>> it
>> >>>>>>> will make our testing there any better. There are also a number of
>> >>>> tests
>> >>>>>>> that stop/start the server under different conditions - not sure
>> how
>> >>>> that
>> >>>>>>> works with testcontainers either. Finally, wouldn't we expect
>> adding
>> >>>>>> docker
>> >>>>>>> to the mix to slow down our integration test runs for java?
>> >>>>>>>
>> >>>>>>> I don't want to seem like i'm completely against testcontainers
>> btw.
>> >> I
>> >>>>>> just
>> >>>>>>> wonder if a script doesn't give us a 100% better experience than
>> what
>> >>>> we
>> >>>>>>> have now (which is nothing). If so, and the script is easy and
>> >>>>>> zero-impact
>> >>>>>>> without new dependencies, then I think I like that approach.
>> Also, It
>> >>>>>> would
>> >>>>>>> be nice to have the script for other sorts of manual testing,
>> such as
>> >>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>> >>>>>>> testcontainers for GLVs I suppose that can be done too, but for my
>> >>>>>>> purposes, since I was the one who asked for help with this
>> initially,
>> >>>>>>> testcontainers doesn't sound like something that is an absolute
>> >>>>>> requirement
>> >>>>>>> for my needs.
>> >>>>>>>
>> >>>>>>> thanks robert/florian for all the discussion and help -
>> appreciated.
>> >>>>>>>
>> >>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>> >>>>>> fh@florian-hockmann.de>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Yes, except for the detail that Testcontainers could make the
>> script
>> >>>>>>>> obsolete if it is also used to build the image by itself. That
>> would
>> >>>>>> have
>> >>>>>>>> the advantage that developers wouldn't have to execute the
>> script to
>> >>>>>>> build
>> >>>>>>>> the image. (Executing the script with Testcontainers could also
>> be a
>> >>>>>>>> possibility, but I'm not sure how good that works.)
>> >>>>>>>>
>> >>>>>>>> If we agree on Testcontainers in general, then I can give this a
>> try
>> >>>>>> and
>> >>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> >>>> Server
>> >>>>>>> for
>> >>>>>>>> testing (gremlin-driver maybe?).
>> >>>>>>>> With such a PR it would probably also be easier for others to see
>> >> what
>> >>>>>>>> advantages / disadvantages using Testcontainers would have.
>> >>>>>>>>
>> >>>>>>>> I think using a Java project first makes most sense as we can use
>> >> all
>> >>>>>>>> functionality Testcontainers has to offer there. The downside is
>> of
>> >>>>>>> course
>> >>>>>>>> that we might not have all the functionality afterwards for other
>> >>>>>>> languages
>> >>>>>>>> and then we might need to extend the Testcontainers libraries for
>> >> the
>> >>>>>>>> different languages. (My assumption here is that we won't need
>> that
>> >>>>>> much
>> >>>>>>>> functionality so it should be limited to 1 or 2 missing
>> features.)
>> >>>>>>>> Alternatively, we could also start with a GLV where the
>> >> Testcontainers
>> >>>>>>>> library doesn't have the same functionality which probably means
>> >> that
>> >>>>>> we
>> >>>>>>>> need a workaround to get an image of our current Gremlin Server
>> >>>> version
>> >>>>>>> for
>> >>>>>>>> testing.
>> >>>>>>>>
>> >>>>>>>> -----Ursprüngliche Nachricht-----
>> >>>>>>>> Von: Robert Dale <ro...@gmail.com>
>> >>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>> >>>>>>>> An: dev@tinkerpop.apache.org
>> >>>>>>>> Betreff: Re: docker and GLV development
>> >>>>>>>>
>> >>>>>>>> I think there are two components here.
>> >>>>>>>> 1) Script to build the dockerized test server. Includes ability
>> to
>> >>>>>> start
>> >>>>>>>> container manually, externally from tests.
>> >>>>>>>> 2) Testcontainers integration which uses the docker image from
>> #1.
>> >>>>>>>> Testcontainers could also include executing the script in #1 to
>> >> build
>> >>>>>> the
>> >>>>>>>> container.
>> >>>>>>>>
>> >>>>>>>> Does this sound right to you?
>> >>>>>>>>
>> >>>>>>>> Robert Dale
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>> >>>>>> fh@florian-hockmann.de
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>>> I do like the idea of being able to test against older versions
>> >>>>>>> though.
>> >>>>>>>>> That shouldn't be a problem in general as we could use multiple
>> >>>>>>>>> different images. See for example this PR that adds
>> Testcontainers
>> >> to
>> >>>>>>>>> JanusGraph to test different Elasticsearch versions:
>> >>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>> >>>>>>>>>
>> >>>>>>>>>> Yes, an on-the-fly container would work
>> >>>>>>>>> We could however also use an on-the-fly container with
>> >>>>>> Testcontainers.
>> >>>>>>>>> We would just need some automation to build the image in the
>> >>>>>>> background.
>> >>>>>>>>> What if we would build an image during the usual Maven build for
>> >>>>>>>>> testing and give it a descriptive tag like "testing". Then, we
>> >> could
>> >>>>>>>>> use that tag with Testcontainers in addition to tags for already
>> >>>>>>>> released versions.
>> >>>>>>>>> Testcontainers-Java can apparently even build images by itself.
>> So,
>> >>>>>>>>> that could also be a possibility to get an image with the latest
>> >>>>>>>>> Gremlin Server with the current (unreleased) changes.
>> >>>>>>>>>
>> >>>>>>>>> In general, I would really like to get away from a situation
>> where
>> >> we
>> >>>>>>>>> have to start some script in the background before tests can be
>> >>>>>>> executed.
>> >>>>>>>>> @Robert:
>> >>>>>>>>>> What is the test configuration/data for GLVs? Could you list
>> the
>> >>>>>>>>> paths/files?
>> >>>>>>>>>
>> >>>>>>>>> For Gremlin.Net for example, you can find the automation
>> happening
>> >> in
>> >>>>>>>>> the pom.xml of the testing part:
>> >>>>>>>>>
>> >>>>>>>>>
>> >> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>> >>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>> >>>>>>>>>
>> >>>>>>>>> It calls the test-server-start.groovy script before the tests
>> are
>> >>>>>>>>> executed and afterwards test-server-stop.groovy script to stop
>> the
>> >>>>>>>> server again.
>> >>>>>>>>> This has the obvious downside that the tests can only be
>> executed
>> >>>>>> from
>> >>>>>>>>> the Maven build which isn't the build tool .NET developers would
>> >> use
>> >>>>>>>>> and also not the tool the IDE will use (the situation is
>> basically
>> >>>>>> the
>> >>>>>>>>> same for other GLVs).
>> >>>>>>>>>
>> >>>>>>>>> -----Ursprüngliche Nachricht-----
>> >>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>> >>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>> >>>>>>>>> An: dev@tinkerpop.apache.org
>> >>>>>>>>> Betreff: Re: docker and GLV development
>> >>>>>>>>>
>> >>>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>> >>>>>>>>>
>> >>>>>>>>> What is the test configuration/data for GLVs? Could you list
>> the
>> >>>>>>>>> paths/files?
>> >>>>>>>>>
>> >>>>>>>>> Robert Dale
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>> >>>>>>>>> <sp...@gmail.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Yes, an on-the-fly container would work assuming it used the
>> test
>> >>>>>>>>>> configuration/data for GLVs. That's why i suggested the
>> command as
>> >>>>>> I
>> >>>>>>>>>> did because it was similar to docker/build.sh.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <robdale@gmail.com
>> >
>> >>>>>>> wrote:
>> >>>>>>>>>>> Do you want to create a docker container on-the-fly using the
>> >>>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Robert Dale
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>> >>>>>>>>>>> <sp...@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> testcontainers seems neat as it gives some good integration
>> and
>> >>>>>>>>>>> automation,
>> >>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>> >>>>>>>>>>>> There are times when changes are occurring on both client and
>> >>>>>>>>>>>> server so a local up-to-the-moment image would be necessary.
>> I
>> >>>>>>>>>>>> also think we might want
>> >>>>>>>>>> to
>> >>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>> >>>>>>>>>>>> more
>> >>>>>>>>>>> clearly
>> >>>>>>>>>>>> how those play into projects (there is a bit of discussion
>> >>>>>> going
>> >>>>>>>>>>>> on
>> >>>>>>>>>> with
>> >>>>>>>>>>>> that now). I do like the idea of being able to test against
>> >>>>>>>>>>>> older
>> >>>>>>>>>>> versions
>> >>>>>>>>>>>> though. Ensuring backward compatibility.is a nice thought.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>> >>>>>>>>>> fh@florian-hockmann.de
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>> >>>>>> tests
>> >>>>>>>>>>>>> for Gremlin.Net from my IDE.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>> >>>>>>>>>>>>> problems
>> >>>>>>>>>>> like
>> >>>>>>>>>>>>> this:
>> >>>>>>>>>>>>> https://www.testcontainers.org/
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>> >>>>>>>>>> container
>> >>>>>>>>>>> in
>> >>>>>>>>>>>>> your tests for whatever external dependencies you need.
>> >>>>>>>>>>>>> Databases
>> >>>>>>>>>> are a
>> >>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>> >>>>>>>>>>>>> Gremlin
>> >>>>>>>>>>> Server
>> >>>>>>>>>>>>> would be such a container.
>> >>>>>>>>>>>>> Testcontainers then uses the ability of the testing
>> framework
>> >>>>>>>> (e.g.
>> >>>>>>>>>>>> JUnit)
>> >>>>>>>>>>>>> to start the Docker container for the tests and waits until
>> >>>>>>>>>>>>> they are
>> >>>>>>>>>>>> fully
>> >>>>>>>>>>>>> started (how they are started can be configured for each
>> >>>>>>>>>>>>> container differently). After the tests have completed, it
>> >>>>>>>>>>>>> also stops the
>> >>>>>>>>>>>> containers
>> >>>>>>>>>>>>> again.
>> >>>>>>>>>>>>> That way, you don't have to manage the containers manually
>> >>>>>>>> anymore.
>> >>>>>>>>>>>>> Everything is done in the background by the IDE / the build
>> >>>>>>> tool.
>> >>>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>> >>>>>>>>>>>>> also have libraries for different languages, including the
>> >>>>>>>>>>>>> ones for which we
>> >>>>>>>>>> have
>> >>>>>>>>>>>>> GLVs:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> testcontainers-python:
>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>> >>>>>>>>>>>>> testcontainers-dotnet
>> >>>>>>>>>>>>> <
>> >>>>>>
>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>> >>>>>>>>>> s-
>> >>>>>>>>>> dotnet
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>> >>>>>>>>>>>>> testcontainers-node
>> >>>>>>>>>>>>> <
>> >>>>>>
>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>> >>>>>>>>>> s-
>> >>>>>>>>>> node
>> >>>>>>>>>>>>> :
>> >>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Testcontainers-java is the most advanced library of these of
>> >>>>>>>>> course.
>> >>>>>>>>>>> The
>> >>>>>>>>>>>>> others still lack some functionality, but I guess that we
>> >>>>>>>>>>>>> could also
>> >>>>>>>>>>> just
>> >>>>>>>>>>>>> contribute to those libraries if an important feature is
>> >>>>>>>>>>>>> missing for
>> >>>>>>>>>>> us.
>> >>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>> >>>>>>>>>>> testcontainers-dotnet
>> >>>>>>>>>>>>> unfortunately can only use images that are available in some
>> >>>>>>>>>> registry.
>> >>>>>>>>>>>> So,
>> >>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>> >>>>>>>>>>>>> DockerHub
>> >>>>>>>>>> to
>> >>>>>>>>>>>> use
>> >>>>>>>>>>>>> it in our tests. That means that we could either only test
>> >>>>>> the
>> >>>>>>>>>>>>> GLVs
>> >>>>>>>>>>>> against
>> >>>>>>>>>>>>> already released versions or we could publish something like
>> >>>>>>>>>>>>> nightly
>> >>>>>>>>>>>> build
>> >>>>>>>>>>>>> images of Gremlin Server.
>> >>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>> >>>>>>>>>>>>> able to
>> >>>>>>>>>> use
>> >>>>>>>>>>>>> images that are only available locally (testcontainers-java
>> >>>>>>>>>>>>> already
>> >>>>>>>>>> has
>> >>>>>>>>>>>>> that capability), but now that I think more about this, I
>> >>>>>>>>>>>>> think that
>> >>>>>>>>>> it
>> >>>>>>>>>>>>> makes actually more sense to use images that are available
>> in
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>> registry
>> >>>>>>>>>>>>> as every developer would otherwise have to build test images
>> >>>>>>>>>>>>> first
>> >>>>>>>>>>>> locally
>> >>>>>>>>>>>>> in order to use them for tests.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>> >>>>>>>>>>>>> see
>> >>>>>>>>>>> whether
>> >>>>>>>>>>>>> you agree that it's a good solution to our problem and then
>> >>>>>> we
>> >>>>>>>>>>>>> can
>> >>>>>>>>>> see
>> >>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>> >>>>>> or
>> >>>>>>>>>>>>> how we
>> >>>>>>>>>>>> want
>> >>>>>>>>>>>>> to handle that.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>> >>>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>> >>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>> >>>>>>>>>>>>> An: dev@tinkerpop.apache.org
>> >>>>>>>>>>>>> Betreff: docker and GLV development
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>> >>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love
>> to
>> >>>>>>>>>>>>> be able to do
>> >>>>>>>>>>>> something
>> >>>>>>>>>>>>> like:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> docker/gremlin-server.sh -test
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> which would start Gremlin Server with our standard test
>> >>>>>>>>>>>>> configuration (currently built into the maven tool chain,
>> but
>> >>>>>>>>>>>>> I think that could be extracted without too much
>> difficulty).
>> >>>>>>>>>>>>> That way, we could easily
>> >>>>>>>>>> open
>> >>>>>>>>>>> up
>> >>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly
>> in
>> >>>>>>>>>>>>> there,
>> >>>>>>>>>>> use
>> >>>>>>>>>>>>> the debugger, etc.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Sound useful? and if so, is that something that is
>> >>>>>>>>>>>>> easy/possible? and
>> >>>>>>>>>>>> then
>> >>>>>>>>>>>>> further....any volunteers with the know-how to get that
>> setup
>> >>>>>>>>>>>>> (i'm
>> >>>>>>>>>>> happy
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>>> assist however i can)?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for
>> GLV
>> >>>>>>>>>>> development
>> >>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>> >>>>>> mind
>> >>>>>>>>>>>>> using
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>> maven integration but i feel like i could be more efficient
>> >>>>>>>>>>>>> with a
>> >>>>>>>>>> more
>> >>>>>>>>>>>>> streamlined setup.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>>
>>
Re: docker and GLV development
Posted by Florian Hockmann <fh...@florian-hockmann.de>.
Wow, for some reason I didn't see your earlier mail and also the linked
PR. So, I didn't know that there is already some automation to try out.
That made my attempts to help of course a bit pointless :-/ Sorry
Stephen if I waisted your time with that.
Regarding our initial discussion here: I think that Testcontainers would
be the best solution for the GLVs (gremlin-driver was probably a bad
example), at least in the long term, as it would also enable us to test
modification steps without breaking other tests. Nevertheless, I agree
that a simpler approach might be the better solution for now and it is
definitely an improvement over our current situation.
I will probably wait for testcontainers-dotnet to support using images
that are only available locally (so we can test with a locally built
Gremlin Server) and then test out how good it works with our build process.
Am 01.03.2019 um 20:50 schrieb Robert Dale:
> I force pushed a fix. Wasn't listening on all nets. Also added a banner
> for IP address that will appear like so:
> ...
> Successfully built b00dc1f2db35
> #######################
> IP is 172.17.0.2
> #######################
> [INFO] GremlinServer - 3.4.1-SNAPSHOT
> ...
>
> Robert Dale
>
>
> On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>> I don't know what the problem could be, but two things I can think of
>> that could be helpful to find the problem:
>>
>> * Connect to the container (via docker exec -it [container_id] bash)
>> and then confirm with netstat (which you have to install, `apk
>> update; apk add ospd-netstat` should work) that port 45940 is
>> actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`)
>> * Check whether you can connect to that port in general without adding
>> complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
>> localhost 45940`.
>>
>> Simply restarting Docker could also help. (At least on Windows some
>> strange Docker networks problems are often gone after a restart.)
>>
>> You're probably using Gremlin Console from your host system and not with
>> our Docker image, right?
>>
>> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
>>> i tried adding that -p thing to the run command in the script, but i
>> still
>>> couldn't connect to it with Gremlin Console
>>>
>>> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
>>> wrote:
>>>
>>>> Did you start the container with port mapping (e.g., docker run -p
>>>> 45940:45940 [image-name]) or how are you trying to connect to the
>>>> container?
>>>>
>>>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>>>>> The server starts but something seems messed up with the ports. It's
>>>> like I
>>>>> can't connect to the 45940 port from the host. I tried a couple things
>> I
>>>>> saw on stackoverflow but nothing i tried seemed to fix it. it's
>> probably
>>>>> something simple...any ideas?
>>>>>
>>>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
>>>>>
>>>>>> I made a first attempt. Let me know what you think.
>>>>>> https://github.com/apache/tinkerpop/pull/1075
>>>>>>
>>>>>> Robert Dale
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
>> spmallette@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>>> If we agree on Testcontainers in general, then I can give this a
>> try
>>>>>> and
>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> Server
>>>>>> for
>>>>>>> testing (gremlin-driver maybe?).
>>>>>>>
>>>>>>> The java integration tests for Gremlin Server flip a lot of
>>>> configuration
>>>>>>> bits per test. I don't know how that works with testcontainers and if
>>>> it
>>>>>>> will make our testing there any better. There are also a number of
>>>> tests
>>>>>>> that stop/start the server under different conditions - not sure how
>>>> that
>>>>>>> works with testcontainers either. Finally, wouldn't we expect adding
>>>>>> docker
>>>>>>> to the mix to slow down our integration test runs for java?
>>>>>>>
>>>>>>> I don't want to seem like i'm completely against testcontainers btw.
>> I
>>>>>> just
>>>>>>> wonder if a script doesn't give us a 100% better experience than what
>>>> we
>>>>>>> have now (which is nothing). If so, and the script is easy and
>>>>>> zero-impact
>>>>>>> without new dependencies, then I think I like that approach. Also, It
>>>>>> would
>>>>>>> be nice to have the script for other sorts of manual testing, such as
>>>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>>>>>> testcontainers for GLVs I suppose that can be done too, but for my
>>>>>>> purposes, since I was the one who asked for help with this initially,
>>>>>>> testcontainers doesn't sound like something that is an absolute
>>>>>> requirement
>>>>>>> for my needs.
>>>>>>>
>>>>>>> thanks robert/florian for all the discussion and help - appreciated.
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>>>>>> fh@florian-hockmann.de>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, except for the detail that Testcontainers could make the script
>>>>>>>> obsolete if it is also used to build the image by itself. That would
>>>>>> have
>>>>>>>> the advantage that developers wouldn't have to execute the script to
>>>>>>> build
>>>>>>>> the image. (Executing the script with Testcontainers could also be a
>>>>>>>> possibility, but I'm not sure how good that works.)
>>>>>>>>
>>>>>>>> If we agree on Testcontainers in general, then I can give this a try
>>>>>> and
>>>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>>>> Server
>>>>>>> for
>>>>>>>> testing (gremlin-driver maybe?).
>>>>>>>> With such a PR it would probably also be easier for others to see
>> what
>>>>>>>> advantages / disadvantages using Testcontainers would have.
>>>>>>>>
>>>>>>>> I think using a Java project first makes most sense as we can use
>> all
>>>>>>>> functionality Testcontainers has to offer there. The downside is of
>>>>>>> course
>>>>>>>> that we might not have all the functionality afterwards for other
>>>>>>> languages
>>>>>>>> and then we might need to extend the Testcontainers libraries for
>> the
>>>>>>>> different languages. (My assumption here is that we won't need that
>>>>>> much
>>>>>>>> functionality so it should be limited to 1 or 2 missing features.)
>>>>>>>> Alternatively, we could also start with a GLV where the
>> Testcontainers
>>>>>>>> library doesn't have the same functionality which probably means
>> that
>>>>>> we
>>>>>>>> need a workaround to get an image of our current Gremlin Server
>>>> version
>>>>>>> for
>>>>>>>> testing.
>>>>>>>>
>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>> Betreff: Re: docker and GLV development
>>>>>>>>
>>>>>>>> I think there are two components here.
>>>>>>>> 1) Script to build the dockerized test server. Includes ability to
>>>>>> start
>>>>>>>> container manually, externally from tests.
>>>>>>>> 2) Testcontainers integration which uses the docker image from #1.
>>>>>>>> Testcontainers could also include executing the script in #1 to
>> build
>>>>>> the
>>>>>>>> container.
>>>>>>>>
>>>>>>>> Does this sound right to you?
>>>>>>>>
>>>>>>>> Robert Dale
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>>>>>> fh@florian-hockmann.de
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>> I do like the idea of being able to test against older versions
>>>>>>> though.
>>>>>>>>> That shouldn't be a problem in general as we could use multiple
>>>>>>>>> different images. See for example this PR that adds Testcontainers
>> to
>>>>>>>>> JanusGraph to test different Elasticsearch versions:
>>>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>>>>>>>>
>>>>>>>>>> Yes, an on-the-fly container would work
>>>>>>>>> We could however also use an on-the-fly container with
>>>>>> Testcontainers.
>>>>>>>>> We would just need some automation to build the image in the
>>>>>>> background.
>>>>>>>>> What if we would build an image during the usual Maven build for
>>>>>>>>> testing and give it a descriptive tag like "testing". Then, we
>> could
>>>>>>>>> use that tag with Testcontainers in addition to tags for already
>>>>>>>> released versions.
>>>>>>>>> Testcontainers-Java can apparently even build images by itself. So,
>>>>>>>>> that could also be a possibility to get an image with the latest
>>>>>>>>> Gremlin Server with the current (unreleased) changes.
>>>>>>>>>
>>>>>>>>> In general, I would really like to get away from a situation where
>> we
>>>>>>>>> have to start some script in the background before tests can be
>>>>>>> executed.
>>>>>>>>> @Robert:
>>>>>>>>>> What is the test configuration/data for GLVs? Could you list the
>>>>>>>>> paths/files?
>>>>>>>>>
>>>>>>>>> For Gremlin.Net for example, you can find the automation happening
>> in
>>>>>>>>> the pom.xml of the testing part:
>>>>>>>>>
>>>>>>>>>
>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>>>>>>>>
>>>>>>>>> It calls the test-server-start.groovy script before the tests are
>>>>>>>>> executed and afterwards test-server-stop.groovy script to stop the
>>>>>>>> server again.
>>>>>>>>> This has the obvious downside that the tests can only be executed
>>>>>> from
>>>>>>>>> the Maven build which isn't the build tool .NET developers would
>> use
>>>>>>>>> and also not the tool the IDE will use (the situation is basically
>>>>>> the
>>>>>>>>> same for other GLVs).
>>>>>>>>>
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>> Betreff: Re: docker and GLV development
>>>>>>>>>
>>>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>>>>>>>>
>>>>>>>>> What is the test configuration/data for GLVs? Could you list the
>>>>>>>>> paths/files?
>>>>>>>>>
>>>>>>>>> Robert Dale
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>>>>>>>> <sp...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, an on-the-fly container would work assuming it used the test
>>>>>>>>>> configuration/data for GLVs. That's why i suggested the command as
>>>>>> I
>>>>>>>>>> did because it was similar to docker/build.sh.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>> Do you want to create a docker container on-the-fly using the
>>>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>>>>>>>>>>>
>>>>>>>>>>> Robert Dale
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>>>>>>>>>> <sp...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> testcontainers seems neat as it gives some good integration and
>>>>>>>>>>> automation,
>>>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>>>>>>>>>>>> There are times when changes are occurring on both client and
>>>>>>>>>>>> server so a local up-to-the-moment image would be necessary. I
>>>>>>>>>>>> also think we might want
>>>>>>>>>> to
>>>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>>>>>>>>>>>> more
>>>>>>>>>>> clearly
>>>>>>>>>>>> how those play into projects (there is a bit of discussion
>>>>>> going
>>>>>>>>>>>> on
>>>>>>>>>> with
>>>>>>>>>>>> that now). I do like the idea of being able to test against
>>>>>>>>>>>> older
>>>>>>>>>>> versions
>>>>>>>>>>>> though. Ensuring backward compatibility.is a nice thought.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>>>>>>>>> fh@florian-hockmann.de
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>>>>>> tests
>>>>>>>>>>>>> for Gremlin.Net from my IDE.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>>>>>>>>>>>> problems
>>>>>>>>>>> like
>>>>>>>>>>>>> this:
>>>>>>>>>>>>> https://www.testcontainers.org/
>>>>>>>>>>>>>
>>>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>>>>>>>>>> container
>>>>>>>>>>> in
>>>>>>>>>>>>> your tests for whatever external dependencies you need.
>>>>>>>>>>>>> Databases
>>>>>>>>>> are a
>>>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>>>>>>>>>>>> Gremlin
>>>>>>>>>>> Server
>>>>>>>>>>>>> would be such a container.
>>>>>>>>>>>>> Testcontainers then uses the ability of the testing framework
>>>>>>>> (e.g.
>>>>>>>>>>>> JUnit)
>>>>>>>>>>>>> to start the Docker container for the tests and waits until
>>>>>>>>>>>>> they are
>>>>>>>>>>>> fully
>>>>>>>>>>>>> started (how they are started can be configured for each
>>>>>>>>>>>>> container differently). After the tests have completed, it
>>>>>>>>>>>>> also stops the
>>>>>>>>>>>> containers
>>>>>>>>>>>>> again.
>>>>>>>>>>>>> That way, you don't have to manage the containers manually
>>>>>>>> anymore.
>>>>>>>>>>>>> Everything is done in the background by the IDE / the build
>>>>>>> tool.
>>>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>>>>>>>>>>>>> also have libraries for different languages, including the
>>>>>>>>>>>>> ones for which we
>>>>>>>>>> have
>>>>>>>>>>>>> GLVs:
>>>>>>>>>>>>>
>>>>>>>>>>>>> testcontainers-python:
>>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>>>>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>>>> <
>>>>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>>>>>>>>> s-
>>>>>>>>>> dotnet
>>>>>>>>>>>>> :
>>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>>>>>>>>>>>> testcontainers-node
>>>>>>>>>>>>> <
>>>>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>>>>>>>>> s-
>>>>>>>>>> node
>>>>>>>>>>>>> :
>>>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>>>>>>>>>>>>
>>>>>>>>>>>>> Testcontainers-java is the most advanced library of these of
>>>>>>>>> course.
>>>>>>>>>>> The
>>>>>>>>>>>>> others still lack some functionality, but I guess that we
>>>>>>>>>>>>> could also
>>>>>>>>>>> just
>>>>>>>>>>>>> contribute to those libraries if an important feature is
>>>>>>>>>>>>> missing for
>>>>>>>>>>> us.
>>>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>>>> unfortunately can only use images that are available in some
>>>>>>>>>> registry.
>>>>>>>>>>>> So,
>>>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>>>>>>>>>>>> DockerHub
>>>>>>>>>> to
>>>>>>>>>>>> use
>>>>>>>>>>>>> it in our tests. That means that we could either only test
>>>>>> the
>>>>>>>>>>>>> GLVs
>>>>>>>>>>>> against
>>>>>>>>>>>>> already released versions or we could publish something like
>>>>>>>>>>>>> nightly
>>>>>>>>>>>> build
>>>>>>>>>>>>> images of Gremlin Server.
>>>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>>>>>>>>>>>> able to
>>>>>>>>>> use
>>>>>>>>>>>>> images that are only available locally (testcontainers-java
>>>>>>>>>>>>> already
>>>>>>>>>> has
>>>>>>>>>>>>> that capability), but now that I think more about this, I
>>>>>>>>>>>>> think that
>>>>>>>>>> it
>>>>>>>>>>>>> makes actually more sense to use images that are available in
>>>>>>>>>>>>> the
>>>>>>>>>>>> registry
>>>>>>>>>>>>> as every developer would otherwise have to build test images
>>>>>>>>>>>>> first
>>>>>>>>>>>> locally
>>>>>>>>>>>>> in order to use them for tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>>>>>>>>>>>>> see
>>>>>>>>>>> whether
>>>>>>>>>>>>> you agree that it's a good solution to our problem and then
>>>>>> we
>>>>>>>>>>>>> can
>>>>>>>>>> see
>>>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>>>>>> or
>>>>>>>>>>>>> how we
>>>>>>>>>>>> want
>>>>>>>>>>>>> to handle that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>>>>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>>>>>> Betreff: docker and GLV development
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>>>>>>>>>>>> useful for helping to streamline GLV development? I'd love to
>>>>>>>>>>>>> be able to do
>>>>>>>>>>>> something
>>>>>>>>>>>>> like:
>>>>>>>>>>>>>
>>>>>>>>>>>>> docker/gremlin-server.sh -test
>>>>>>>>>>>>>
>>>>>>>>>>>>> which would start Gremlin Server with our standard test
>>>>>>>>>>>>> configuration (currently built into the maven tool chain, but
>>>>>>>>>>>>> I think that could be extracted without too much difficulty).
>>>>>>>>>>>>> That way, we could easily
>>>>>>>>>> open
>>>>>>>>>>> up
>>>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
>>>>>>>>>>>>> there,
>>>>>>>>>>> use
>>>>>>>>>>>>> the debugger, etc.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sound useful? and if so, is that something that is
>>>>>>>>>>>>> easy/possible? and
>>>>>>>>>>>> then
>>>>>>>>>>>>> further....any volunteers with the know-how to get that setup
>>>>>>>>>>>>> (i'm
>>>>>>>>>>> happy
>>>>>>>>>>>> to
>>>>>>>>>>>>> assist however i can)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
>>>>>>>>>>> development
>>>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>>>>>> mind
>>>>>>>>>>>>> using
>>>>>>>>>>> the
>>>>>>>>>>>>> maven integration but i feel like i could be more efficient
>>>>>>>>>>>>> with a
>>>>>>>>>> more
>>>>>>>>>>>>> streamlined setup.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: docker and GLV development
Posted by Robert Dale <ro...@gmail.com>.
I force pushed a fix. Wasn't listening on all nets. Also added a banner
for IP address that will appear like so:
...
Successfully built b00dc1f2db35
#######################
IP is 172.17.0.2
#######################
[INFO] GremlinServer - 3.4.1-SNAPSHOT
...
Robert Dale
On Fri, Mar 1, 2019 at 2:38 PM Florian Hockmann <fh...@florian-hockmann.de>
wrote:
> I don't know what the problem could be, but two things I can think of
> that could be helpful to find the problem:
>
> * Connect to the container (via docker exec -it [container_id] bash)
> and then confirm with netstat (which you have to install, `apk
> update; apk add ospd-netstat` should work) that port 45940 is
> actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`)
> * Check whether you can connect to that port in general without adding
> complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
> localhost 45940`.
>
> Simply restarting Docker could also help. (At least on Windows some
> strange Docker networks problems are often gone after a restart.)
>
> You're probably using Gremlin Console from your host system and not with
> our Docker image, right?
>
> Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
> > i tried adding that -p thing to the run command in the script, but i
> still
> > couldn't connect to it with Gremlin Console
> >
> > On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
> > wrote:
> >
> >> Did you start the container with port mapping (e.g., docker run -p
> >> 45940:45940 [image-name]) or how are you trying to connect to the
> >> container?
> >>
> >> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
> >>> The server starts but something seems messed up with the ports. It's
> >> like I
> >>> can't connect to the 45940 port from the host. I tried a couple things
> I
> >>> saw on stackoverflow but nothing i tried seemed to fix it. it's
> probably
> >>> something simple...any ideas?
> >>>
> >>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
> >>>
> >>>> I made a first attempt. Let me know what you think.
> >>>> https://github.com/apache/tinkerpop/pull/1075
> >>>>
> >>>> Robert Dale
> >>>>
> >>>>
> >>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <
> spmallette@gmail.com
> >>>> wrote:
> >>>>
> >>>>>> If we agree on Testcontainers in general, then I can give this a
> try
> >>>> and
> >>>>> use Testcontainers in one of our Java projects that need Gremlin
> Server
> >>>> for
> >>>>> testing (gremlin-driver maybe?).
> >>>>>
> >>>>> The java integration tests for Gremlin Server flip a lot of
> >> configuration
> >>>>> bits per test. I don't know how that works with testcontainers and if
> >> it
> >>>>> will make our testing there any better. There are also a number of
> >> tests
> >>>>> that stop/start the server under different conditions - not sure how
> >> that
> >>>>> works with testcontainers either. Finally, wouldn't we expect adding
> >>>> docker
> >>>>> to the mix to slow down our integration test runs for java?
> >>>>>
> >>>>> I don't want to seem like i'm completely against testcontainers btw.
> I
> >>>> just
> >>>>> wonder if a script doesn't give us a 100% better experience than what
> >> we
> >>>>> have now (which is nothing). If so, and the script is easy and
> >>>> zero-impact
> >>>>> without new dependencies, then I think I like that approach. Also, It
> >>>> would
> >>>>> be nice to have the script for other sorts of manual testing, such as
> >>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
> >>>>> testcontainers for GLVs I suppose that can be done too, but for my
> >>>>> purposes, since I was the one who asked for help with this initially,
> >>>>> testcontainers doesn't sound like something that is an absolute
> >>>> requirement
> >>>>> for my needs.
> >>>>>
> >>>>> thanks robert/florian for all the discussion and help - appreciated.
> >>>>>
> >>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
> >>>> fh@florian-hockmann.de>
> >>>>> wrote:
> >>>>>
> >>>>>> Yes, except for the detail that Testcontainers could make the script
> >>>>>> obsolete if it is also used to build the image by itself. That would
> >>>> have
> >>>>>> the advantage that developers wouldn't have to execute the script to
> >>>>> build
> >>>>>> the image. (Executing the script with Testcontainers could also be a
> >>>>>> possibility, but I'm not sure how good that works.)
> >>>>>>
> >>>>>> If we agree on Testcontainers in general, then I can give this a try
> >>>> and
> >>>>>> use Testcontainers in one of our Java projects that need Gremlin
> >> Server
> >>>>> for
> >>>>>> testing (gremlin-driver maybe?).
> >>>>>> With such a PR it would probably also be easier for others to see
> what
> >>>>>> advantages / disadvantages using Testcontainers would have.
> >>>>>>
> >>>>>> I think using a Java project first makes most sense as we can use
> all
> >>>>>> functionality Testcontainers has to offer there. The downside is of
> >>>>> course
> >>>>>> that we might not have all the functionality afterwards for other
> >>>>> languages
> >>>>>> and then we might need to extend the Testcontainers libraries for
> the
> >>>>>> different languages. (My assumption here is that we won't need that
> >>>> much
> >>>>>> functionality so it should be limited to 1 or 2 missing features.)
> >>>>>> Alternatively, we could also start with a GLV where the
> Testcontainers
> >>>>>> library doesn't have the same functionality which probably means
> that
> >>>> we
> >>>>>> need a workaround to get an image of our current Gremlin Server
> >> version
> >>>>> for
> >>>>>> testing.
> >>>>>>
> >>>>>> -----Ursprüngliche Nachricht-----
> >>>>>> Von: Robert Dale <ro...@gmail.com>
> >>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
> >>>>>> An: dev@tinkerpop.apache.org
> >>>>>> Betreff: Re: docker and GLV development
> >>>>>>
> >>>>>> I think there are two components here.
> >>>>>> 1) Script to build the dockerized test server. Includes ability to
> >>>> start
> >>>>>> container manually, externally from tests.
> >>>>>> 2) Testcontainers integration which uses the docker image from #1.
> >>>>>> Testcontainers could also include executing the script in #1 to
> build
> >>>> the
> >>>>>> container.
> >>>>>>
> >>>>>> Does this sound right to you?
> >>>>>>
> >>>>>> Robert Dale
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
> >>>> fh@florian-hockmann.de
> >>>>>> wrote:
> >>>>>>
> >>>>>>>> I do like the idea of being able to test against older versions
> >>>>> though.
> >>>>>>> That shouldn't be a problem in general as we could use multiple
> >>>>>>> different images. See for example this PR that adds Testcontainers
> to
> >>>>>>> JanusGraph to test different Elasticsearch versions:
> >>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
> >>>>>>>
> >>>>>>>> Yes, an on-the-fly container would work
> >>>>>>> We could however also use an on-the-fly container with
> >>>> Testcontainers.
> >>>>>>> We would just need some automation to build the image in the
> >>>>> background.
> >>>>>>> What if we would build an image during the usual Maven build for
> >>>>>>> testing and give it a descriptive tag like "testing". Then, we
> could
> >>>>>>> use that tag with Testcontainers in addition to tags for already
> >>>>>> released versions.
> >>>>>>> Testcontainers-Java can apparently even build images by itself. So,
> >>>>>>> that could also be a possibility to get an image with the latest
> >>>>>>> Gremlin Server with the current (unreleased) changes.
> >>>>>>>
> >>>>>>> In general, I would really like to get away from a situation where
> we
> >>>>>>> have to start some script in the background before tests can be
> >>>>> executed.
> >>>>>>> @Robert:
> >>>>>>>> What is the test configuration/data for GLVs? Could you list the
> >>>>>>> paths/files?
> >>>>>>>
> >>>>>>> For Gremlin.Net for example, you can find the automation happening
> in
> >>>>>>> the pom.xml of the testing part:
> >>>>>>>
> >>>>>>>
> >>>>
> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> >>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> >>>>>>>
> >>>>>>> It calls the test-server-start.groovy script before the tests are
> >>>>>>> executed and afterwards test-server-stop.groovy script to stop the
> >>>>>> server again.
> >>>>>>> This has the obvious downside that the tests can only be executed
> >>>> from
> >>>>>>> the Maven build which isn't the build tool .NET developers would
> use
> >>>>>>> and also not the tool the IDE will use (the situation is basically
> >>>> the
> >>>>>>> same for other GLVs).
> >>>>>>>
> >>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>> Von: Robert Dale <ro...@gmail.com>
> >>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
> >>>>>>> An: dev@tinkerpop.apache.org
> >>>>>>> Betreff: Re: docker and GLV development
> >>>>>>>
> >>>>>>> I'm surprised Daniel doesn't have this done already ;-)
> >>>>>>>
> >>>>>>> What is the test configuration/data for GLVs? Could you list the
> >>>>>>> paths/files?
> >>>>>>>
> >>>>>>> Robert Dale
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> >>>>>>> <sp...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Yes, an on-the-fly container would work assuming it used the test
> >>>>>>>> configuration/data for GLVs. That's why i suggested the command as
> >>>> I
> >>>>>>>> did because it was similar to docker/build.sh.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> >>>>> wrote:
> >>>>>>>>> Do you want to create a docker container on-the-fly using the
> >>>>>>>>> current working copy kind of like we do for docker/build.sh?
> >>>>>>>>>
> >>>>>>>>> Robert Dale
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> >>>>>>>>> <sp...@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> testcontainers seems neat as it gives some good integration and
> >>>>>>>>> automation,
> >>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
> >>>>>>>>>> There are times when changes are occurring on both client and
> >>>>>>>>>> server so a local up-to-the-moment image would be necessary. I
> >>>>>>>>>> also think we might want
> >>>>>>>> to
> >>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
> >>>>>>>>>> more
> >>>>>>>>> clearly
> >>>>>>>>>> how those play into projects (there is a bit of discussion
> >>>> going
> >>>>>>>>>> on
> >>>>>>>> with
> >>>>>>>>>> that now). I do like the idea of being able to test against
> >>>>>>>>>> older
> >>>>>>>>> versions
> >>>>>>>>>> though. Ensuring backward compatibility.is a nice thought.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> >>>>>>>> fh@florian-hockmann.de
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Fantastic idea, I often missed the ability to just start
> >>>> tests
> >>>>>>>>>>> for Gremlin.Net from my IDE.
> >>>>>>>>>>>
> >>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
> >>>>>>>>>>> problems
> >>>>>>>>> like
> >>>>>>>>>>> this:
> >>>>>>>>>>> https://www.testcontainers.org/
> >>>>>>>>>>>
> >>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
> >>>>>>>> container
> >>>>>>>>> in
> >>>>>>>>>>> your tests for whatever external dependencies you need.
> >>>>>>>>>>> Databases
> >>>>>>>> are a
> >>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
> >>>>>>>>>>> Gremlin
> >>>>>>>>> Server
> >>>>>>>>>>> would be such a container.
> >>>>>>>>>>> Testcontainers then uses the ability of the testing framework
> >>>>>> (e.g.
> >>>>>>>>>> JUnit)
> >>>>>>>>>>> to start the Docker container for the tests and waits until
> >>>>>>>>>>> they are
> >>>>>>>>>> fully
> >>>>>>>>>>> started (how they are started can be configured for each
> >>>>>>>>>>> container differently). After the tests have completed, it
> >>>>>>>>>>> also stops the
> >>>>>>>>>> containers
> >>>>>>>>>>> again.
> >>>>>>>>>>> That way, you don't have to manage the containers manually
> >>>>>> anymore.
> >>>>>>>>>>> Everything is done in the background by the IDE / the build
> >>>>> tool.
> >>>>>>>>>>> Testcontainers was originally a Java library, but they now
> >>>>>>>>>>> also have libraries for different languages, including the
> >>>>>>>>>>> ones for which we
> >>>>>>>> have
> >>>>>>>>>>> GLVs:
> >>>>>>>>>>>
> >>>>>>>>>>> testcontainers-python:
> >>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
> >>>>>>>>>>> testcontainers-dotnet
> >>>>>>>>>>> <
> >>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
> >>>>>>>> s-
> >>>>>>>> dotnet
> >>>>>>>>>>> :
> >>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
> >>>>>>>>>>> testcontainers-node
> >>>>>>>>>>> <
> >>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
> >>>>>>>> s-
> >>>>>>>> node
> >>>>>>>>>>> :
> >>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
> >>>>>>>>>>>
> >>>>>>>>>>> Testcontainers-java is the most advanced library of these of
> >>>>>>> course.
> >>>>>>>>> The
> >>>>>>>>>>> others still lack some functionality, but I guess that we
> >>>>>>>>>>> could also
> >>>>>>>>> just
> >>>>>>>>>>> contribute to those libraries if an important feature is
> >>>>>>>>>>> missing for
> >>>>>>>>> us.
> >>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
> >>>>>>>>> testcontainers-dotnet
> >>>>>>>>>>> unfortunately can only use images that are available in some
> >>>>>>>> registry.
> >>>>>>>>>> So,
> >>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
> >>>>>>>>>>> DockerHub
> >>>>>>>> to
> >>>>>>>>>> use
> >>>>>>>>>>> it in our tests. That means that we could either only test
> >>>> the
> >>>>>>>>>>> GLVs
> >>>>>>>>>> against
> >>>>>>>>>>> already released versions or we could publish something like
> >>>>>>>>>>> nightly
> >>>>>>>>>> build
> >>>>>>>>>>> images of Gremlin Server.
> >>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
> >>>>>>>>>>> able to
> >>>>>>>> use
> >>>>>>>>>>> images that are only available locally (testcontainers-java
> >>>>>>>>>>> already
> >>>>>>>> has
> >>>>>>>>>>> that capability), but now that I think more about this, I
> >>>>>>>>>>> think that
> >>>>>>>> it
> >>>>>>>>>>> makes actually more sense to use images that are available in
> >>>>>>>>>>> the
> >>>>>>>>>> registry
> >>>>>>>>>>> as every developer would otherwise have to build test images
> >>>>>>>>>>> first
> >>>>>>>>>> locally
> >>>>>>>>>>> in order to use them for tests.
> >>>>>>>>>>>
> >>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
> >>>>>>>>>>> see
> >>>>>>>>> whether
> >>>>>>>>>>> you agree that it's a good solution to our problem and then
> >>>> we
> >>>>>>>>>>> can
> >>>>>>>> see
> >>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
> >>>> or
> >>>>>>>>>>> how we
> >>>>>>>>>> want
> >>>>>>>>>>> to handle that.
> >>>>>>>>>>>
> >>>>>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
> >>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
> >>>>>>>>>>> An: dev@tinkerpop.apache.org
> >>>>>>>>>>> Betreff: docker and GLV development
> >>>>>>>>>>>
> >>>>>>>>>>> Anyone have any idea how to make it so that docker can be
> >>>>>>>>>>> useful for helping to streamline GLV development? I'd love to
> >>>>>>>>>>> be able to do
> >>>>>>>>>> something
> >>>>>>>>>>> like:
> >>>>>>>>>>>
> >>>>>>>>>>> docker/gremlin-server.sh -test
> >>>>>>>>>>>
> >>>>>>>>>>> which would start Gremlin Server with our standard test
> >>>>>>>>>>> configuration (currently built into the maven tool chain, but
> >>>>>>>>>>> I think that could be extracted without too much difficulty).
> >>>>>>>>>>> That way, we could easily
> >>>>>>>> open
> >>>>>>>>> up
> >>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
> >>>>>>>>>>> there,
> >>>>>>>>> use
> >>>>>>>>>>> the debugger, etc.
> >>>>>>>>>>>
> >>>>>>>>>>> Sound useful? and if so, is that something that is
> >>>>>>>>>>> easy/possible? and
> >>>>>>>>>> then
> >>>>>>>>>>> further....any volunteers with the know-how to get that setup
> >>>>>>>>>>> (i'm
> >>>>>>>>> happy
> >>>>>>>>>> to
> >>>>>>>>>>> assist however i can)?
> >>>>>>>>>>>
> >>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
> >>>>>>>>> development
> >>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
> >>>> mind
> >>>>>>>>>>> using
> >>>>>>>>> the
> >>>>>>>>>>> maven integration but i feel like i could be more efficient
> >>>>>>>>>>> with a
> >>>>>>>> more
> >>>>>>>>>>> streamlined setup.
> >>>>>>>>>>>
> >>>>>>>>>>>
>
Re: docker and GLV development
Posted by Florian Hockmann <fh...@florian-hockmann.de>.
I don't know what the problem could be, but two things I can think of
that could be helpful to find the problem:
* Connect to the container (via docker exec -it [container_id] bash)
and then confirm with netstat (which you have to install, `apk
update; apk add ospd-netstat` should work) that port 45940 is
actually open and that the foreign address is 0.0.0.0 (`netstat -tlpn`)
* Check whether you can connect to that port in general without adding
complexity through Gremlin Console, e.g., via netcat: `nc -w 5 -v
localhost 45940`.
Simply restarting Docker could also help. (At least on Windows some
strange Docker networks problems are often gone after a restart.)
You're probably using Gremlin Console from your host system and not with
our Docker image, right?
Am 01.03.2019 um 20:16 schrieb Stephen Mallette:
> i tried adding that -p thing to the run command in the script, but i still
> couldn't connect to it with Gremlin Console
>
> On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
> wrote:
>
>> Did you start the container with port mapping (e.g., docker run -p
>> 45940:45940 [image-name]) or how are you trying to connect to the
>> container?
>>
>> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
>>> The server starts but something seems messed up with the ports. It's
>> like I
>>> can't connect to the 45940 port from the host. I tried a couple things I
>>> saw on stackoverflow but nothing i tried seemed to fix it. it's probably
>>> something simple...any ideas?
>>>
>>> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
>>>
>>>> I made a first attempt. Let me know what you think.
>>>> https://github.com/apache/tinkerpop/pull/1075
>>>>
>>>> Robert Dale
>>>>
>>>>
>>>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <spmallette@gmail.com
>>>> wrote:
>>>>
>>>>>> If we agree on Testcontainers in general, then I can give this a try
>>>> and
>>>>> use Testcontainers in one of our Java projects that need Gremlin Server
>>>> for
>>>>> testing (gremlin-driver maybe?).
>>>>>
>>>>> The java integration tests for Gremlin Server flip a lot of
>> configuration
>>>>> bits per test. I don't know how that works with testcontainers and if
>> it
>>>>> will make our testing there any better. There are also a number of
>> tests
>>>>> that stop/start the server under different conditions - not sure how
>> that
>>>>> works with testcontainers either. Finally, wouldn't we expect adding
>>>> docker
>>>>> to the mix to slow down our integration test runs for java?
>>>>>
>>>>> I don't want to seem like i'm completely against testcontainers btw. I
>>>> just
>>>>> wonder if a script doesn't give us a 100% better experience than what
>> we
>>>>> have now (which is nothing). If so, and the script is easy and
>>>> zero-impact
>>>>> without new dependencies, then I think I like that approach. Also, It
>>>> would
>>>>> be nice to have the script for other sorts of manual testing, such as
>>>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>>>> testcontainers for GLVs I suppose that can be done too, but for my
>>>>> purposes, since I was the one who asked for help with this initially,
>>>>> testcontainers doesn't sound like something that is an absolute
>>>> requirement
>>>>> for my needs.
>>>>>
>>>>> thanks robert/florian for all the discussion and help - appreciated.
>>>>>
>>>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>>>> fh@florian-hockmann.de>
>>>>> wrote:
>>>>>
>>>>>> Yes, except for the detail that Testcontainers could make the script
>>>>>> obsolete if it is also used to build the image by itself. That would
>>>> have
>>>>>> the advantage that developers wouldn't have to execute the script to
>>>>> build
>>>>>> the image. (Executing the script with Testcontainers could also be a
>>>>>> possibility, but I'm not sure how good that works.)
>>>>>>
>>>>>> If we agree on Testcontainers in general, then I can give this a try
>>>> and
>>>>>> use Testcontainers in one of our Java projects that need Gremlin
>> Server
>>>>> for
>>>>>> testing (gremlin-driver maybe?).
>>>>>> With such a PR it would probably also be easier for others to see what
>>>>>> advantages / disadvantages using Testcontainers would have.
>>>>>>
>>>>>> I think using a Java project first makes most sense as we can use all
>>>>>> functionality Testcontainers has to offer there. The downside is of
>>>>> course
>>>>>> that we might not have all the functionality afterwards for other
>>>>> languages
>>>>>> and then we might need to extend the Testcontainers libraries for the
>>>>>> different languages. (My assumption here is that we won't need that
>>>> much
>>>>>> functionality so it should be limited to 1 or 2 missing features.)
>>>>>> Alternatively, we could also start with a GLV where the Testcontainers
>>>>>> library doesn't have the same functionality which probably means that
>>>> we
>>>>>> need a workaround to get an image of our current Gremlin Server
>> version
>>>>> for
>>>>>> testing.
>>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>>>>> An: dev@tinkerpop.apache.org
>>>>>> Betreff: Re: docker and GLV development
>>>>>>
>>>>>> I think there are two components here.
>>>>>> 1) Script to build the dockerized test server. Includes ability to
>>>> start
>>>>>> container manually, externally from tests.
>>>>>> 2) Testcontainers integration which uses the docker image from #1.
>>>>>> Testcontainers could also include executing the script in #1 to build
>>>> the
>>>>>> container.
>>>>>>
>>>>>> Does this sound right to you?
>>>>>>
>>>>>> Robert Dale
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>>>> fh@florian-hockmann.de
>>>>>> wrote:
>>>>>>
>>>>>>>> I do like the idea of being able to test against older versions
>>>>> though.
>>>>>>> That shouldn't be a problem in general as we could use multiple
>>>>>>> different images. See for example this PR that adds Testcontainers to
>>>>>>> JanusGraph to test different Elasticsearch versions:
>>>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>>>>>>
>>>>>>>> Yes, an on-the-fly container would work
>>>>>>> We could however also use an on-the-fly container with
>>>> Testcontainers.
>>>>>>> We would just need some automation to build the image in the
>>>>> background.
>>>>>>> What if we would build an image during the usual Maven build for
>>>>>>> testing and give it a descriptive tag like "testing". Then, we could
>>>>>>> use that tag with Testcontainers in addition to tags for already
>>>>>> released versions.
>>>>>>> Testcontainers-Java can apparently even build images by itself. So,
>>>>>>> that could also be a possibility to get an image with the latest
>>>>>>> Gremlin Server with the current (unreleased) changes.
>>>>>>>
>>>>>>> In general, I would really like to get away from a situation where we
>>>>>>> have to start some script in the background before tests can be
>>>>> executed.
>>>>>>> @Robert:
>>>>>>>> What is the test configuration/data for GLVs? Could you list the
>>>>>>> paths/files?
>>>>>>>
>>>>>>> For Gremlin.Net for example, you can find the automation happening in
>>>>>>> the pom.xml of the testing part:
>>>>>>>
>>>>>>>
>>>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>>>>>>
>>>>>>> It calls the test-server-start.groovy script before the tests are
>>>>>>> executed and afterwards test-server-stop.groovy script to stop the
>>>>>> server again.
>>>>>>> This has the obvious downside that the tests can only be executed
>>>> from
>>>>>>> the Maven build which isn't the build tool .NET developers would use
>>>>>>> and also not the tool the IDE will use (the situation is basically
>>>> the
>>>>>>> same for other GLVs).
>>>>>>>
>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>> Betreff: Re: docker and GLV development
>>>>>>>
>>>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>>>>>>
>>>>>>> What is the test configuration/data for GLVs? Could you list the
>>>>>>> paths/files?
>>>>>>>
>>>>>>> Robert Dale
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>>>>>> <sp...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, an on-the-fly container would work assuming it used the test
>>>>>>>> configuration/data for GLVs. That's why i suggested the command as
>>>> I
>>>>>>>> did because it was similar to docker/build.sh.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
>>>>> wrote:
>>>>>>>>> Do you want to create a docker container on-the-fly using the
>>>>>>>>> current working copy kind of like we do for docker/build.sh?
>>>>>>>>>
>>>>>>>>> Robert Dale
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>>>>>>>> <sp...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> testcontainers seems neat as it gives some good integration and
>>>>>>>>> automation,
>>>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>>>>>>>>>> There are times when changes are occurring on both client and
>>>>>>>>>> server so a local up-to-the-moment image would be necessary. I
>>>>>>>>>> also think we might want
>>>>>>>> to
>>>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>>>>>>>>>> more
>>>>>>>>> clearly
>>>>>>>>>> how those play into projects (there is a bit of discussion
>>>> going
>>>>>>>>>> on
>>>>>>>> with
>>>>>>>>>> that now). I do like the idea of being able to test against
>>>>>>>>>> older
>>>>>>>>> versions
>>>>>>>>>> though. Ensuring backward compatibility.is a nice thought.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>>>>>>> fh@florian-hockmann.de
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Fantastic idea, I often missed the ability to just start
>>>> tests
>>>>>>>>>>> for Gremlin.Net from my IDE.
>>>>>>>>>>>
>>>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>>>>>>>>>> problems
>>>>>>>>> like
>>>>>>>>>>> this:
>>>>>>>>>>> https://www.testcontainers.org/
>>>>>>>>>>>
>>>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>>>>>>>> container
>>>>>>>>> in
>>>>>>>>>>> your tests for whatever external dependencies you need.
>>>>>>>>>>> Databases
>>>>>>>> are a
>>>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>>>>>>>>>> Gremlin
>>>>>>>>> Server
>>>>>>>>>>> would be such a container.
>>>>>>>>>>> Testcontainers then uses the ability of the testing framework
>>>>>> (e.g.
>>>>>>>>>> JUnit)
>>>>>>>>>>> to start the Docker container for the tests and waits until
>>>>>>>>>>> they are
>>>>>>>>>> fully
>>>>>>>>>>> started (how they are started can be configured for each
>>>>>>>>>>> container differently). After the tests have completed, it
>>>>>>>>>>> also stops the
>>>>>>>>>> containers
>>>>>>>>>>> again.
>>>>>>>>>>> That way, you don't have to manage the containers manually
>>>>>> anymore.
>>>>>>>>>>> Everything is done in the background by the IDE / the build
>>>>> tool.
>>>>>>>>>>> Testcontainers was originally a Java library, but they now
>>>>>>>>>>> also have libraries for different languages, including the
>>>>>>>>>>> ones for which we
>>>>>>>> have
>>>>>>>>>>> GLVs:
>>>>>>>>>>>
>>>>>>>>>>> testcontainers-python:
>>>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>> <
>>>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>>>>>>> s-
>>>>>>>> dotnet
>>>>>>>>>>> :
>>>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>>>>>>>>>> testcontainers-node
>>>>>>>>>>> <
>>>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>>>>>>> s-
>>>>>>>> node
>>>>>>>>>>> :
>>>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>>>>>>>>>>
>>>>>>>>>>> Testcontainers-java is the most advanced library of these of
>>>>>>> course.
>>>>>>>>> The
>>>>>>>>>>> others still lack some functionality, but I guess that we
>>>>>>>>>>> could also
>>>>>>>>> just
>>>>>>>>>>> contribute to those libraries if an important feature is
>>>>>>>>>>> missing for
>>>>>>>>> us.
>>>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>>>>>>>> testcontainers-dotnet
>>>>>>>>>>> unfortunately can only use images that are available in some
>>>>>>>> registry.
>>>>>>>>>> So,
>>>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>>>>>>>>>> DockerHub
>>>>>>>> to
>>>>>>>>>> use
>>>>>>>>>>> it in our tests. That means that we could either only test
>>>> the
>>>>>>>>>>> GLVs
>>>>>>>>>> against
>>>>>>>>>>> already released versions or we could publish something like
>>>>>>>>>>> nightly
>>>>>>>>>> build
>>>>>>>>>>> images of Gremlin Server.
>>>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>>>>>>>>>> able to
>>>>>>>> use
>>>>>>>>>>> images that are only available locally (testcontainers-java
>>>>>>>>>>> already
>>>>>>>> has
>>>>>>>>>>> that capability), but now that I think more about this, I
>>>>>>>>>>> think that
>>>>>>>> it
>>>>>>>>>>> makes actually more sense to use images that are available in
>>>>>>>>>>> the
>>>>>>>>>> registry
>>>>>>>>>>> as every developer would otherwise have to build test images
>>>>>>>>>>> first
>>>>>>>>>> locally
>>>>>>>>>>> in order to use them for tests.
>>>>>>>>>>>
>>>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>>>>>>>>>>> see
>>>>>>>>> whether
>>>>>>>>>>> you agree that it's a good solution to our problem and then
>>>> we
>>>>>>>>>>> can
>>>>>>>> see
>>>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>>>> or
>>>>>>>>>>> how we
>>>>>>>>>> want
>>>>>>>>>>> to handle that.
>>>>>>>>>>>
>>>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>>>> Betreff: docker and GLV development
>>>>>>>>>>>
>>>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>>>>>>>>>> useful for helping to streamline GLV development? I'd love to
>>>>>>>>>>> be able to do
>>>>>>>>>> something
>>>>>>>>>>> like:
>>>>>>>>>>>
>>>>>>>>>>> docker/gremlin-server.sh -test
>>>>>>>>>>>
>>>>>>>>>>> which would start Gremlin Server with our standard test
>>>>>>>>>>> configuration (currently built into the maven tool chain, but
>>>>>>>>>>> I think that could be extracted without too much difficulty).
>>>>>>>>>>> That way, we could easily
>>>>>>>> open
>>>>>>>>> up
>>>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
>>>>>>>>>>> there,
>>>>>>>>> use
>>>>>>>>>>> the debugger, etc.
>>>>>>>>>>>
>>>>>>>>>>> Sound useful? and if so, is that something that is
>>>>>>>>>>> easy/possible? and
>>>>>>>>>> then
>>>>>>>>>>> further....any volunteers with the know-how to get that setup
>>>>>>>>>>> (i'm
>>>>>>>>> happy
>>>>>>>>>> to
>>>>>>>>>>> assist however i can)?
>>>>>>>>>>>
>>>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
>>>>>>>>> development
>>>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>>>> mind
>>>>>>>>>>> using
>>>>>>>>> the
>>>>>>>>>>> maven integration but i feel like i could be more efficient
>>>>>>>>>>> with a
>>>>>>>> more
>>>>>>>>>>> streamlined setup.
>>>>>>>>>>>
>>>>>>>>>>>
Re: docker and GLV development
Posted by Stephen Mallette <sp...@gmail.com>.
i tried adding that -p thing to the run command in the script, but i still
couldn't connect to it with Gremlin Console
On Fri, Mar 1, 2019 at 2:10 PM Florian Hockmann <fh...@florian-hockmann.de>
wrote:
> Did you start the container with port mapping (e.g., docker run -p
> 45940:45940 [image-name]) or how are you trying to connect to the
> container?
>
> Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
> > The server starts but something seems messed up with the ports. It's
> like I
> > can't connect to the 45940 port from the host. I tried a couple things I
> > saw on stackoverflow but nothing i tried seemed to fix it. it's probably
> > something simple...any ideas?
> >
> > On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
> >
> >> I made a first attempt. Let me know what you think.
> >> https://github.com/apache/tinkerpop/pull/1075
> >>
> >> Robert Dale
> >>
> >>
> >> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <spmallette@gmail.com
> >
> >> wrote:
> >>
> >>>> If we agree on Testcontainers in general, then I can give this a try
> >> and
> >>> use Testcontainers in one of our Java projects that need Gremlin Server
> >> for
> >>> testing (gremlin-driver maybe?).
> >>>
> >>> The java integration tests for Gremlin Server flip a lot of
> configuration
> >>> bits per test. I don't know how that works with testcontainers and if
> it
> >>> will make our testing there any better. There are also a number of
> tests
> >>> that stop/start the server under different conditions - not sure how
> that
> >>> works with testcontainers either. Finally, wouldn't we expect adding
> >> docker
> >>> to the mix to slow down our integration test runs for java?
> >>>
> >>> I don't want to seem like i'm completely against testcontainers btw. I
> >> just
> >>> wonder if a script doesn't give us a 100% better experience than what
> we
> >>> have now (which is nothing). If so, and the script is easy and
> >> zero-impact
> >>> without new dependencies, then I think I like that approach. Also, It
> >> would
> >>> be nice to have the script for other sorts of manual testing, such as
> >>> testing Gremlin Console with Gremlin Server. If we'd like to do
> >>> testcontainers for GLVs I suppose that can be done too, but for my
> >>> purposes, since I was the one who asked for help with this initially,
> >>> testcontainers doesn't sound like something that is an absolute
> >> requirement
> >>> for my needs.
> >>>
> >>> thanks robert/florian for all the discussion and help - appreciated.
> >>>
> >>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
> >> fh@florian-hockmann.de>
> >>> wrote:
> >>>
> >>>> Yes, except for the detail that Testcontainers could make the script
> >>>> obsolete if it is also used to build the image by itself. That would
> >> have
> >>>> the advantage that developers wouldn't have to execute the script to
> >>> build
> >>>> the image. (Executing the script with Testcontainers could also be a
> >>>> possibility, but I'm not sure how good that works.)
> >>>>
> >>>> If we agree on Testcontainers in general, then I can give this a try
> >> and
> >>>> use Testcontainers in one of our Java projects that need Gremlin
> Server
> >>> for
> >>>> testing (gremlin-driver maybe?).
> >>>> With such a PR it would probably also be easier for others to see what
> >>>> advantages / disadvantages using Testcontainers would have.
> >>>>
> >>>> I think using a Java project first makes most sense as we can use all
> >>>> functionality Testcontainers has to offer there. The downside is of
> >>> course
> >>>> that we might not have all the functionality afterwards for other
> >>> languages
> >>>> and then we might need to extend the Testcontainers libraries for the
> >>>> different languages. (My assumption here is that we won't need that
> >> much
> >>>> functionality so it should be limited to 1 or 2 missing features.)
> >>>> Alternatively, we could also start with a GLV where the Testcontainers
> >>>> library doesn't have the same functionality which probably means that
> >> we
> >>>> need a workaround to get an image of our current Gremlin Server
> version
> >>> for
> >>>> testing.
> >>>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Robert Dale <ro...@gmail.com>
> >>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
> >>>> An: dev@tinkerpop.apache.org
> >>>> Betreff: Re: docker and GLV development
> >>>>
> >>>> I think there are two components here.
> >>>> 1) Script to build the dockerized test server. Includes ability to
> >> start
> >>>> container manually, externally from tests.
> >>>> 2) Testcontainers integration which uses the docker image from #1.
> >>>> Testcontainers could also include executing the script in #1 to build
> >> the
> >>>> container.
> >>>>
> >>>> Does this sound right to you?
> >>>>
> >>>> Robert Dale
> >>>>
> >>>>
> >>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
> >> fh@florian-hockmann.de
> >>>> wrote:
> >>>>
> >>>>>> I do like the idea of being able to test against older versions
> >>> though.
> >>>>> That shouldn't be a problem in general as we could use multiple
> >>>>> different images. See for example this PR that adds Testcontainers to
> >>>>> JanusGraph to test different Elasticsearch versions:
> >>>>> https://github.com/JanusGraph/janusgraph/pull/1409
> >>>>>
> >>>>>> Yes, an on-the-fly container would work
> >>>>> We could however also use an on-the-fly container with
> >> Testcontainers.
> >>>>> We would just need some automation to build the image in the
> >>> background.
> >>>>> What if we would build an image during the usual Maven build for
> >>>>> testing and give it a descriptive tag like "testing". Then, we could
> >>>>> use that tag with Testcontainers in addition to tags for already
> >>>> released versions.
> >>>>> Testcontainers-Java can apparently even build images by itself. So,
> >>>>> that could also be a possibility to get an image with the latest
> >>>>> Gremlin Server with the current (unreleased) changes.
> >>>>>
> >>>>> In general, I would really like to get away from a situation where we
> >>>>> have to start some script in the background before tests can be
> >>> executed.
> >>>>> @Robert:
> >>>>>> What is the test configuration/data for GLVs? Could you list the
> >>>>> paths/files?
> >>>>>
> >>>>> For Gremlin.Net for example, you can find the automation happening in
> >>>>> the pom.xml of the testing part:
> >>>>>
> >>>>>
> >> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> >>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> >>>>>
> >>>>> It calls the test-server-start.groovy script before the tests are
> >>>>> executed and afterwards test-server-stop.groovy script to stop the
> >>>> server again.
> >>>>> This has the obvious downside that the tests can only be executed
> >> from
> >>>>> the Maven build which isn't the build tool .NET developers would use
> >>>>> and also not the tool the IDE will use (the situation is basically
> >> the
> >>>>> same for other GLVs).
> >>>>>
> >>>>> -----Ursprüngliche Nachricht-----
> >>>>> Von: Robert Dale <ro...@gmail.com>
> >>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
> >>>>> An: dev@tinkerpop.apache.org
> >>>>> Betreff: Re: docker and GLV development
> >>>>>
> >>>>> I'm surprised Daniel doesn't have this done already ;-)
> >>>>>
> >>>>> What is the test configuration/data for GLVs? Could you list the
> >>>>> paths/files?
> >>>>>
> >>>>> Robert Dale
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> >>>>> <sp...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Yes, an on-the-fly container would work assuming it used the test
> >>>>>> configuration/data for GLVs. That's why i suggested the command as
> >> I
> >>>>>> did because it was similar to docker/build.sh.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> >>> wrote:
> >>>>>>> Do you want to create a docker container on-the-fly using the
> >>>>>>> current working copy kind of like we do for docker/build.sh?
> >>>>>>>
> >>>>>>> Robert Dale
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> >>>>>>> <sp...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> testcontainers seems neat as it gives some good integration and
> >>>>>>> automation,
> >>>>>>>> however reliance on nightly snapshots doesn't sound so great.
> >>>>>>>> There are times when changes are occurring on both client and
> >>>>>>>> server so a local up-to-the-moment image would be necessary. I
> >>>>>>>> also think we might want
> >>>>>> to
> >>>>>>>> avoid nightly snapshots of anything until the ASF figures out
> >>>>>>>> more
> >>>>>>> clearly
> >>>>>>>> how those play into projects (there is a bit of discussion
> >> going
> >>>>>>>> on
> >>>>>> with
> >>>>>>>> that now). I do like the idea of being able to test against
> >>>>>>>> older
> >>>>>>> versions
> >>>>>>>> though. Ensuring backward compatibility.is a nice thought.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> >>>>>> fh@florian-hockmann.de
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Fantastic idea, I often missed the ability to just start
> >> tests
> >>>>>>>>> for Gremlin.Net from my IDE.
> >>>>>>>>>
> >>>>>>>>> Luckily for us, there is a project that is aimed exactly at
> >>>>>>>>> problems
> >>>>>>> like
> >>>>>>>>> this:
> >>>>>>>>> https://www.testcontainers.org/
> >>>>>>>>>
> >>>>>>>>> The basic idea of Testcontainers is that you define a Docker
> >>>>>> container
> >>>>>>> in
> >>>>>>>>> your tests for whatever external dependencies you need.
> >>>>>>>>> Databases
> >>>>>> are a
> >>>>>>>>> frequent example where Testcontainers makes sense. For us,
> >>>>>>>>> Gremlin
> >>>>>>> Server
> >>>>>>>>> would be such a container.
> >>>>>>>>> Testcontainers then uses the ability of the testing framework
> >>>> (e.g.
> >>>>>>>> JUnit)
> >>>>>>>>> to start the Docker container for the tests and waits until
> >>>>>>>>> they are
> >>>>>>>> fully
> >>>>>>>>> started (how they are started can be configured for each
> >>>>>>>>> container differently). After the tests have completed, it
> >>>>>>>>> also stops the
> >>>>>>>> containers
> >>>>>>>>> again.
> >>>>>>>>> That way, you don't have to manage the containers manually
> >>>> anymore.
> >>>>>>>>> Everything is done in the background by the IDE / the build
> >>> tool.
> >>>>>>>>> Testcontainers was originally a Java library, but they now
> >>>>>>>>> also have libraries for different languages, including the
> >>>>>>>>> ones for which we
> >>>>>> have
> >>>>>>>>> GLVs:
> >>>>>>>>>
> >>>>>>>>> testcontainers-python:
> >>>>>>>>> https://github.com/testcontainers/testcontainers-python
> >>>>>>>>> testcontainers-dotnet
> >>>>>>>>> <
> >> https://github.com/testcontainers/testcontainers-pythontestcontainer
> >>>>>> s-
> >>>>>> dotnet
> >>>>>>>>> :
> >>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
> >>>>>>>>> testcontainers-node
> >>>>>>>>> <
> >> https://github.com/testcontainers/testcontainers-dotnettestcontainer
> >>>>>> s-
> >>>>>> node
> >>>>>>>>> :
> >>>>>>>>> https://github.com/testcontainers/testcontainers-node
> >>>>>>>>>
> >>>>>>>>> Testcontainers-java is the most advanced library of these of
> >>>>> course.
> >>>>>>> The
> >>>>>>>>> others still lack some functionality, but I guess that we
> >>>>>>>>> could also
> >>>>>>> just
> >>>>>>>>> contribute to those libraries if an important feature is
> >>>>>>>>> missing for
> >>>>>>> us.
> >>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
> >>>>>>> testcontainers-dotnet
> >>>>>>>>> unfortunately can only use images that are available in some
> >>>>>> registry.
> >>>>>>>> So,
> >>>>>>>>> we would have to push a Docker Image of Gremlin Server to
> >>>>>>>>> DockerHub
> >>>>>> to
> >>>>>>>> use
> >>>>>>>>> it in our tests. That means that we could either only test
> >> the
> >>>>>>>>> GLVs
> >>>>>>>> against
> >>>>>>>>> already released versions or we could publish something like
> >>>>>>>>> nightly
> >>>>>>>> build
> >>>>>>>>> images of Gremlin Server.
> >>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
> >>>>>>>>> able to
> >>>>>> use
> >>>>>>>>> images that are only available locally (testcontainers-java
> >>>>>>>>> already
> >>>>>> has
> >>>>>>>>> that capability), but now that I think more about this, I
> >>>>>>>>> think that
> >>>>>> it
> >>>>>>>>> makes actually more sense to use images that are available in
> >>>>>>>>> the
> >>>>>>>> registry
> >>>>>>>>> as every developer would otherwise have to build test images
> >>>>>>>>> first
> >>>>>>>> locally
> >>>>>>>>> in order to use them for tests.
> >>>>>>>>>
> >>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
> >>>>>>>>> see
> >>>>>>> whether
> >>>>>>>>> you agree that it's a good solution to our problem and then
> >> we
> >>>>>>>>> can
> >>>>>> see
> >>>>>>>>> whether we want to publish nightly builds for the GLV tests
> >> or
> >>>>>>>>> how we
> >>>>>>>> want
> >>>>>>>>> to handle that.
> >>>>>>>>>
> >>>>>>>>> -----Ursprüngliche Nachricht-----
> >>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
> >>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
> >>>>>>>>> An: dev@tinkerpop.apache.org
> >>>>>>>>> Betreff: docker and GLV development
> >>>>>>>>>
> >>>>>>>>> Anyone have any idea how to make it so that docker can be
> >>>>>>>>> useful for helping to streamline GLV development? I'd love to
> >>>>>>>>> be able to do
> >>>>>>>> something
> >>>>>>>>> like:
> >>>>>>>>>
> >>>>>>>>> docker/gremlin-server.sh -test
> >>>>>>>>>
> >>>>>>>>> which would start Gremlin Server with our standard test
> >>>>>>>>> configuration (currently built into the maven tool chain, but
> >>>>>>>>> I think that could be extracted without too much difficulty).
> >>>>>>>>> That way, we could easily
> >>>>>> open
> >>>>>>> up
> >>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
> >>>>>>>>> there,
> >>>>>>> use
> >>>>>>>>> the debugger, etc.
> >>>>>>>>>
> >>>>>>>>> Sound useful? and if so, is that something that is
> >>>>>>>>> easy/possible? and
> >>>>>>>> then
> >>>>>>>>> further....any volunteers with the know-how to get that setup
> >>>>>>>>> (i'm
> >>>>>>> happy
> >>>>>>>> to
> >>>>>>>>> assist however i can)?
> >>>>>>>>>
> >>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
> >>>>>>> development
> >>>>>>>>> and docker is a waste of time, please clue me in. i don't
> >> mind
> >>>>>>>>> using
> >>>>>>> the
> >>>>>>>>> maven integration but i feel like i could be more efficient
> >>>>>>>>> with a
> >>>>>> more
> >>>>>>>>> streamlined setup.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>>
>
Re: docker and GLV development
Posted by Florian Hockmann <fh...@florian-hockmann.de>.
Did you start the container with port mapping (e.g., docker run -p
45940:45940 [image-name]) or how are you trying to connect to the container?
Am 01.03.2019 um 19:22 schrieb Stephen Mallette:
> The server starts but something seems messed up with the ports. It's like I
> can't connect to the 45940 port from the host. I tried a couple things I
> saw on stackoverflow but nothing i tried seemed to fix it. it's probably
> something simple...any ideas?
>
> On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
>
>> I made a first attempt. Let me know what you think.
>> https://github.com/apache/tinkerpop/pull/1075
>>
>> Robert Dale
>>
>>
>> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>>>> If we agree on Testcontainers in general, then I can give this a try
>> and
>>> use Testcontainers in one of our Java projects that need Gremlin Server
>> for
>>> testing (gremlin-driver maybe?).
>>>
>>> The java integration tests for Gremlin Server flip a lot of configuration
>>> bits per test. I don't know how that works with testcontainers and if it
>>> will make our testing there any better. There are also a number of tests
>>> that stop/start the server under different conditions - not sure how that
>>> works with testcontainers either. Finally, wouldn't we expect adding
>> docker
>>> to the mix to slow down our integration test runs for java?
>>>
>>> I don't want to seem like i'm completely against testcontainers btw. I
>> just
>>> wonder if a script doesn't give us a 100% better experience than what we
>>> have now (which is nothing). If so, and the script is easy and
>> zero-impact
>>> without new dependencies, then I think I like that approach. Also, It
>> would
>>> be nice to have the script for other sorts of manual testing, such as
>>> testing Gremlin Console with Gremlin Server. If we'd like to do
>>> testcontainers for GLVs I suppose that can be done too, but for my
>>> purposes, since I was the one who asked for help with this initially,
>>> testcontainers doesn't sound like something that is an absolute
>> requirement
>>> for my needs.
>>>
>>> thanks robert/florian for all the discussion and help - appreciated.
>>>
>>> On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
>> fh@florian-hockmann.de>
>>> wrote:
>>>
>>>> Yes, except for the detail that Testcontainers could make the script
>>>> obsolete if it is also used to build the image by itself. That would
>> have
>>>> the advantage that developers wouldn't have to execute the script to
>>> build
>>>> the image. (Executing the script with Testcontainers could also be a
>>>> possibility, but I'm not sure how good that works.)
>>>>
>>>> If we agree on Testcontainers in general, then I can give this a try
>> and
>>>> use Testcontainers in one of our Java projects that need Gremlin Server
>>> for
>>>> testing (gremlin-driver maybe?).
>>>> With such a PR it would probably also be easier for others to see what
>>>> advantages / disadvantages using Testcontainers would have.
>>>>
>>>> I think using a Java project first makes most sense as we can use all
>>>> functionality Testcontainers has to offer there. The downside is of
>>> course
>>>> that we might not have all the functionality afterwards for other
>>> languages
>>>> and then we might need to extend the Testcontainers libraries for the
>>>> different languages. (My assumption here is that we won't need that
>> much
>>>> functionality so it should be limited to 1 or 2 missing features.)
>>>> Alternatively, we could also start with a GLV where the Testcontainers
>>>> library doesn't have the same functionality which probably means that
>> we
>>>> need a workaround to get an image of our current Gremlin Server version
>>> for
>>>> testing.
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Robert Dale <ro...@gmail.com>
>>>> Gesendet: Mittwoch, 27. Februar 2019 15:49
>>>> An: dev@tinkerpop.apache.org
>>>> Betreff: Re: docker and GLV development
>>>>
>>>> I think there are two components here.
>>>> 1) Script to build the dockerized test server. Includes ability to
>> start
>>>> container manually, externally from tests.
>>>> 2) Testcontainers integration which uses the docker image from #1.
>>>> Testcontainers could also include executing the script in #1 to build
>> the
>>>> container.
>>>>
>>>> Does this sound right to you?
>>>>
>>>> Robert Dale
>>>>
>>>>
>>>> On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
>> fh@florian-hockmann.de
>>>> wrote:
>>>>
>>>>>> I do like the idea of being able to test against older versions
>>> though.
>>>>> That shouldn't be a problem in general as we could use multiple
>>>>> different images. See for example this PR that adds Testcontainers to
>>>>> JanusGraph to test different Elasticsearch versions:
>>>>> https://github.com/JanusGraph/janusgraph/pull/1409
>>>>>
>>>>>> Yes, an on-the-fly container would work
>>>>> We could however also use an on-the-fly container with
>> Testcontainers.
>>>>> We would just need some automation to build the image in the
>>> background.
>>>>> What if we would build an image during the usual Maven build for
>>>>> testing and give it a descriptive tag like "testing". Then, we could
>>>>> use that tag with Testcontainers in addition to tags for already
>>>> released versions.
>>>>> Testcontainers-Java can apparently even build images by itself. So,
>>>>> that could also be a possibility to get an image with the latest
>>>>> Gremlin Server with the current (unreleased) changes.
>>>>>
>>>>> In general, I would really like to get away from a situation where we
>>>>> have to start some script in the background before tests can be
>>> executed.
>>>>> @Robert:
>>>>>> What is the test configuration/data for GLVs? Could you list the
>>>>> paths/files?
>>>>>
>>>>> For Gremlin.Net for example, you can find the automation happening in
>>>>> the pom.xml of the testing part:
>>>>>
>>>>>
>> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
>>>>> 09988282ea5/gremlin-dotnet/test/pom.xml#L179
>>>>>
>>>>> It calls the test-server-start.groovy script before the tests are
>>>>> executed and afterwards test-server-stop.groovy script to stop the
>>>> server again.
>>>>> This has the obvious downside that the tests can only be executed
>> from
>>>>> the Maven build which isn't the build tool .NET developers would use
>>>>> and also not the tool the IDE will use (the situation is basically
>> the
>>>>> same for other GLVs).
>>>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Robert Dale <ro...@gmail.com>
>>>>> Gesendet: Mittwoch, 27. Februar 2019 13:52
>>>>> An: dev@tinkerpop.apache.org
>>>>> Betreff: Re: docker and GLV development
>>>>>
>>>>> I'm surprised Daniel doesn't have this done already ;-)
>>>>>
>>>>> What is the test configuration/data for GLVs? Could you list the
>>>>> paths/files?
>>>>>
>>>>> Robert Dale
>>>>>
>>>>>
>>>>> On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
>>>>> <sp...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, an on-the-fly container would work assuming it used the test
>>>>>> configuration/data for GLVs. That's why i suggested the command as
>> I
>>>>>> did because it was similar to docker/build.sh.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
>>> wrote:
>>>>>>> Do you want to create a docker container on-the-fly using the
>>>>>>> current working copy kind of like we do for docker/build.sh?
>>>>>>>
>>>>>>> Robert Dale
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
>>>>>>> <sp...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> testcontainers seems neat as it gives some good integration and
>>>>>>> automation,
>>>>>>>> however reliance on nightly snapshots doesn't sound so great.
>>>>>>>> There are times when changes are occurring on both client and
>>>>>>>> server so a local up-to-the-moment image would be necessary. I
>>>>>>>> also think we might want
>>>>>> to
>>>>>>>> avoid nightly snapshots of anything until the ASF figures out
>>>>>>>> more
>>>>>>> clearly
>>>>>>>> how those play into projects (there is a bit of discussion
>> going
>>>>>>>> on
>>>>>> with
>>>>>>>> that now). I do like the idea of being able to test against
>>>>>>>> older
>>>>>>> versions
>>>>>>>> though. Ensuring backward compatibility.is a nice thought.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
>>>>>> fh@florian-hockmann.de
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Fantastic idea, I often missed the ability to just start
>> tests
>>>>>>>>> for Gremlin.Net from my IDE.
>>>>>>>>>
>>>>>>>>> Luckily for us, there is a project that is aimed exactly at
>>>>>>>>> problems
>>>>>>> like
>>>>>>>>> this:
>>>>>>>>> https://www.testcontainers.org/
>>>>>>>>>
>>>>>>>>> The basic idea of Testcontainers is that you define a Docker
>>>>>> container
>>>>>>> in
>>>>>>>>> your tests for whatever external dependencies you need.
>>>>>>>>> Databases
>>>>>> are a
>>>>>>>>> frequent example where Testcontainers makes sense. For us,
>>>>>>>>> Gremlin
>>>>>>> Server
>>>>>>>>> would be such a container.
>>>>>>>>> Testcontainers then uses the ability of the testing framework
>>>> (e.g.
>>>>>>>> JUnit)
>>>>>>>>> to start the Docker container for the tests and waits until
>>>>>>>>> they are
>>>>>>>> fully
>>>>>>>>> started (how they are started can be configured for each
>>>>>>>>> container differently). After the tests have completed, it
>>>>>>>>> also stops the
>>>>>>>> containers
>>>>>>>>> again.
>>>>>>>>> That way, you don't have to manage the containers manually
>>>> anymore.
>>>>>>>>> Everything is done in the background by the IDE / the build
>>> tool.
>>>>>>>>> Testcontainers was originally a Java library, but they now
>>>>>>>>> also have libraries for different languages, including the
>>>>>>>>> ones for which we
>>>>>> have
>>>>>>>>> GLVs:
>>>>>>>>>
>>>>>>>>> testcontainers-python:
>>>>>>>>> https://github.com/testcontainers/testcontainers-python
>>>>>>>>> testcontainers-dotnet
>>>>>>>>> <
>> https://github.com/testcontainers/testcontainers-pythontestcontainer
>>>>>> s-
>>>>>> dotnet
>>>>>>>>> :
>>>>>>>>> https://github.com/testcontainers/testcontainers-dotnet
>>>>>>>>> testcontainers-node
>>>>>>>>> <
>> https://github.com/testcontainers/testcontainers-dotnettestcontainer
>>>>>> s-
>>>>>> node
>>>>>>>>> :
>>>>>>>>> https://github.com/testcontainers/testcontainers-node
>>>>>>>>>
>>>>>>>>> Testcontainers-java is the most advanced library of these of
>>>>> course.
>>>>>>> The
>>>>>>>>> others still lack some functionality, but I guess that we
>>>>>>>>> could also
>>>>>>> just
>>>>>>>>> contribute to those libraries if an important feature is
>>>>>>>>> missing for
>>>>>>> us.
>>>>>>>>> I wanted to use Testcontainers for Gremlin.Net, but
>>>>>>> testcontainers-dotnet
>>>>>>>>> unfortunately can only use images that are available in some
>>>>>> registry.
>>>>>>>> So,
>>>>>>>>> we would have to push a Docker Image of Gremlin Server to
>>>>>>>>> DockerHub
>>>>>> to
>>>>>>>> use
>>>>>>>>> it in our tests. That means that we could either only test
>> the
>>>>>>>>> GLVs
>>>>>>>> against
>>>>>>>>> already released versions or we could publish something like
>>>>>>>>> nightly
>>>>>>>> build
>>>>>>>>> images of Gremlin Server.
>>>>>>>>> I originally wanted to wait until testcontainers-dotnet is
>>>>>>>>> able to
>>>>>> use
>>>>>>>>> images that are only available locally (testcontainers-java
>>>>>>>>> already
>>>>>> has
>>>>>>>>> that capability), but now that I think more about this, I
>>>>>>>>> think that
>>>>>> it
>>>>>>>>> makes actually more sense to use images that are available in
>>>>>>>>> the
>>>>>>>> registry
>>>>>>>>> as every developer would otherwise have to build test images
>>>>>>>>> first
>>>>>>>> locally
>>>>>>>>> in order to use them for tests.
>>>>>>>>>
>>>>>>>>> So, I suggest that you guys take a look at Testcontainers to
>>>>>>>>> see
>>>>>>> whether
>>>>>>>>> you agree that it's a good solution to our problem and then
>> we
>>>>>>>>> can
>>>>>> see
>>>>>>>>> whether we want to publish nightly builds for the GLV tests
>> or
>>>>>>>>> how we
>>>>>>>> want
>>>>>>>>> to handle that.
>>>>>>>>>
>>>>>>>>> -----Ursprüngliche Nachricht-----
>>>>>>>>> Von: Stephen Mallette <sp...@gmail.com>
>>>>>>>>> Gesendet: Dienstag, 26. Februar 2019 22:24
>>>>>>>>> An: dev@tinkerpop.apache.org
>>>>>>>>> Betreff: docker and GLV development
>>>>>>>>>
>>>>>>>>> Anyone have any idea how to make it so that docker can be
>>>>>>>>> useful for helping to streamline GLV development? I'd love to
>>>>>>>>> be able to do
>>>>>>>> something
>>>>>>>>> like:
>>>>>>>>>
>>>>>>>>> docker/gremlin-server.sh -test
>>>>>>>>>
>>>>>>>>> which would start Gremlin Server with our standard test
>>>>>>>>> configuration (currently built into the maven tool chain, but
>>>>>>>>> I think that could be extracted without too much difficulty).
>>>>>>>>> That way, we could easily
>>>>>> open
>>>>>>> up
>>>>>>>>> whatever GLV specific IDE we wanted and run tests directly in
>>>>>>>>> there,
>>>>>>> use
>>>>>>>>> the debugger, etc.
>>>>>>>>>
>>>>>>>>> Sound useful? and if so, is that something that is
>>>>>>>>> easy/possible? and
>>>>>>>> then
>>>>>>>>> further....any volunteers with the know-how to get that setup
>>>>>>>>> (i'm
>>>>>>> happy
>>>>>>>> to
>>>>>>>>> assist however i can)?
>>>>>>>>>
>>>>>>>>> btw, if you have a nicer way to set up an environment for GLV
>>>>>>> development
>>>>>>>>> and docker is a waste of time, please clue me in. i don't
>> mind
>>>>>>>>> using
>>>>>>> the
>>>>>>>>> maven integration but i feel like i could be more efficient
>>>>>>>>> with a
>>>>>> more
>>>>>>>>> streamlined setup.
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>
Re: docker and GLV development
Posted by Stephen Mallette <sp...@gmail.com>.
The server starts but something seems messed up with the ports. It's like I
can't connect to the 45940 port from the host. I tried a couple things I
saw on stackoverflow but nothing i tried seemed to fix it. it's probably
something simple...any ideas?
On Fri, Mar 1, 2019 at 10:05 AM Robert Dale <ro...@gmail.com> wrote:
> I made a first attempt. Let me know what you think.
> https://github.com/apache/tinkerpop/pull/1075
>
> Robert Dale
>
>
> On Wed, Feb 27, 2019 at 11:32 AM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > > If we agree on Testcontainers in general, then I can give this a try
> and
> > use Testcontainers in one of our Java projects that need Gremlin Server
> for
> > testing (gremlin-driver maybe?).
> >
> > The java integration tests for Gremlin Server flip a lot of configuration
> > bits per test. I don't know how that works with testcontainers and if it
> > will make our testing there any better. There are also a number of tests
> > that stop/start the server under different conditions - not sure how that
> > works with testcontainers either. Finally, wouldn't we expect adding
> docker
> > to the mix to slow down our integration test runs for java?
> >
> > I don't want to seem like i'm completely against testcontainers btw. I
> just
> > wonder if a script doesn't give us a 100% better experience than what we
> > have now (which is nothing). If so, and the script is easy and
> zero-impact
> > without new dependencies, then I think I like that approach. Also, It
> would
> > be nice to have the script for other sorts of manual testing, such as
> > testing Gremlin Console with Gremlin Server. If we'd like to do
> > testcontainers for GLVs I suppose that can be done too, but for my
> > purposes, since I was the one who asked for help with this initially,
> > testcontainers doesn't sound like something that is an absolute
> requirement
> > for my needs.
> >
> > thanks robert/florian for all the discussion and help - appreciated.
> >
> > On Wed, Feb 27, 2019 at 10:01 AM Florian Hockmann <
> fh@florian-hockmann.de>
> > wrote:
> >
> > > Yes, except for the detail that Testcontainers could make the script
> > > obsolete if it is also used to build the image by itself. That would
> have
> > > the advantage that developers wouldn't have to execute the script to
> > build
> > > the image. (Executing the script with Testcontainers could also be a
> > > possibility, but I'm not sure how good that works.)
> > >
> > > If we agree on Testcontainers in general, then I can give this a try
> and
> > > use Testcontainers in one of our Java projects that need Gremlin Server
> > for
> > > testing (gremlin-driver maybe?).
> > > With such a PR it would probably also be easier for others to see what
> > > advantages / disadvantages using Testcontainers would have.
> > >
> > > I think using a Java project first makes most sense as we can use all
> > > functionality Testcontainers has to offer there. The downside is of
> > course
> > > that we might not have all the functionality afterwards for other
> > languages
> > > and then we might need to extend the Testcontainers libraries for the
> > > different languages. (My assumption here is that we won't need that
> much
> > > functionality so it should be limited to 1 or 2 missing features.)
> > > Alternatively, we could also start with a GLV where the Testcontainers
> > > library doesn't have the same functionality which probably means that
> we
> > > need a workaround to get an image of our current Gremlin Server version
> > for
> > > testing.
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Robert Dale <ro...@gmail.com>
> > > Gesendet: Mittwoch, 27. Februar 2019 15:49
> > > An: dev@tinkerpop.apache.org
> > > Betreff: Re: docker and GLV development
> > >
> > > I think there are two components here.
> > > 1) Script to build the dockerized test server. Includes ability to
> start
> > > container manually, externally from tests.
> > > 2) Testcontainers integration which uses the docker image from #1.
> > > Testcontainers could also include executing the script in #1 to build
> the
> > > container.
> > >
> > > Does this sound right to you?
> > >
> > > Robert Dale
> > >
> > >
> > > On Wed, Feb 27, 2019 at 8:48 AM Florian Hockmann <
> fh@florian-hockmann.de
> > >
> > > wrote:
> > >
> > > > > I do like the idea of being able to test against older versions
> > though.
> > > >
> > > > That shouldn't be a problem in general as we could use multiple
> > > > different images. See for example this PR that adds Testcontainers to
> > > > JanusGraph to test different Elasticsearch versions:
> > > > https://github.com/JanusGraph/janusgraph/pull/1409
> > > >
> > > > > Yes, an on-the-fly container would work
> > > >
> > > > We could however also use an on-the-fly container with
> Testcontainers.
> > > > We would just need some automation to build the image in the
> > background.
> > > > What if we would build an image during the usual Maven build for
> > > > testing and give it a descriptive tag like "testing". Then, we could
> > > > use that tag with Testcontainers in addition to tags for already
> > > released versions.
> > > > Testcontainers-Java can apparently even build images by itself. So,
> > > > that could also be a possibility to get an image with the latest
> > > > Gremlin Server with the current (unreleased) changes.
> > > >
> > > > In general, I would really like to get away from a situation where we
> > > > have to start some script in the background before tests can be
> > executed.
> > > >
> > > > @Robert:
> > > > > What is the test configuration/data for GLVs? Could you list the
> > > > paths/files?
> > > >
> > > > For Gremlin.Net for example, you can find the automation happening in
> > > > the pom.xml of the testing part:
> > > >
> > > >
> https://github.com/apache/tinkerpop/blob/e81280836feed5df8d6adc8f3a03d
> > > > 09988282ea5/gremlin-dotnet/test/pom.xml#L179
> > > >
> > > > It calls the test-server-start.groovy script before the tests are
> > > > executed and afterwards test-server-stop.groovy script to stop the
> > > server again.
> > > > This has the obvious downside that the tests can only be executed
> from
> > > > the Maven build which isn't the build tool .NET developers would use
> > > > and also not the tool the IDE will use (the situation is basically
> the
> > > > same for other GLVs).
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Robert Dale <ro...@gmail.com>
> > > > Gesendet: Mittwoch, 27. Februar 2019 13:52
> > > > An: dev@tinkerpop.apache.org
> > > > Betreff: Re: docker and GLV development
> > > >
> > > > I'm surprised Daniel doesn't have this done already ;-)
> > > >
> > > > What is the test configuration/data for GLVs? Could you list the
> > > > paths/files?
> > > >
> > > > Robert Dale
> > > >
> > > >
> > > > On Wed, Feb 27, 2019 at 7:49 AM Stephen Mallette
> > > > <sp...@gmail.com>
> > > > wrote:
> > > >
> > > > > Yes, an on-the-fly container would work assuming it used the test
> > > > > configuration/data for GLVs. That's why i suggested the command as
> I
> > > > > did because it was similar to docker/build.sh.
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2019 at 7:25 AM Robert Dale <ro...@gmail.com>
> > wrote:
> > > > >
> > > > > > Do you want to create a docker container on-the-fly using the
> > > > > > current working copy kind of like we do for docker/build.sh?
> > > > > >
> > > > > > Robert Dale
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2019 at 7:16 AM Stephen Mallette
> > > > > > <sp...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > testcontainers seems neat as it gives some good integration and
> > > > > > automation,
> > > > > > > however reliance on nightly snapshots doesn't sound so great.
> > > > > > > There are times when changes are occurring on both client and
> > > > > > > server so a local up-to-the-moment image would be necessary. I
> > > > > > > also think we might want
> > > > > to
> > > > > > > avoid nightly snapshots of anything until the ASF figures out
> > > > > > > more
> > > > > > clearly
> > > > > > > how those play into projects (there is a bit of discussion
> going
> > > > > > > on
> > > > > with
> > > > > > > that now). I do like the idea of being able to test against
> > > > > > > older
> > > > > > versions
> > > > > > > though. Ensuring backward compatibility.is a nice thought.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 27, 2019 at 4:38 AM Florian Hockmann <
> > > > > fh@florian-hockmann.de
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Fantastic idea, I often missed the ability to just start
> tests
> > > > > > > > for Gremlin.Net from my IDE.
> > > > > > > >
> > > > > > > > Luckily for us, there is a project that is aimed exactly at
> > > > > > > > problems
> > > > > > like
> > > > > > > > this:
> > > > > > > > https://www.testcontainers.org/
> > > > > > > >
> > > > > > > > The basic idea of Testcontainers is that you define a Docker
> > > > > container
> > > > > > in
> > > > > > > > your tests for whatever external dependencies you need.
> > > > > > > > Databases
> > > > > are a
> > > > > > > > frequent example where Testcontainers makes sense. For us,
> > > > > > > > Gremlin
> > > > > > Server
> > > > > > > > would be such a container.
> > > > > > > > Testcontainers then uses the ability of the testing framework
> > > (e.g.
> > > > > > > JUnit)
> > > > > > > > to start the Docker container for the tests and waits until
> > > > > > > > they are
> > > > > > > fully
> > > > > > > > started (how they are started can be configured for each
> > > > > > > > container differently). After the tests have completed, it
> > > > > > > > also stops the
> > > > > > > containers
> > > > > > > > again.
> > > > > > > > That way, you don't have to manage the containers manually
> > > anymore.
> > > > > > > > Everything is done in the background by the IDE / the build
> > tool.
> > > > > > > >
> > > > > > > > Testcontainers was originally a Java library, but they now
> > > > > > > > also have libraries for different languages, including the
> > > > > > > > ones for which we
> > > > > have
> > > > > > > > GLVs:
> > > > > > > >
> > > > > > > > testcontainers-python:
> > > > > > > > https://github.com/testcontainers/testcontainers-python
> > > > > > > > testcontainers-dotnet
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> https://github.com/testcontainers/testcontainers-pythontestcontainer
> > > > > s-
> > > > > dotnet
> > > > > > > >:
> > > > > > > > https://github.com/testcontainers/testcontainers-dotnet
> > > > > > > > testcontainers-node
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> https://github.com/testcontainers/testcontainers-dotnettestcontainer
> > > > > s-
> > > > > node
> > > > > > > >:
> > > > > > > > https://github.com/testcontainers/testcontainers-node
> > > > > > > >
> > > > > > > > Testcontainers-java is the most advanced library of these of
> > > > course.
> > > > > > The
> > > > > > > > others still lack some functionality, but I guess that we
> > > > > > > > could also
> > > > > > just
> > > > > > > > contribute to those libraries if an important feature is
> > > > > > > > missing for
> > > > > > us.
> > > > > > > >
> > > > > > > > I wanted to use Testcontainers for Gremlin.Net, but
> > > > > > testcontainers-dotnet
> > > > > > > > unfortunately can only use images that are available in some
> > > > > registry.
> > > > > > > So,
> > > > > > > > we would have to push a Docker Image of Gremlin Server to
> > > > > > > > DockerHub
> > > > > to
> > > > > > > use
> > > > > > > > it in our tests. That means that we could either only test
> the
> > > > > > > > GLVs
> > > > > > > against
> > > > > > > > already released versions or we could publish something like
> > > > > > > > nightly
> > > > > > > build
> > > > > > > > images of Gremlin Server.
> > > > > > > > I originally wanted to wait until testcontainers-dotnet is
> > > > > > > > able to
> > > > > use
> > > > > > > > images that are only available locally (testcontainers-java
> > > > > > > > already
> > > > > has
> > > > > > > > that capability), but now that I think more about this, I
> > > > > > > > think that
> > > > > it
> > > > > > > > makes actually more sense to use images that are available in
> > > > > > > > the
> > > > > > > registry
> > > > > > > > as every developer would otherwise have to build test images
> > > > > > > > first
> > > > > > > locally
> > > > > > > > in order to use them for tests.
> > > > > > > >
> > > > > > > > So, I suggest that you guys take a look at Testcontainers to
> > > > > > > > see
> > > > > > whether
> > > > > > > > you agree that it's a good solution to our problem and then
> we
> > > > > > > > can
> > > > > see
> > > > > > > > whether we want to publish nightly builds for the GLV tests
> or
> > > > > > > > how we
> > > > > > > want
> > > > > > > > to handle that.
> > > > > > > >
> > > > > > > > -----Ursprüngliche Nachricht-----
> > > > > > > > Von: Stephen Mallette <sp...@gmail.com>
> > > > > > > > Gesendet: Dienstag, 26. Februar 2019 22:24
> > > > > > > > An: dev@tinkerpop.apache.org
> > > > > > > > Betreff: docker and GLV development
> > > > > > > >
> > > > > > > > Anyone have any idea how to make it so that docker can be
> > > > > > > > useful for helping to streamline GLV development? I'd love to
> > > > > > > > be able to do
> > > > > > > something
> > > > > > > > like:
> > > > > > > >
> > > > > > > > docker/gremlin-server.sh -test
> > > > > > > >
> > > > > > > > which would start Gremlin Server with our standard test
> > > > > > > > configuration (currently built into the maven tool chain, but
> > > > > > > > I think that could be extracted without too much difficulty).
> > > > > > > > That way, we could easily
> > > > > open
> > > > > > up
> > > > > > > > whatever GLV specific IDE we wanted and run tests directly in
> > > > > > > > there,
> > > > > > use
> > > > > > > > the debugger, etc.
> > > > > > > >
> > > > > > > > Sound useful? and if so, is that something that is
> > > > > > > > easy/possible? and
> > > > > > > then
> > > > > > > > further....any volunteers with the know-how to get that setup
> > > > > > > > (i'm
> > > > > > happy
> > > > > > > to
> > > > > > > > assist however i can)?
> > > > > > > >
> > > > > > > > btw, if you have a nicer way to set up an environment for GLV
> > > > > > development
> > > > > > > > and docker is a waste of time, please clue me in. i don't
> mind
> > > > > > > > using
> > > > > > the
> > > > > > > > maven integration but i feel like i could be more efficient
> > > > > > > > with a
> > > > > more
> > > > > > > > streamlined setup.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
>