Phoenix Docker Image for Integration Tests

[Francis Chuang <fr...@apache.org>]

Hello,

I have a Phoenix and HBase all-in-one docker image[1] that we use to run 
integration tests for the Apache Calcite Avatica Go SQL driver[2].

In the past, I have sent some emails here to see if the Phoenix project 
would like to officially maintain these images to allow users to quickly 
spin up a working Phoenix HBase instance for testing and so on, but 
there hasn't been much interest.

I would like to renew this discussion again as Docker will now sunset 
all free team organizations in around a month. This means that my Docker 
organization will be deleted and the phoenix-hbase-all-in-one image will 
no longer be available.

As a temporary workaround, I can move the image to my personal account, 
but I would like to have a more permanent solution in place. Ideally, I 
would love to see the image being maintained by the Phoenix team. By 
adding some GitHub Actions workflows, it should be quite trivial for an 
image to be automatically built on each release.

Please let me know what you think.

Francis

[1] https://hub.docker.com/r/boostport/hbase-phoenix-all-in-one
[2] 
https://github.com/apache/calcite-avatica-go/blob/main/docker-compose.yml#L19

Re: Phoenix Docker Image for Integration Tests

[Francis Chuang <fr...@apache.org>]

The proposed plan sounds excellent to me! :)

I don't think we would be blocked due to HBase not producing docker 
images. Since they release binaries, we should be able to copy the 
appropriate HBase binaries into the Phoenix image during build time 
(much like how I my images are built).

I agree with the snapshots being "nice to have" as we don't strictly 
need them at the moment.

Francis

On 20/03/2023 8:20 pm, Istvan Toth wrote:
> Thank you.
> 
> Unfortunately your current images are even less up to date than the
> Cloudera one, each of the Phoenix versions built are already EOL.
> 
> We could build the Phoenix image on top of HBase, but AFAICT HBase doesn't
> publish Docker images yet, either.
> 
> As Phoenix consists of several independently released sub-projects, making
> snapshot images that contain up-to-date versions of all sub-projects is not
> trivial.
> 
> What we could do:
> 
> Minimal tasks:
> 
> - Copy the Cloudera docker image sources into the main Phoenix repo
> - Update the versions to the latest releases
> - Add scripts to generate the image on release
> - Perform any necessary admin tasks to set up the docker image
> - Add the Docker image generation task to the release process
> 
> Stretch goals:
> 
> - Generate and publish snapshot images from the main repo each commit
> - Use the snapshot versions of Omid, phoenix-thirdparty and PQS.
> - Once HBase starts publishing Docker images, use those as a base.
> 
> Would this work for your use case Francis ?
> 
> What does the rest of the Phoenix community think ?
> 
> 
> On Mon, Mar 20, 2023 at 9:19 AM Francis Chuang <fr...@apache.org>
> wrote:
> 
>> The source of my image is here:
>> https://github.com/F21/hbase-phoenix-all-in-one
>>
>> Previously, it was at
>> https://github.com/Boostport/hbase-phoenix-all-in-one, but I moved it to
>> my own personal account to avoid issues caused by docker's upcoming
>> changes.
>>
>> The ASF already has an official docker org on docker hub. As part of the
>> calcite-avatica releases, we also post images for avatica.
>>
>> See:
>> - https://hub.docker.com/r/apache
>> - https://hub.docker.com/r/apache/calcite-avatica
>>
>> This is currently how we're automatically building the avatica images
>> for each release:
>> https://github.com/apache/calcite-avatica/tree/main/docker
>>
>> Francis
>>
>> On 20/03/2023 6:28 pm, Istvan Toth wrote:
>>> We have a similar image at https://github.com/cloudera/opdb-docker, but
>> it
>>> has several drawbacks:
>>> - It is only updated sporadically
>>> - It is source only, not pushed to docker hub
>>> - It is not part of the official project.
>>>
>>> There is definitely value in having something similar that is generated
>> as
>>> part of the Phoenix CI and/or release process,
>>> which would guarantee that it is always up-to-date.
>>>
>>> I could not find the source (Dockerfiles, etc) for your image, the
>> calcite
>>> link above only seems to refer to the built image.
>>> Can you link to the sources ?
>>> I wonder if a single official image could serve both as a base for
>> Calcite
>>> tests, and general development tool,
>>> replacing the Cloudera image I linked, and how much work merging the two
>>> would be.
>>>
>>> I am also unclear on how the Docker organisation would work ? Is there a
>>> single Apache org that we could use,
>>> or is there some process to request/generate one specifically for the
>>> Phoenix project ?
>>>
>>> regards
>>> Istvan
>>>
>>> On Tue, Mar 14, 2023 at 10:26 PM Francis Chuang <
>> francischuang@apache.org>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have a Phoenix and HBase all-in-one docker image[1] that we use to run
>>>> integration tests for the Apache Calcite Avatica Go SQL driver[2].
>>>>
>>>> In the past, I have sent some emails here to see if the Phoenix project
>>>> would like to officially maintain these images to allow users to quickly
>>>> spin up a working Phoenix HBase instance for testing and so on, but
>>>> there hasn't been much interest.
>>>>
>>>> I would like to renew this discussion again as Docker will now sunset
>>>> all free team organizations in around a month. This means that my Docker
>>>> organization will be deleted and the phoenix-hbase-all-in-one image will
>>>> no longer be available.
>>>>
>>>> As a temporary workaround, I can move the image to my personal account,
>>>> but I would like to have a more permanent solution in place. Ideally, I
>>>> would love to see the image being maintained by the Phoenix team. By
>>>> adding some GitHub Actions workflows, it should be quite trivial for an
>>>> image to be automatically built on each release.
>>>>
>>>> Please let me know what you think.
>>>>
>>>> Francis
>>>>
>>>> [1] https://hub.docker.com/r/boostport/hbase-phoenix-all-in-one
>>>> [2]
>>>>
>>>>
>> https://github.com/apache/calcite-avatica-go/blob/main/docker-compose.yml#L19
>>>>
>>>
>>
> 
> 

Re: Phoenix Docker Image for Integration Tests

[Istvan Toth <st...@cloudera.com.INVALID>]

Thank you.

Unfortunately your current images are even less up to date than the
Cloudera one, each of the Phoenix versions built are already EOL.

We could build the Phoenix image on top of HBase, but AFAICT HBase doesn't
publish Docker images yet, either.

As Phoenix consists of several independently released sub-projects, making
snapshot images that contain up-to-date versions of all sub-projects is not
trivial.

What we could do:

Minimal tasks:

- Copy the Cloudera docker image sources into the main Phoenix repo
- Update the versions to the latest releases
- Add scripts to generate the image on release
- Perform any necessary admin tasks to set up the docker image
- Add the Docker image generation task to the release process

Stretch goals:

- Generate and publish snapshot images from the main repo each commit
- Use the snapshot versions of Omid, phoenix-thirdparty and PQS.
- Once HBase starts publishing Docker images, use those as a base.

Would this work for your use case Francis ?

What does the rest of the Phoenix community think ?


On Mon, Mar 20, 2023 at 9:19 AM Francis Chuang <fr...@apache.org>
wrote:

> The source of my image is here:
> https://github.com/F21/hbase-phoenix-all-in-one
>
> Previously, it was at
> https://github.com/Boostport/hbase-phoenix-all-in-one, but I moved it to
> my own personal account to avoid issues caused by docker's upcoming
> changes.
>
> The ASF already has an official docker org on docker hub. As part of the
> calcite-avatica releases, we also post images for avatica.
>
> See:
> - https://hub.docker.com/r/apache
> - https://hub.docker.com/r/apache/calcite-avatica
>
> This is currently how we're automatically building the avatica images
> for each release:
> https://github.com/apache/calcite-avatica/tree/main/docker
>
> Francis
>
> On 20/03/2023 6:28 pm, Istvan Toth wrote:
> > We have a similar image at https://github.com/cloudera/opdb-docker, but
> it
> > has several drawbacks:
> > - It is only updated sporadically
> > - It is source only, not pushed to docker hub
> > - It is not part of the official project.
> >
> > There is definitely value in having something similar that is generated
> as
> > part of the Phoenix CI and/or release process,
> > which would guarantee that it is always up-to-date.
> >
> > I could not find the source (Dockerfiles, etc) for your image, the
> calcite
> > link above only seems to refer to the built image.
> > Can you link to the sources ?
> > I wonder if a single official image could serve both as a base for
> Calcite
> > tests, and general development tool,
> > replacing the Cloudera image I linked, and how much work merging the two
> > would be.
> >
> > I am also unclear on how the Docker organisation would work ? Is there a
> > single Apache org that we could use,
> > or is there some process to request/generate one specifically for the
> > Phoenix project ?
> >
> > regards
> > Istvan
> >
> > On Tue, Mar 14, 2023 at 10:26 PM Francis Chuang <
> francischuang@apache.org>
> > wrote:
> >
> >> Hello,
> >>
> >> I have a Phoenix and HBase all-in-one docker image[1] that we use to run
> >> integration tests for the Apache Calcite Avatica Go SQL driver[2].
> >>
> >> In the past, I have sent some emails here to see if the Phoenix project
> >> would like to officially maintain these images to allow users to quickly
> >> spin up a working Phoenix HBase instance for testing and so on, but
> >> there hasn't been much interest.
> >>
> >> I would like to renew this discussion again as Docker will now sunset
> >> all free team organizations in around a month. This means that my Docker
> >> organization will be deleted and the phoenix-hbase-all-in-one image will
> >> no longer be available.
> >>
> >> As a temporary workaround, I can move the image to my personal account,
> >> but I would like to have a more permanent solution in place. Ideally, I
> >> would love to see the image being maintained by the Phoenix team. By
> >> adding some GitHub Actions workflows, it should be quite trivial for an
> >> image to be automatically built on each release.
> >>
> >> Please let me know what you think.
> >>
> >> Francis
> >>
> >> [1] https://hub.docker.com/r/boostport/hbase-phoenix-all-in-one
> >> [2]
> >>
> >>
> https://github.com/apache/calcite-avatica-go/blob/main/docker-compose.yml#L19
> >>
> >
>


-- 
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------

Re: Phoenix Docker Image for Integration Tests

[Francis Chuang <fr...@apache.org>]

The source of my image is here: 
https://github.com/F21/hbase-phoenix-all-in-one

Previously, it was at 
https://github.com/Boostport/hbase-phoenix-all-in-one, but I moved it to 
my own personal account to avoid issues caused by docker's upcoming changes.

The ASF already has an official docker org on docker hub. As part of the 
calcite-avatica releases, we also post images for avatica.

See:
- https://hub.docker.com/r/apache
- https://hub.docker.com/r/apache/calcite-avatica

This is currently how we're automatically building the avatica images 
for each release: https://github.com/apache/calcite-avatica/tree/main/docker

Francis

On 20/03/2023 6:28 pm, Istvan Toth wrote:
> We have a similar image at https://github.com/cloudera/opdb-docker, but it
> has several drawbacks:
> - It is only updated sporadically
> - It is source only, not pushed to docker hub
> - It is not part of the official project.
> 
> There is definitely value in having something similar that is generated as
> part of the Phoenix CI and/or release process,
> which would guarantee that it is always up-to-date.
> 
> I could not find the source (Dockerfiles, etc) for your image, the calcite
> link above only seems to refer to the built image.
> Can you link to the sources ?
> I wonder if a single official image could serve both as a base for Calcite
> tests, and general development tool,
> replacing the Cloudera image I linked, and how much work merging the two
> would be.
> 
> I am also unclear on how the Docker organisation would work ? Is there a
> single Apache org that we could use,
> or is there some process to request/generate one specifically for the
> Phoenix project ?
> 
> regards
> Istvan
> 
> On Tue, Mar 14, 2023 at 10:26 PM Francis Chuang <fr...@apache.org>
> wrote:
> 
>> Hello,
>>
>> I have a Phoenix and HBase all-in-one docker image[1] that we use to run
>> integration tests for the Apache Calcite Avatica Go SQL driver[2].
>>
>> In the past, I have sent some emails here to see if the Phoenix project
>> would like to officially maintain these images to allow users to quickly
>> spin up a working Phoenix HBase instance for testing and so on, but
>> there hasn't been much interest.
>>
>> I would like to renew this discussion again as Docker will now sunset
>> all free team organizations in around a month. This means that my Docker
>> organization will be deleted and the phoenix-hbase-all-in-one image will
>> no longer be available.
>>
>> As a temporary workaround, I can move the image to my personal account,
>> but I would like to have a more permanent solution in place. Ideally, I
>> would love to see the image being maintained by the Phoenix team. By
>> adding some GitHub Actions workflows, it should be quite trivial for an
>> image to be automatically built on each release.
>>
>> Please let me know what you think.
>>
>> Francis
>>
>> [1] https://hub.docker.com/r/boostport/hbase-phoenix-all-in-one
>> [2]
>>
>> https://github.com/apache/calcite-avatica-go/blob/main/docker-compose.yml#L19
>>
> 

Re: Phoenix Docker Image for Integration Tests

[Istvan Toth <st...@apache.org>]

We have a similar image at https://github.com/cloudera/opdb-docker, but it
has several drawbacks:
- It is only updated sporadically
- It is source only, not pushed to docker hub
- It is not part of the official project.

There is definitely value in having something similar that is generated as
part of the Phoenix CI and/or release process,
which would guarantee that it is always up-to-date.

I could not find the source (Dockerfiles, etc) for your image, the calcite
link above only seems to refer to the built image.
Can you link to the sources ?
I wonder if a single official image could serve both as a base for Calcite
tests, and general development tool,
replacing the Cloudera image I linked, and how much work merging the two
would be.

I am also unclear on how the Docker organisation would work ? Is there a
single Apache org that we could use,
or is there some process to request/generate one specifically for the
Phoenix project ?

regards
Istvan

On Tue, Mar 14, 2023 at 10:26 PM Francis Chuang <fr...@apache.org>
wrote:

> Hello,
>
> I have a Phoenix and HBase all-in-one docker image[1] that we use to run
> integration tests for the Apache Calcite Avatica Go SQL driver[2].
>
> In the past, I have sent some emails here to see if the Phoenix project
> would like to officially maintain these images to allow users to quickly
> spin up a working Phoenix HBase instance for testing and so on, but
> there hasn't been much interest.
>
> I would like to renew this discussion again as Docker will now sunset
> all free team organizations in around a month. This means that my Docker
> organization will be deleted and the phoenix-hbase-all-in-one image will
> no longer be available.
>
> As a temporary workaround, I can move the image to my personal account,
> but I would like to have a more permanent solution in place. Ideally, I
> would love to see the image being maintained by the Phoenix team. By
> adding some GitHub Actions workflows, it should be quite trivial for an
> image to be automatically built on each release.
>
> Please let me know what you think.
>
> Francis
>
> [1] https://hub.docker.com/r/boostport/hbase-phoenix-all-in-one
> [2]
>
> https://github.com/apache/calcite-avatica-go/blob/main/docker-compose.yml#L19
>