Re: [DISCUSS] Questions About deploy/ Directory and Vendor-Specific Artifacts

[Dianjin Wang <wa...@gmail.com>]

Hi,

As there has been no further input from the community, I created a PR
to remove the unused files under the `deploy` dir:
https://github.com/apache/cloudberry/pull/1090. Please help review it.

I kept the `deploy/build` files unremoved as a revision base in the
coming months to avoid creating them from scratch.

Best,
Dianjin Wang

On Fri, Apr 25, 2025 at 6:01 PM Ed Espino <es...@apache.org> wrote:
>
> Hi Dianjin,
>
> Thanks for the detailed context and thoughtful suggestions.
>
> I agree with your assessment that much of the content under
> deploy/—particularly
> docker/, k8s/, script/, vagrant, cbdb_deploy.sh, and cbdb_env.sh—is
> outdated and not aligned with Cloudberry's current or future direction as a
> community-led Apache project.
>
> Given that these artifacts rely on internal or deprecated setups and are
> not part of any active public workflow, I propose we proceed with deleting
> the entire deploy/ directory. This would remove ambiguity for new
> contributors and make the codebase cleaner and more compliant with ASF’s
> principles of openness and transparency.
>
> We can leave a placeholder issue or GitHub Discussion open to encourage
> community contributions toward a modern, accessible deployment workflow.
> Docker Compose seems like a natural starting point, and it would be great
> to see contributors take the lead in shaping tools that better reflect
> real-world usage scenarios across platforms.
>
> Let’s continue to gather input from others, but unless there are
> objections, we could aim to move forward with the cleanup in the next
> couple of weeks.
>
> Best,
>
> -=e
>
> -=e
> --
> Ed Espino
> Apache Cloudberry (Incubating) & MADlib
>
>
> On Fri, Apr 25, 2025 at 1:28 AM Dianjin Wang <wa...@gmail.com> wrote:
>
> > Hi Ed,
> >
> > Thanks for bringing this up — I’d like to contribute some context and
> > suggestions as follows.
> >
> > Best,
> > Dianjin Wang
> >
> >
> > On Fri, Apr 25, 2025 at 5:32 AM Ed Espino <es...@apache.org> wrote:
> > >
> > > - Is the `deploy/` directory still in active use or maintained?
> >
> > Files in `deploy/build/` were originally scattered under the top-level
> > dir in Greenplum to guide users in building the project. We moved them
> > here for better organization. These scripts and files are used to help
> > users build Cloudberry on Linux and macOS. However, they do need
> > updates — including bash compatibility, dependency management, and OS
> > support. We might cherry-pick updates from Greenplum or revise them
> > ourselves.
> >
> > The `vagrant/` directory, previously under `src/tools/vagrant/` in
> > Greenplum, was also moved here. However, these files have not been
> > maintained upstream for over 5–6 years and use VirtualBox as the
> > provider, which has poor support for Apple silicon in my previous
> > testing (I'm not sure how it goes now).
> >
> > > - Was it contributed primarily to support internal workflows, or is it
> > > intended to serve as a general deployment guide for the community?
> >
> > Files under `k8s/`, `docker/`, `script/`, as well as `cbdb_deploy.sh`
> > and `cbdb_env.sh`, were initially to support internal workflows. As
> > you can see, some internal URLs referenced in the files are not usable
> > by the general public.
> >
> > To my knowledge, these are also not used in the public Cloudberry
> > workflows.
> >
> > > - If it isn’t currently usable without access to internal resources,
> > should
> > > we clarify that in the README or revisit how it’s presented?
> >
> > For legacy and unused files, I propose we remove them entirely:
> > - `docker/`
> > - `k8s/`
> > - `script`
> > - `vagrant`
> > - `cbdb_deploy.sh`
> > - `cbdb_env.sh`
> >
> > > - Would it be valuable to provide a community-accessible deployment path
> > > using standard open tools (e.g., Docker Compose, Helm charts, Terraform)?
> >
> > +1. Starting with Docker Compose would be a great first step. It could
> > help both new users and contributors get started more easily.
> >
> > > - Does the current setup align with ASF expectations around transparency
> > > and community-driven development?
> >
> > As suggested above, once we clean up the internal and deprecated
> > files, it should be more aligned with ASF’s expectations.
> >
> > > While it’s entirely understandable that early artifacts might reflect
> > their
> > > original environment, continued reliance on internal infrastructure could
> > > make onboarding harder for external users — and may not reflect the
> > Apache
> > > spirit of openness and independence.
> >
> > Agree, I acknowledged with the vendor engineers that these files are
> > not used anymore in their internal workflows. So we can clean up these
> > files for better openness and public access.
> >
> > ～～～
> >
> > BTW, I started a GitHub Discussion on cleanup of `deploy` dir[1] back
> > in Feb, but we can continue the deep discussion on this mailing thread
> > to gather more feedback and suggestions.
> >
> > [1] https://github.com/apache/cloudberry/discussions/948
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
> > For additional commands, e-mail: dev-help@cloudberry.apache.org
> >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: dev-help@cloudberry.apache.org

Re: [DISCUSS] Questions About deploy/ Directory and Vendor-Specific Artifacts

[Ed Espino <es...@apache.org>]

I provided one approval for the PR. We need one more.

Thank you Dianjin.

-=e

Ed Espino
925.389.4640


On Tue, May 6, 2025 at 8:59 PM Dianjin Wang <wa...@gmail.com> wrote:

> Hi,
>
> As there has been no further input from the community, I created a PR
> to remove the unused files under the `deploy` dir:
> https://github.com/apache/cloudberry/pull/1090. Please help review it.
>
> I kept the `deploy/build` files unremoved as a revision base in the
> coming months to avoid creating them from scratch.
>
> Best,
> Dianjin Wang
>
> On Fri, Apr 25, 2025 at 6:01 PM Ed Espino <es...@apache.org> wrote:
> >
> > Hi Dianjin,
> >
> > Thanks for the detailed context and thoughtful suggestions.
> >
> > I agree with your assessment that much of the content under
> > deploy/—particularly
> > docker/, k8s/, script/, vagrant, cbdb_deploy.sh, and cbdb_env.sh—is
> > outdated and not aligned with Cloudberry's current or future direction
> as a
> > community-led Apache project.
> >
> > Given that these artifacts rely on internal or deprecated setups and are
> > not part of any active public workflow, I propose we proceed with
> deleting
> > the entire deploy/ directory. This would remove ambiguity for new
> > contributors and make the codebase cleaner and more compliant with ASF’s
> > principles of openness and transparency.
> >
> > We can leave a placeholder issue or GitHub Discussion open to encourage
> > community contributions toward a modern, accessible deployment workflow.
> > Docker Compose seems like a natural starting point, and it would be great
> > to see contributors take the lead in shaping tools that better reflect
> > real-world usage scenarios across platforms.
> >
> > Let’s continue to gather input from others, but unless there are
> > objections, we could aim to move forward with the cleanup in the next
> > couple of weeks.
> >
> > Best,
> >
> > -=e
> >
> > -=e
> > --
> > Ed Espino
> > Apache Cloudberry (Incubating) & MADlib
> >
> >
> > On Fri, Apr 25, 2025 at 1:28 AM Dianjin Wang <wa...@gmail.com>
> wrote:
> >
> > > Hi Ed,
> > >
> > > Thanks for bringing this up — I’d like to contribute some context and
> > > suggestions as follows.
> > >
> > > Best,
> > > Dianjin Wang
> > >
> > >
> > > On Fri, Apr 25, 2025 at 5:32 AM Ed Espino <es...@apache.org> wrote:
> > > >
> > > > - Is the `deploy/` directory still in active use or maintained?
> > >
> > > Files in `deploy/build/` were originally scattered under the top-level
> > > dir in Greenplum to guide users in building the project. We moved them
> > > here for better organization. These scripts and files are used to help
> > > users build Cloudberry on Linux and macOS. However, they do need
> > > updates — including bash compatibility, dependency management, and OS
> > > support. We might cherry-pick updates from Greenplum or revise them
> > > ourselves.
> > >
> > > The `vagrant/` directory, previously under `src/tools/vagrant/` in
> > > Greenplum, was also moved here. However, these files have not been
> > > maintained upstream for over 5–6 years and use VirtualBox as the
> > > provider, which has poor support for Apple silicon in my previous
> > > testing (I'm not sure how it goes now).
> > >
> > > > - Was it contributed primarily to support internal workflows, or is
> it
> > > > intended to serve as a general deployment guide for the community?
> > >
> > > Files under `k8s/`, `docker/`, `script/`, as well as `cbdb_deploy.sh`
> > > and `cbdb_env.sh`, were initially to support internal workflows. As
> > > you can see, some internal URLs referenced in the files are not usable
> > > by the general public.
> > >
> > > To my knowledge, these are also not used in the public Cloudberry
> > > workflows.
> > >
> > > > - If it isn’t currently usable without access to internal resources,
> > > should
> > > > we clarify that in the README or revisit how it’s presented?
> > >
> > > For legacy and unused files, I propose we remove them entirely:
> > > - `docker/`
> > > - `k8s/`
> > > - `script`
> > > - `vagrant`
> > > - `cbdb_deploy.sh`
> > > - `cbdb_env.sh`
> > >
> > > > - Would it be valuable to provide a community-accessible deployment
> path
> > > > using standard open tools (e.g., Docker Compose, Helm charts,
> Terraform)?
> > >
> > > +1. Starting with Docker Compose would be a great first step. It could
> > > help both new users and contributors get started more easily.
> > >
> > > > - Does the current setup align with ASF expectations around
> transparency
> > > > and community-driven development?
> > >
> > > As suggested above, once we clean up the internal and deprecated
> > > files, it should be more aligned with ASF’s expectations.
> > >
> > > > While it’s entirely understandable that early artifacts might reflect
> > > their
> > > > original environment, continued reliance on internal infrastructure
> could
> > > > make onboarding harder for external users — and may not reflect the
> > > Apache
> > > > spirit of openness and independence.
> > >
> > > Agree, I acknowledged with the vendor engineers that these files are
> > > not used anymore in their internal workflows. So we can clean up these
> > > files for better openness and public access.
> > >
> > > ～～～
> > >
> > > BTW, I started a GitHub Discussion on cleanup of `deploy` dir[1] back
> > > in Feb, but we can continue the deep discussion on this mailing thread
> > > to gather more feedback and suggestions.
> > >
> > > [1] https://github.com/apache/cloudberry/discussions/948
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
> > > For additional commands, e-mail: dev-help@cloudberry.apache.org
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cloudberry.apache.org
> For additional commands, e-mail: dev-help@cloudberry.apache.org
>
>