You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Mick Semb Wever <mc...@apache.org> on 2025/04/27 16:20:19 UTC

[DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Pulling out to a separate thread, as requested.

Should golang auto-install when it's not found, when executing `ant
gen-doc` ?

Our build system has a number of path dependencies: e.g. ant, maven,
python, golang, gradle, git, grep, curl, docker, make.  Gradle and golang
will auto-install if not found.

Golang was recently introduced for CASSANDRA-19915, helping consolidate and
improve the website documentation pages for our native protocol
specification.  This was work that came over from the gocql community.
https://cassandra.apache.org/doc/latest/cassandra/reference/native-protocol.html

https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc

Both python and golang are dependencies for in-tree docs generation (`ant
gen-doc`).  Python is also used for tests.

While python is expected to be found, golang like gradle will be downloaded
and installed if it is not.  But unlike gradle, golang is only temporarily
installed into your $TMPDIR and only used for the native protocol doc page
generation.
https://github.com/apache/cassandra/blob/trunk/doc/scripts/process-native-protocol-specs-in-docker.sh


If you don't want to be executing gen-doc (note that `ant jar` doesn't
already), you can just add `-Dant.gen-doc.skip=true` to your ant
cmdline/config.  `ant check` does invoke it, because code changes can break
doc generation.

Is this a problem ? Why ?  What can/should be done ?




On Fri, 25 Apr 2025 at 13:13, Alex Petrov <al...@coffeenco.de> wrote:

> Thank you for input.
>
> I will disable gen-doc in addition to checkstyle locally, but I would
> still suggest that in short-term we should at very least remove/disable the
> part that auto-installs go (or at least have a discussion or vote about it,
> in case folks believe auto-installing is a good idea), but ideally in the
> long term we can find a solution that does not require go.
>
> On Thu, Apr 24, 2025, at 12:39 PM, Štefan Miklošovič wrote:
>
> I don't see it necessary to port Gradle stuff to Ant. I have noticed zero
> problems with it after CEP-15 merge. Basically a non-event.
>
> It just looks strange that we have Ant + Maven poms / resolver wired into
> it + submodule on Gradle. I mean ... wow. There is nothing wrong per se but
> ... strange. One would expect that more into the future we go, the more
> coherent / consistent everything related to the build system will be, not
> the other way around. But I totally get that this is just how things are
> for now and there is nobody to blame really.
>
> On Thu, Apr 24, 2025 at 10:39 AM Benedict <be...@apache.org> wrote:
>
>
> We should separate out any grade discussion. I’m happy to migrate accord
> to ant if that’s the project preference, but there’s continual discussion
> to begin modularising Cassandra (at least a little), and a proposal to use
> grade for the modules - which might be a happy medium for everyone’s
> competing priorities.
>
>
> On 24 Apr 2025, at 07:26, Mick Semb Wever <mc...@apache.org> wrote:
>
> 
> Does this also apply to gradle, which now gets downloaded and installed,
> and is the most recent addition ?
>
> The python requirement from gen-doc has been around for over three years
> now.
>
> I agree with the rationale that `ant` should default to `ant check`,
> keeping newcomers in mind while being more just a final pre-commit action
> for seasoned devs where `ant jar` is the typical dev loop.  I'm sure this
> has been discussed in the past.
>
>
>
> On Thu, 24 Apr 2025 at 08:16, Alex Petrov <al...@coffeenco.de> wrote:
>
>
> I didn’t say we should disable checks. I thought this was a purpose of CI
> : to check everything: there’s likely more to break semantically anyways,
> also harder to detect and fix. I’m also not proposing removing something
> that was long in place, these dependencies were introduced very recently.
>
> On Wed, Apr 23, 2025, at 11:51 PM, Jeremiah Jordan wrote:
>
> I think the default build should be to build and check everything.  I
> think that if someone is new it is better to have everything built and
> checked by default to flag issues.
>
> If someone knows what they are doing and wants to speed up the process it
> is very easy to add the right settings to the ant command so things are
> faster.
>
> -Jeremiah
>
> On Wed, Apr 23, 2025 at 4:36 PM Jordan West <jo...@gmail.com> wrote:
>
> Should we consider making that the default and then passing false
> explicitly in CI/builds? I agree with Alex it’s a bit surprising and
> shorter build times when developing would be helpful.
>
> Jordan
>
> On Wed, Apr 23, 2025 at 13:37 Mick Semb Wever <mc...@apache.org> wrote:
>
> Python and Go are used by the gen-doc target.
>
> Code changes can break these, hence it is part of `ant check`.
> It is not called by `ant jar`
>
> If you want to run check but skip it, it's to add `-Dant.gen-doc.skip=true`
>
>
>
> On Wed, 23 Apr 2025 at 22:06, Alex Petrov <al...@coffeenco.de> wrote:
>
>
> Hi folks,
>
> Building Cassandra jar has been getting increasingly slow, and now it
> looks like we depend not only on python3 (which was already not optimal),
> but also on go:
>
>     ant -Dno-checkstyle=true
>
> ...
>
>      [exec] python3 ./scripts/gen-nodetool-docs.py
>      [exec] python3 ./scripts/convert_yaml_to_adoc.py
> ../conf/cassandra.yaml
> ./modules/cassandra/pages/managing/configuration/cass_yaml_file.adoc
>      [exec] ./scripts/process-native-protocol-specs-in-docker.sh
>      [exec] Go env not found in your system, proceeding with installation.
>      [exec] Downloading Go 1.23.1...
>      [exec] Installing Go 1.23.1...
>      [exec] Building the cqlprotodoc...
>      [exec] Cloning into 'cassandra-website'...
>      [exec] Your branch is up to date with 'origin/trunk'.
>      [exec] go: downloading github.com/mvdan/xurls v1.1.0
>      [exec] Processing the .spec files...
>
> I personally consider this extremely dangerous, but also unnecessary. My
> current stance is that functionality introducing python3 and go should be
> moved to a separate task that only runs on demand / ci / release.  I
> welcome convincing arguments that would suggest otherwise.
>
> If you agree we should not require python and go to run `ant
> -Dno-checkstyle=true`, please also write a short message, this will be very
> helpful as well.
>
> Thank you,
> --Alex
>
>
>
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Mick Semb Wever <mc...@apache.org>.
Correct, the website is not built here.  `ant gen-doc` is just about the
code to adoc generated pages.

The asciidoc to webpage (antora build) is only done by cassandra-website.
We would like to be able to build a html/man bundle and ship it in
releases.  But that's not decided, and it wouldn't be part of `ant check`
or any other normal dev target.



On Mon, 28 Apr 2025 at 20:16, Patrick McFadin <pm...@gmail.com> wrote:

> I don't think the website is generated outside of the Cassandra website
> repo. Did I miss that it's built when building the server?
>
> On Mon, Apr 28, 2025 at 10:19 AM Jon Haddad <jo...@rustyrazorblade.com>
> wrote:
>
>> I strongly prefer we didn't install things on users systems they didn't
>> ask for.  I think we're also wasting people's time by building the website
>> by default.
>>
>> ant isn't a popular tool.  The more we ask people to know about it, the
>> more we turn people away from the project.  I'm personally unwilling to
>> invest the time in learning what's otherwise a dead technology, and I can't
>> be the only one.  We should by default do the thing that most people want
>> by default, which is building the server.
>>
>> Jon
>>
>> On Mon, Apr 28, 2025 at 9:54 AM Alex Petrov <al...@coffeenco.de> wrote:
>>
>>> I’ve talked to some folks privately and looks like ant jar is a more
>>> common workflow than just ant. I also don’t mind disabling check style and
>>> doc gen for my purposes. But I would suggest to not auto install, or hide
>>> installation behind a flag and suggest user to opt in to this.
>>>
>>> One more reason to not install is that this way we commit to maintaining
>>> installing go.
>>>
>>> Ideally we would not pull in one more language for a small task but it’s
>>> hard for me to judge how difficult it would’ve been to implement this
>>> feature differently.
>>>
>>> On Sun, Apr 27, 2025, at 6:20 PM, Mick Semb Wever wrote:
>>>
>>> Pulling out to a separate thread, as requested.
>>>
>>> Should golang auto-install when it's not found, when executing `ant
>>> gen-doc` ?
>>>
>>> Our build system has a number of path dependencies: e.g. ant, maven,
>>> python, golang, gradle, git, grep, curl, docker, make.  Gradle and golang
>>> will auto-install if not found.
>>>
>>> Golang was recently introduced for CASSANDRA-19915, helping consolidate
>>> and improve the website documentation pages for our native protocol
>>> specification.  This was work that came over from the gocql community.
>>>
>>> https://cassandra.apache.org/doc/latest/cassandra/reference/native-protocol.html
>>>
>>> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc
>>>
>>> Both python and golang are dependencies for in-tree docs generation
>>> (`ant gen-doc`).  Python is also used for tests.
>>>
>>> While python is expected to be found, golang like gradle will be
>>> downloaded and installed if it is not.  But unlike gradle, golang is only
>>> temporarily installed into your $TMPDIR and only used for the native
>>> protocol doc page generation.
>>>
>>> https://github.com/apache/cassandra/blob/trunk/doc/scripts/process-native-protocol-specs-in-docker.sh
>>>
>>>
>>> If you don't want to be executing gen-doc (note that `ant jar` doesn't
>>> already), you can just add `-Dant.gen-doc.skip=true` to your ant
>>> cmdline/config.  `ant check` does invoke it, because code changes can break
>>> doc generation.
>>>
>>> Is this a problem ? Why ?  What can/should be done ?
>>>
>>>
>>>
>>>
>>> On Fri, 25 Apr 2025 at 13:13, Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> Thank you for input.
>>>
>>> I will disable gen-doc in addition to checkstyle locally, but I would
>>> still suggest that in short-term we should at very least remove/disable the
>>> part that auto-installs go (or at least have a discussion or vote about it,
>>> in case folks believe auto-installing is a good idea), but ideally in the
>>> long term we can find a solution that does not require go.
>>>
>>> On Thu, Apr 24, 2025, at 12:39 PM, Štefan Miklošovič wrote:
>>>
>>> I don't see it necessary to port Gradle stuff to Ant. I have noticed
>>> zero problems with it after CEP-15 merge. Basically a non-event.
>>>
>>> It just looks strange that we have Ant + Maven poms / resolver wired
>>> into it + submodule on Gradle. I mean ... wow. There is nothing wrong per
>>> se but ... strange. One would expect that more into the future we go, the
>>> more coherent / consistent everything related to the build system will be,
>>> not the other way around. But I totally get that this is just how things
>>> are for now and there is nobody to blame really.
>>>
>>> On Thu, Apr 24, 2025 at 10:39 AM Benedict <be...@apache.org> wrote:
>>>
>>>
>>> We should separate out any grade discussion. I’m happy to migrate accord
>>> to ant if that’s the project preference, but there’s continual discussion
>>> to begin modularising Cassandra (at least a little), and a proposal to use
>>> grade for the modules - which might be a happy medium for everyone’s
>>> competing priorities.
>>>
>>>
>>> On 24 Apr 2025, at 07:26, Mick Semb Wever <mc...@apache.org> wrote:
>>>
>>> 
>>> Does this also apply to gradle, which now gets downloaded and installed,
>>> and is the most recent addition ?
>>>
>>> The python requirement from gen-doc has been around for over three years
>>> now.
>>>
>>> I agree with the rationale that `ant` should default to `ant check`,
>>> keeping newcomers in mind while being more just a final pre-commit action
>>> for seasoned devs where `ant jar` is the typical dev loop.  I'm sure this
>>> has been discussed in the past.
>>>
>>>
>>>
>>> On Thu, 24 Apr 2025 at 08:16, Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> I didn’t say we should disable checks. I thought this was a purpose of
>>> CI : to check everything: there’s likely more to break semantically
>>> anyways, also harder to detect and fix. I’m also not proposing removing
>>> something that was long in place, these dependencies were introduced very
>>> recently.
>>>
>>> On Wed, Apr 23, 2025, at 11:51 PM, Jeremiah Jordan wrote:
>>>
>>> I think the default build should be to build and check everything.  I
>>> think that if someone is new it is better to have everything built and
>>> checked by default to flag issues.
>>>
>>> If someone knows what they are doing and wants to speed up the process
>>> it is very easy to add the right settings to the ant command so things are
>>> faster.
>>>
>>> -Jeremiah
>>>
>>> On Wed, Apr 23, 2025 at 4:36 PM Jordan West <jo...@gmail.com> wrote:
>>>
>>> Should we consider making that the default and then passing false
>>> explicitly in CI/builds? I agree with Alex it’s a bit surprising and
>>> shorter build times when developing would be helpful.
>>>
>>> Jordan
>>>
>>> On Wed, Apr 23, 2025 at 13:37 Mick Semb Wever <mc...@apache.org> wrote:
>>>
>>> Python and Go are used by the gen-doc target.
>>>
>>> Code changes can break these, hence it is part of `ant check`.
>>> It is not called by `ant jar`
>>>
>>> If you want to run check but skip it, it's to add
>>> `-Dant.gen-doc.skip=true`
>>>
>>>
>>>
>>> On Wed, 23 Apr 2025 at 22:06, Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Building Cassandra jar has been getting increasingly slow, and now it
>>> looks like we depend not only on python3 (which was already not optimal),
>>> but also on go:
>>>
>>>     ant -Dno-checkstyle=true
>>>
>>> ...
>>>
>>>      [exec] python3 ./scripts/gen-nodetool-docs.py
>>>      [exec] python3 ./scripts/convert_yaml_to_adoc.py
>>> ../conf/cassandra.yaml
>>> ./modules/cassandra/pages/managing/configuration/cass_yaml_file.adoc
>>>      [exec] ./scripts/process-native-protocol-specs-in-docker.sh
>>>      [exec] Go env not found in your system, proceeding with
>>> installation.
>>>      [exec] Downloading Go 1.23.1...
>>>      [exec] Installing Go 1.23.1...
>>>      [exec] Building the cqlprotodoc...
>>>      [exec] Cloning into 'cassandra-website'...
>>>      [exec] Your branch is up to date with 'origin/trunk'.
>>>      [exec] go: downloading github.com/mvdan/xurls v1.1.0
>>>      [exec] Processing the .spec files...
>>>
>>> I personally consider this extremely dangerous, but also unnecessary. My
>>> current stance is that functionality introducing python3 and go should be
>>> moved to a separate task that only runs on demand / ci / release.  I
>>> welcome convincing arguments that would suggest otherwise.
>>>
>>> If you agree we should not require python and go to run `ant
>>> -Dno-checkstyle=true`, please also write a short message, this will be very
>>> helpful as well.
>>>
>>> Thank you,
>>> --Alex
>>>
>>>
>>>
>>>
>>>
>>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Jon Haddad <jo...@rustyrazorblade.com>.
Sorry, I meant the docs, but re-running the `ant` task now I'm not seeing
doc generation either.  I must have misunderstood the earlier thread.

:disappears into bushes:



On Mon, Apr 28, 2025 at 11:16 AM Patrick McFadin <pm...@gmail.com> wrote:

> I don't think the website is generated outside of the Cassandra website
> repo. Did I miss that it's built when building the server?
>
> On Mon, Apr 28, 2025 at 10:19 AM Jon Haddad <jo...@rustyrazorblade.com>
> wrote:
>
>> I strongly prefer we didn't install things on users systems they didn't
>> ask for.  I think we're also wasting people's time by building the website
>> by default.
>>
>> ant isn't a popular tool.  The more we ask people to know about it, the
>> more we turn people away from the project.  I'm personally unwilling to
>> invest the time in learning what's otherwise a dead technology, and I can't
>> be the only one.  We should by default do the thing that most people want
>> by default, which is building the server.
>>
>> Jon
>>
>> On Mon, Apr 28, 2025 at 9:54 AM Alex Petrov <al...@coffeenco.de> wrote:
>>
>>> I’ve talked to some folks privately and looks like ant jar is a more
>>> common workflow than just ant. I also don’t mind disabling check style and
>>> doc gen for my purposes. But I would suggest to not auto install, or hide
>>> installation behind a flag and suggest user to opt in to this.
>>>
>>> One more reason to not install is that this way we commit to maintaining
>>> installing go.
>>>
>>> Ideally we would not pull in one more language for a small task but it’s
>>> hard for me to judge how difficult it would’ve been to implement this
>>> feature differently.
>>>
>>> On Sun, Apr 27, 2025, at 6:20 PM, Mick Semb Wever wrote:
>>>
>>> Pulling out to a separate thread, as requested.
>>>
>>> Should golang auto-install when it's not found, when executing `ant
>>> gen-doc` ?
>>>
>>> Our build system has a number of path dependencies: e.g. ant, maven,
>>> python, golang, gradle, git, grep, curl, docker, make.  Gradle and golang
>>> will auto-install if not found.
>>>
>>> Golang was recently introduced for CASSANDRA-19915, helping consolidate
>>> and improve the website documentation pages for our native protocol
>>> specification.  This was work that came over from the gocql community.
>>>
>>> https://cassandra.apache.org/doc/latest/cassandra/reference/native-protocol.html
>>>
>>> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc
>>>
>>> Both python and golang are dependencies for in-tree docs generation
>>> (`ant gen-doc`).  Python is also used for tests.
>>>
>>> While python is expected to be found, golang like gradle will be
>>> downloaded and installed if it is not.  But unlike gradle, golang is only
>>> temporarily installed into your $TMPDIR and only used for the native
>>> protocol doc page generation.
>>>
>>> https://github.com/apache/cassandra/blob/trunk/doc/scripts/process-native-protocol-specs-in-docker.sh
>>>
>>>
>>> If you don't want to be executing gen-doc (note that `ant jar` doesn't
>>> already), you can just add `-Dant.gen-doc.skip=true` to your ant
>>> cmdline/config.  `ant check` does invoke it, because code changes can break
>>> doc generation.
>>>
>>> Is this a problem ? Why ?  What can/should be done ?
>>>
>>>
>>>
>>>
>>> On Fri, 25 Apr 2025 at 13:13, Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> Thank you for input.
>>>
>>> I will disable gen-doc in addition to checkstyle locally, but I would
>>> still suggest that in short-term we should at very least remove/disable the
>>> part that auto-installs go (or at least have a discussion or vote about it,
>>> in case folks believe auto-installing is a good idea), but ideally in the
>>> long term we can find a solution that does not require go.
>>>
>>> On Thu, Apr 24, 2025, at 12:39 PM, Štefan Miklošovič wrote:
>>>
>>> I don't see it necessary to port Gradle stuff to Ant. I have noticed
>>> zero problems with it after CEP-15 merge. Basically a non-event.
>>>
>>> It just looks strange that we have Ant + Maven poms / resolver wired
>>> into it + submodule on Gradle. I mean ... wow. There is nothing wrong per
>>> se but ... strange. One would expect that more into the future we go, the
>>> more coherent / consistent everything related to the build system will be,
>>> not the other way around. But I totally get that this is just how things
>>> are for now and there is nobody to blame really.
>>>
>>> On Thu, Apr 24, 2025 at 10:39 AM Benedict <be...@apache.org> wrote:
>>>
>>>
>>> We should separate out any grade discussion. I’m happy to migrate accord
>>> to ant if that’s the project preference, but there’s continual discussion
>>> to begin modularising Cassandra (at least a little), and a proposal to use
>>> grade for the modules - which might be a happy medium for everyone’s
>>> competing priorities.
>>>
>>>
>>> On 24 Apr 2025, at 07:26, Mick Semb Wever <mc...@apache.org> wrote:
>>>
>>> 
>>> Does this also apply to gradle, which now gets downloaded and installed,
>>> and is the most recent addition ?
>>>
>>> The python requirement from gen-doc has been around for over three years
>>> now.
>>>
>>> I agree with the rationale that `ant` should default to `ant check`,
>>> keeping newcomers in mind while being more just a final pre-commit action
>>> for seasoned devs where `ant jar` is the typical dev loop.  I'm sure this
>>> has been discussed in the past.
>>>
>>>
>>>
>>> On Thu, 24 Apr 2025 at 08:16, Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> I didn’t say we should disable checks. I thought this was a purpose of
>>> CI : to check everything: there’s likely more to break semantically
>>> anyways, also harder to detect and fix. I’m also not proposing removing
>>> something that was long in place, these dependencies were introduced very
>>> recently.
>>>
>>> On Wed, Apr 23, 2025, at 11:51 PM, Jeremiah Jordan wrote:
>>>
>>> I think the default build should be to build and check everything.  I
>>> think that if someone is new it is better to have everything built and
>>> checked by default to flag issues.
>>>
>>> If someone knows what they are doing and wants to speed up the process
>>> it is very easy to add the right settings to the ant command so things are
>>> faster.
>>>
>>> -Jeremiah
>>>
>>> On Wed, Apr 23, 2025 at 4:36 PM Jordan West <jo...@gmail.com> wrote:
>>>
>>> Should we consider making that the default and then passing false
>>> explicitly in CI/builds? I agree with Alex it’s a bit surprising and
>>> shorter build times when developing would be helpful.
>>>
>>> Jordan
>>>
>>> On Wed, Apr 23, 2025 at 13:37 Mick Semb Wever <mc...@apache.org> wrote:
>>>
>>> Python and Go are used by the gen-doc target.
>>>
>>> Code changes can break these, hence it is part of `ant check`.
>>> It is not called by `ant jar`
>>>
>>> If you want to run check but skip it, it's to add
>>> `-Dant.gen-doc.skip=true`
>>>
>>>
>>>
>>> On Wed, 23 Apr 2025 at 22:06, Alex Petrov <al...@coffeenco.de> wrote:
>>>
>>>
>>> Hi folks,
>>>
>>> Building Cassandra jar has been getting increasingly slow, and now it
>>> looks like we depend not only on python3 (which was already not optimal),
>>> but also on go:
>>>
>>>     ant -Dno-checkstyle=true
>>>
>>> ...
>>>
>>>      [exec] python3 ./scripts/gen-nodetool-docs.py
>>>      [exec] python3 ./scripts/convert_yaml_to_adoc.py
>>> ../conf/cassandra.yaml
>>> ./modules/cassandra/pages/managing/configuration/cass_yaml_file.adoc
>>>      [exec] ./scripts/process-native-protocol-specs-in-docker.sh
>>>      [exec] Go env not found in your system, proceeding with
>>> installation.
>>>      [exec] Downloading Go 1.23.1...
>>>      [exec] Installing Go 1.23.1...
>>>      [exec] Building the cqlprotodoc...
>>>      [exec] Cloning into 'cassandra-website'...
>>>      [exec] Your branch is up to date with 'origin/trunk'.
>>>      [exec] go: downloading github.com/mvdan/xurls v1.1.0
>>>      [exec] Processing the .spec files...
>>>
>>> I personally consider this extremely dangerous, but also unnecessary. My
>>> current stance is that functionality introducing python3 and go should be
>>> moved to a separate task that only runs on demand / ci / release.  I
>>> welcome convincing arguments that would suggest otherwise.
>>>
>>> If you agree we should not require python and go to run `ant
>>> -Dno-checkstyle=true`, please also write a short message, this will be very
>>> helpful as well.
>>>
>>> Thank you,
>>> --Alex
>>>
>>>
>>>
>>>
>>>
>>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Patrick McFadin <pm...@gmail.com>.
I don't think the website is generated outside of the Cassandra website
repo. Did I miss that it's built when building the server?

On Mon, Apr 28, 2025 at 10:19 AM Jon Haddad <jo...@rustyrazorblade.com> wrote:

> I strongly prefer we didn't install things on users systems they didn't
> ask for.  I think we're also wasting people's time by building the website
> by default.
>
> ant isn't a popular tool.  The more we ask people to know about it, the
> more we turn people away from the project.  I'm personally unwilling to
> invest the time in learning what's otherwise a dead technology, and I can't
> be the only one.  We should by default do the thing that most people want
> by default, which is building the server.
>
> Jon
>
> On Mon, Apr 28, 2025 at 9:54 AM Alex Petrov <al...@coffeenco.de> wrote:
>
>> I’ve talked to some folks privately and looks like ant jar is a more
>> common workflow than just ant. I also don’t mind disabling check style and
>> doc gen for my purposes. But I would suggest to not auto install, or hide
>> installation behind a flag and suggest user to opt in to this.
>>
>> One more reason to not install is that this way we commit to maintaining
>> installing go.
>>
>> Ideally we would not pull in one more language for a small task but it’s
>> hard for me to judge how difficult it would’ve been to implement this
>> feature differently.
>>
>> On Sun, Apr 27, 2025, at 6:20 PM, Mick Semb Wever wrote:
>>
>> Pulling out to a separate thread, as requested.
>>
>> Should golang auto-install when it's not found, when executing `ant
>> gen-doc` ?
>>
>> Our build system has a number of path dependencies: e.g. ant, maven,
>> python, golang, gradle, git, grep, curl, docker, make.  Gradle and golang
>> will auto-install if not found.
>>
>> Golang was recently introduced for CASSANDRA-19915, helping consolidate
>> and improve the website documentation pages for our native protocol
>> specification.  This was work that came over from the gocql community.
>>
>> https://cassandra.apache.org/doc/latest/cassandra/reference/native-protocol.html
>>
>> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc
>>
>> Both python and golang are dependencies for in-tree docs generation (`ant
>> gen-doc`).  Python is also used for tests.
>>
>> While python is expected to be found, golang like gradle will be
>> downloaded and installed if it is not.  But unlike gradle, golang is only
>> temporarily installed into your $TMPDIR and only used for the native
>> protocol doc page generation.
>>
>> https://github.com/apache/cassandra/blob/trunk/doc/scripts/process-native-protocol-specs-in-docker.sh
>>
>>
>> If you don't want to be executing gen-doc (note that `ant jar` doesn't
>> already), you can just add `-Dant.gen-doc.skip=true` to your ant
>> cmdline/config.  `ant check` does invoke it, because code changes can break
>> doc generation.
>>
>> Is this a problem ? Why ?  What can/should be done ?
>>
>>
>>
>>
>> On Fri, 25 Apr 2025 at 13:13, Alex Petrov <al...@coffeenco.de> wrote:
>>
>>
>> Thank you for input.
>>
>> I will disable gen-doc in addition to checkstyle locally, but I would
>> still suggest that in short-term we should at very least remove/disable the
>> part that auto-installs go (or at least have a discussion or vote about it,
>> in case folks believe auto-installing is a good idea), but ideally in the
>> long term we can find a solution that does not require go.
>>
>> On Thu, Apr 24, 2025, at 12:39 PM, Štefan Miklošovič wrote:
>>
>> I don't see it necessary to port Gradle stuff to Ant. I have noticed zero
>> problems with it after CEP-15 merge. Basically a non-event.
>>
>> It just looks strange that we have Ant + Maven poms / resolver wired into
>> it + submodule on Gradle. I mean ... wow. There is nothing wrong per se but
>> ... strange. One would expect that more into the future we go, the more
>> coherent / consistent everything related to the build system will be, not
>> the other way around. But I totally get that this is just how things are
>> for now and there is nobody to blame really.
>>
>> On Thu, Apr 24, 2025 at 10:39 AM Benedict <be...@apache.org> wrote:
>>
>>
>> We should separate out any grade discussion. I’m happy to migrate accord
>> to ant if that’s the project preference, but there’s continual discussion
>> to begin modularising Cassandra (at least a little), and a proposal to use
>> grade for the modules - which might be a happy medium for everyone’s
>> competing priorities.
>>
>>
>> On 24 Apr 2025, at 07:26, Mick Semb Wever <mc...@apache.org> wrote:
>>
>> 
>> Does this also apply to gradle, which now gets downloaded and installed,
>> and is the most recent addition ?
>>
>> The python requirement from gen-doc has been around for over three years
>> now.
>>
>> I agree with the rationale that `ant` should default to `ant check`,
>> keeping newcomers in mind while being more just a final pre-commit action
>> for seasoned devs where `ant jar` is the typical dev loop.  I'm sure this
>> has been discussed in the past.
>>
>>
>>
>> On Thu, 24 Apr 2025 at 08:16, Alex Petrov <al...@coffeenco.de> wrote:
>>
>>
>> I didn’t say we should disable checks. I thought this was a purpose of CI
>> : to check everything: there’s likely more to break semantically anyways,
>> also harder to detect and fix. I’m also not proposing removing something
>> that was long in place, these dependencies were introduced very recently.
>>
>> On Wed, Apr 23, 2025, at 11:51 PM, Jeremiah Jordan wrote:
>>
>> I think the default build should be to build and check everything.  I
>> think that if someone is new it is better to have everything built and
>> checked by default to flag issues.
>>
>> If someone knows what they are doing and wants to speed up the process it
>> is very easy to add the right settings to the ant command so things are
>> faster.
>>
>> -Jeremiah
>>
>> On Wed, Apr 23, 2025 at 4:36 PM Jordan West <jo...@gmail.com> wrote:
>>
>> Should we consider making that the default and then passing false
>> explicitly in CI/builds? I agree with Alex it’s a bit surprising and
>> shorter build times when developing would be helpful.
>>
>> Jordan
>>
>> On Wed, Apr 23, 2025 at 13:37 Mick Semb Wever <mc...@apache.org> wrote:
>>
>> Python and Go are used by the gen-doc target.
>>
>> Code changes can break these, hence it is part of `ant check`.
>> It is not called by `ant jar`
>>
>> If you want to run check but skip it, it's to add
>> `-Dant.gen-doc.skip=true`
>>
>>
>>
>> On Wed, 23 Apr 2025 at 22:06, Alex Petrov <al...@coffeenco.de> wrote:
>>
>>
>> Hi folks,
>>
>> Building Cassandra jar has been getting increasingly slow, and now it
>> looks like we depend not only on python3 (which was already not optimal),
>> but also on go:
>>
>>     ant -Dno-checkstyle=true
>>
>> ...
>>
>>      [exec] python3 ./scripts/gen-nodetool-docs.py
>>      [exec] python3 ./scripts/convert_yaml_to_adoc.py
>> ../conf/cassandra.yaml
>> ./modules/cassandra/pages/managing/configuration/cass_yaml_file.adoc
>>      [exec] ./scripts/process-native-protocol-specs-in-docker.sh
>>      [exec] Go env not found in your system, proceeding with installation.
>>      [exec] Downloading Go 1.23.1...
>>      [exec] Installing Go 1.23.1...
>>      [exec] Building the cqlprotodoc...
>>      [exec] Cloning into 'cassandra-website'...
>>      [exec] Your branch is up to date with 'origin/trunk'.
>>      [exec] go: downloading github.com/mvdan/xurls v1.1.0
>>      [exec] Processing the .spec files...
>>
>> I personally consider this extremely dangerous, but also unnecessary. My
>> current stance is that functionality introducing python3 and go should be
>> moved to a separate task that only runs on demand / ci / release.  I
>> welcome convincing arguments that would suggest otherwise.
>>
>> If you agree we should not require python and go to run `ant
>> -Dno-checkstyle=true`, please also write a short message, this will be very
>> helpful as well.
>>
>> Thank you,
>> --Alex
>>
>>
>>
>>
>>
>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Dmitry Konstantinov <ne...@gmail.com>.
+1 for the idea to re-implement this logic using python or some java
libraries/ant tasks. The fewer languages we have in our build system the
easier it is to support and manage at an infrastructure level.

On Tue, 29 Apr 2025 at 12:30, Maxim Muzafarov <mm...@apache.org> wrote:

> Hey,
>
> I've prepared a python script that generates the same docs (no go
> dependency). I use the jinja2 dependency, not sure if it's optimal
> because I had to google how to use it though (also not sure if it has
> to be run in docker).
> I haven't tested the generated files with the website, but I've
> compared the results with the same files in the trunk, and they look
> similar (almost).
>
>
> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
>
> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
> >
> > We should never download and install software via adhoc scripts without
> user consent. Was this ever discussed on this mailing list? If not, it’s a
> clear breach of policy (introducing a new dependency) and a severe one in
> my opinion, as it seems to introduce a new supply chain attack vector for
> all developers of Cassandra.
> >
> >
> >
> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
> >
> > 
> >
> >   .
> >
> >
> >>
> >> But that doesn’t seem to be the case here, the script checks for arm vs
> amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
> There is no verification of the download.  The only check is if curl
> returned non 0.
> >
> >
> >
> > Thanks for catching this, the sha256 check should always have been in
> place.  Adding this is just a one-liner, so that alone shouldn't force the
> decision.
> >
> >
> >
> >> It looks to be very simple to install  Go
> >
> >
> >
> > It takes a bit to ensure all build and CI systems are updated, and we
> never catch everything (esp what's downstream).
> >
> >
> > While it's "simple", multiplied by everyone (and every system) it adds
> up to be a significant time demand.
> >
> > Again, this too shouldn't be forcing the decision either way on what we
> want to do.
> >
> >
> >
>


-- 
Dmitry Konstantinov

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Ekaterina Dimitrova <e....@gmail.com>.
“  I can only hope they understand and appreciate this *is* their work
being carried forward, and our legitimate reasons for the golang to python
rewrite (i.e. there's nothing against golang in the community). “

+1

Also, I would love to explore Maxim’s suggestion more. Option 1 still leads
to people installing more things and it may have negative effect if people
start opting out of running those checks


Thanks for looking into this, Maxim
On Fri, 2 May 2025 at 7:24, Brandon Williams <dr...@gmail.com> wrote:

> When approaching the code for the first time, I think option 2 will be
> the most consistent and have less friction since it won't be a
> one-off.  +1 from me.
>
> Kind Regards,
> Brandon
>
>
> On Fri, May 2, 2025 at 5:47 AM Mick Semb Wever <mc...@apache.org> wrote:
> >
> >
> >
> > This can be done now using the following command (instead of `ant
> check`):
> >
> > .build/docker/check-code.sh
> >
> >
> > See readme under `.build/`
> >
> > But layering in the ability to do it with docker is opposed to requiring
> the use of docker.  i.e. I don't think we should make docker mandatory
> inside `ant check`.
> >
> >
> > There are two possible paths forward
> >  option 1) remove the auto-install, and just print the command the user
> needs to run to do the install, and/or how to skip that target.
> >  option 2) build on Maxim's contribution to rewrite it to
> python+virtualenv+jinja
> >
> >
> > With option (2) i feel it would be a shame that the original work in
> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from
> our newly joined gocql community would be redone.   I can only hope they
> understand and appreciate this *is* their work being carried forward, and
> our legitimate reasons for the golang to python rewrite (i.e. there's
> nothing against golang in the community).  (I've cc'd them.)
> >
> > Otherwise, I'm entirely on the fence – all ears.  Slightly leaning
> towards option (1).
> >
> >
> >
> > On Wed, 30 Apr 2025 at 16:20, Joseph Lynch <jo...@gmail.com>
> wrote:
> >>
> >> I also feel like this is what docker is for? It should be relatively
> straightforward to start from a golang Dockerfile, add the files you need,
> generate the docs and copy the result back out?
> >>
> >> -Joey
> >>
> >> On Wed, Apr 30, 2025 at 10:14 AM Chris Lohfink <cl...@gmail.com>
> wrote:
> >>>
> >>> Cassandra's rube goldberg build system is so incredibly painful to
> integrate inside corporate CI environments already... maybe docker
> containers so you dont actually install random tools on the host computer
> it might not have privileges to do?
> >>>
> >>> On Wed, Apr 30, 2025 at 6:13 AM Josh McKenzie <jm...@apache.org>
> wrote:
> >>>>
> >>>> So while it would be nice to keep things such that someone just runs
> ant and gets everything built, given this does not seem to be a standard
> method of dealing with a go install in build scripts, I would suggest we
> stop doing it.  It looks to be very simple to install  Go, so maybe switch
> to telling someone how to install it if it is not found, as well as giving
> them the setting to disable that artifact.
> >>>>
> >>>> +1 to Jeremiah's thoughts here.
> >>>>
> >>>> Passing thought - maybe introduce an "ant install-deps" target
> that'll install deps if not found?
> >>>>
> >>>> On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
> >>>>
> >>>> Hey,
> >>>>
> >>>> I've prepared a python script that generates the same docs (no go
> >>>> dependency). I use the jinja2 dependency, not sure if it's optimal
> >>>> because I had to google how to use it though (also not sure if it has
> >>>> to be run in docker).
> >>>> I haven't tested the generated files with the website, but I've
> >>>> compared the results with the same files in the trunk, and they look
> >>>> similar (almost).
> >>>>
> >>>>
> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
> >>>>
> >>>> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
> >>>> >
> >>>> > We should never download and install software via adhoc scripts
> without user consent. Was this ever discussed on this mailing list? If not,
> it’s a clear breach of policy (introducing a new dependency) and a severe
> one in my opinion, as it seems to introduce a new supply chain attack
> vector for all developers of Cassandra.
> >>>> >
> >>>> >
> >>>> >
> >>>> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
> >>>> >
> >>>> > 
> >>>> >
> >>>> >   .
> >>>> >
> >>>> >
> >>>> >>
> >>>> >> But that doesn’t seem to be the case here, the script checks for
> arm vs amd64, Linux vs Mac, and then fetches and untars the go distro into
> tmp. There is no verification of the download.  The only check is if curl
> returned non 0.
> >>>> >
> >>>> >
> >>>> >
> >>>> > Thanks for catching this, the sha256 check should always have been
> in place.  Adding this is just a one-liner, so that alone shouldn't force
> the decision.
> >>>> >
> >>>> >
> >>>> >
> >>>> >> It looks to be very simple to install  Go
> >>>> >
> >>>> >
> >>>> >
> >>>> > It takes a bit to ensure all build and CI systems are updated, and
> we never catch everything (esp what's downstream).
> >>>> >
> >>>> >
> >>>> > While it's "simple", multiplied by everyone (and every system) it
> adds up to be a significant time demand.
> >>>> >
> >>>> > Again, this too shouldn't be forcing the decision either way on
> what we want to do.
> >>>> >
> >>>> >
> >>>> >
> >>>>
> >>>>
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Brandon Williams <dr...@gmail.com>.
When approaching the code for the first time, I think option 2 will be
the most consistent and have less friction since it won't be a
one-off.  +1 from me.

Kind Regards,
Brandon


On Fri, May 2, 2025 at 5:47 AM Mick Semb Wever <mc...@apache.org> wrote:
>
>
>
> This can be done now using the following command (instead of `ant check`):
>
> .build/docker/check-code.sh
>
>
> See readme under `.build/`
>
> But layering in the ability to do it with docker is opposed to requiring the use of docker.  i.e. I don't think we should make docker mandatory inside `ant check`.
>
>
> There are two possible paths forward
>  option 1) remove the auto-install, and just print the command the user needs to run to do the install, and/or how to skip that target.
>  option 2) build on Maxim's contribution to rewrite it to python+virtualenv+jinja
>
>
> With option (2) i feel it would be a shame that the original work in https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from our newly joined gocql community would be redone.   I can only hope they understand and appreciate this *is* their work being carried forward, and our legitimate reasons for the golang to python rewrite (i.e. there's nothing against golang in the community).  (I've cc'd them.)
>
> Otherwise, I'm entirely on the fence – all ears.  Slightly leaning towards option (1).
>
>
>
> On Wed, 30 Apr 2025 at 16:20, Joseph Lynch <jo...@gmail.com> wrote:
>>
>> I also feel like this is what docker is for? It should be relatively straightforward to start from a golang Dockerfile, add the files you need, generate the docs and copy the result back out?
>>
>> -Joey
>>
>> On Wed, Apr 30, 2025 at 10:14 AM Chris Lohfink <cl...@gmail.com> wrote:
>>>
>>> Cassandra's rube goldberg build system is so incredibly painful to integrate inside corporate CI environments already... maybe docker containers so you dont actually install random tools on the host computer it might not have privileges to do?
>>>
>>> On Wed, Apr 30, 2025 at 6:13 AM Josh McKenzie <jm...@apache.org> wrote:
>>>>
>>>> So while it would be nice to keep things such that someone just runs ant and gets everything built, given this does not seem to be a standard method of dealing with a go install in build scripts, I would suggest we stop doing it.  It looks to be very simple to install  Go, so maybe switch to telling someone how to install it if it is not found, as well as giving them the setting to disable that artifact.
>>>>
>>>> +1 to Jeremiah's thoughts here.
>>>>
>>>> Passing thought - maybe introduce an "ant install-deps" target that'll install deps if not found?
>>>>
>>>> On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
>>>>
>>>> Hey,
>>>>
>>>> I've prepared a python script that generates the same docs (no go
>>>> dependency). I use the jinja2 dependency, not sure if it's optimal
>>>> because I had to google how to use it though (also not sure if it has
>>>> to be run in docker).
>>>> I haven't tested the generated files with the website, but I've
>>>> compared the results with the same files in the trunk, and they look
>>>> similar (almost).
>>>>
>>>> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
>>>>
>>>> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
>>>> >
>>>> > We should never download and install software via adhoc scripts without user consent. Was this ever discussed on this mailing list? If not, it’s a clear breach of policy (introducing a new dependency) and a severe one in my opinion, as it seems to introduce a new supply chain attack vector for all developers of Cassandra.
>>>> >
>>>> >
>>>> >
>>>> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
>>>> >
>>>> > 
>>>> >
>>>> >   .
>>>> >
>>>> >
>>>> >>
>>>> >> But that doesn’t seem to be the case here, the script checks for arm vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp. There is no verification of the download.  The only check is if curl returned non 0.
>>>> >
>>>> >
>>>> >
>>>> > Thanks for catching this, the sha256 check should always have been in place.  Adding this is just a one-liner, so that alone shouldn't force the decision.
>>>> >
>>>> >
>>>> >
>>>> >> It looks to be very simple to install  Go
>>>> >
>>>> >
>>>> >
>>>> > It takes a bit to ensure all build and CI systems are updated, and we never catch everything (esp what's downstream).
>>>> >
>>>> >
>>>> > While it's "simple", multiplied by everyone (and every system) it adds up to be a significant time demand.
>>>> >
>>>> > Again, this too shouldn't be forcing the decision either way on what we want to do.
>>>> >
>>>> >
>>>> >
>>>>
>>>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Maxim Muzafarov <mm...@apache.org>.
Hello everyone,

https://issues.apache.org/jira/browse/CASSANDRA-21432
We've removed the golang dependency from the working tree. Please let
us know if you encounter any issues and I will try to resolve them
asap.

On Sun, 25 May 2025 at 23:37, Maxim Muzafarov <mm...@apache.org> wrote:
>
> Created an issue:
> https://issues.apache.org/jira/browse/CASSANDRA-20679
>
> On Sun, 25 May 2025 at 12:57, Mick Semb Wever <mc...@apache.org> wrote:
> >
> >
> > Coming back to this.  There is no clear path forward looking at folks' preferences.
> >
> > The docker approach sounds beneficial, but comes with a few challenges that haven't been addressed.  We don't have docker at the ant build.xml or .build/*.sh level intentionally, with docker layered on at the .build/docker/*.sh level.  We would need docker be invoked inside another docker command, and for jenkins this means three layers of docker-in-docker…  and I'm not comfortable with our code check `ant check`, that we especially want to apply to newcomers, to become by default optional (just because docker isn't installed/running).  Also no one has stepped forward to offer doing the work for a docker approach, so i think until it's thrashed out a bit more and with some sense of ownership we should exclude it as an option.
> >
> > My suggestion is the quick fix in option 1, while working on option 2.  Maxim, do you want to create a jira ticket for option 2 and we can continue that work there.   Pending objections we can start on this.
> >
> >
> >
> > On Fri, 2 May 2025 at 20:01, Jordan West <jw...@apache.org> wrote:
> >>
> >> +1 to docker. It’s a relatively universal dependency at this point, allows for more repeatable builds, and doesn’t require installing software in the build. It’s not any more heavyweight that go or python and has less environment nuances. It lets us pick what python (or go) we use among many other benefits.
> >>
> >> +1 to skipping parts of the build w an info/warning if docker is missing. We should NEVER be installing software without promoting.
> >>
> >> +1 to unifying generation on python. Absolutely appreciate the work done by the go folks. Separate from that keeping the build simple to reason about is important.
> >>
> >> Jordan
> >>
> >> On Fri, May 2, 2025 at 05:54 Joseph Lynch <jo...@gmail.com> wrote:
> >>>
> >>> On Fri, May 2, 2025 at 6:46 AM Mick Semb Wever <mc...@apache.org> wrote:
> >>>>
> >>>> But layering in the ability to do it with docker is opposed to requiring the use of docker.  i.e. I don't think we should make docker mandatory inside `ant check`.
> >>>
> >>>
> >>> I think if you don't have docker it could be reasonable not to build the docs as part of ant check (the golang part of gen-doc could be skipped). Docker is fairly standard build tooling these days since it obviates the need to have the right versions of things on peoples systems (e.g. golang or python).
> >>>
> >>>>
> >>>> There are two possible paths forward
> >>>>  option 1) remove the auto-install, and just print the command the user needs to run to do the install, and/or how to skip that target.
> >>>>
> >>>>  option 2) build on Maxim's contribution to rewrite it to python+virtualenv+jinja
> >>>
> >>>
> >>> I don't object to #2, but once you start talking about virtualenv and python libraries you may want to be building that via docker as well as libraries often have minimum requirements of python versions.
> >>>
> >>>>
> >>>> With option (2) i feel it would be a shame that the original work in https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from our newly joined gocql community would be redone.   I can only hope they understand and appreciate this *is* their work being carried forward, and our legitimate reasons for the golang to python rewrite (i.e. there's nothing against golang in the community).  (I've cc'd them.)
> >>>>
> >>>> Otherwise, I'm entirely on the fence – all ears.  Slightly leaning towards option (1).
> >>>
> >>>
> >>> Is there an option 3? gen-doc (specifically the part that uses golang) requires docker, and if you don't have docker that component of the docs is skipped? That seems to move towards a possible world where we could move towards just one build requirement (docker) instead of 10? If you feel strongly we should always try first with native and only ask people to use docker if their system isn't running the right versions then 1 seems better to me given we have existing functional code and the official builds would have golang available.
> >>>
> >>> -Joey

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Maxim Muzafarov <mm...@apache.org>.
Created an issue:
https://issues.apache.org/jira/browse/CASSANDRA-20679

On Sun, 25 May 2025 at 12:57, Mick Semb Wever <mc...@apache.org> wrote:
>
>
> Coming back to this.  There is no clear path forward looking at folks' preferences.
>
> The docker approach sounds beneficial, but comes with a few challenges that haven't been addressed.  We don't have docker at the ant build.xml or .build/*.sh level intentionally, with docker layered on at the .build/docker/*.sh level.  We would need docker be invoked inside another docker command, and for jenkins this means three layers of docker-in-docker…  and I'm not comfortable with our code check `ant check`, that we especially want to apply to newcomers, to become by default optional (just because docker isn't installed/running).  Also no one has stepped forward to offer doing the work for a docker approach, so i think until it's thrashed out a bit more and with some sense of ownership we should exclude it as an option.
>
> My suggestion is the quick fix in option 1, while working on option 2.  Maxim, do you want to create a jira ticket for option 2 and we can continue that work there.   Pending objections we can start on this.
>
>
>
> On Fri, 2 May 2025 at 20:01, Jordan West <jw...@apache.org> wrote:
>>
>> +1 to docker. It’s a relatively universal dependency at this point, allows for more repeatable builds, and doesn’t require installing software in the build. It’s not any more heavyweight that go or python and has less environment nuances. It lets us pick what python (or go) we use among many other benefits.
>>
>> +1 to skipping parts of the build w an info/warning if docker is missing. We should NEVER be installing software without promoting.
>>
>> +1 to unifying generation on python. Absolutely appreciate the work done by the go folks. Separate from that keeping the build simple to reason about is important.
>>
>> Jordan
>>
>> On Fri, May 2, 2025 at 05:54 Joseph Lynch <jo...@gmail.com> wrote:
>>>
>>> On Fri, May 2, 2025 at 6:46 AM Mick Semb Wever <mc...@apache.org> wrote:
>>>>
>>>> But layering in the ability to do it with docker is opposed to requiring the use of docker.  i.e. I don't think we should make docker mandatory inside `ant check`.
>>>
>>>
>>> I think if you don't have docker it could be reasonable not to build the docs as part of ant check (the golang part of gen-doc could be skipped). Docker is fairly standard build tooling these days since it obviates the need to have the right versions of things on peoples systems (e.g. golang or python).
>>>
>>>>
>>>> There are two possible paths forward
>>>>  option 1) remove the auto-install, and just print the command the user needs to run to do the install, and/or how to skip that target.
>>>>
>>>>  option 2) build on Maxim's contribution to rewrite it to python+virtualenv+jinja
>>>
>>>
>>> I don't object to #2, but once you start talking about virtualenv and python libraries you may want to be building that via docker as well as libraries often have minimum requirements of python versions.
>>>
>>>>
>>>> With option (2) i feel it would be a shame that the original work in https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from our newly joined gocql community would be redone.   I can only hope they understand and appreciate this *is* their work being carried forward, and our legitimate reasons for the golang to python rewrite (i.e. there's nothing against golang in the community).  (I've cc'd them.)
>>>>
>>>> Otherwise, I'm entirely on the fence – all ears.  Slightly leaning towards option (1).
>>>
>>>
>>> Is there an option 3? gen-doc (specifically the part that uses golang) requires docker, and if you don't have docker that component of the docs is skipped? That seems to move towards a possible world where we could move towards just one build requirement (docker) instead of 10? If you feel strongly we should always try first with native and only ask people to use docker if their system isn't running the right versions then 1 seems better to me given we have existing functional code and the official builds would have golang available.
>>>
>>> -Joey

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Mick Semb Wever <mc...@apache.org>.
Coming back to this.  There is no clear path forward looking at folks'
preferences.

The docker approach sounds beneficial, but comes with a few challenges that
haven't been addressed.  We don't have docker at the ant build.xml or
.build/*.sh level intentionally, with docker layered on at the
.build/docker/*.sh level.  We would need docker be invoked inside another
docker command, and for jenkins this means three layers of
docker-in-docker…  and I'm not comfortable with our code check `ant check`,
that we especially want to apply to newcomers, to become by default
optional (just because docker isn't installed/running).  Also no one has
stepped forward to offer doing the work for a docker approach, so i think
until it's thrashed out a bit more and with some sense of ownership we
should exclude it as an option.

My suggestion is the quick fix in option 1, while working on option 2.
Maxim, do you want to create a jira ticket for option 2 and we can continue
that work there.   Pending objections we can start on this.



On Fri, 2 May 2025 at 20:01, Jordan West <jw...@apache.org> wrote:

> +1 to docker. It’s a relatively universal dependency at this point, allows
> for more repeatable builds, and doesn’t require installing software in the
> build. It’s not any more heavyweight that go or python and has less
> environment nuances. It lets us pick what python (or go) we use among many
> other benefits.
>
> +1 to skipping parts of the build w an info/warning if docker is missing.
> We should NEVER be installing software without promoting.
>
> +1 to unifying generation on python. Absolutely appreciate the work done
> by the go folks. Separate from that keeping the build simple to reason
> about is important.
>
> Jordan
>
> On Fri, May 2, 2025 at 05:54 Joseph Lynch <jo...@gmail.com> wrote:
>
>> On Fri, May 2, 2025 at 6:46 AM Mick Semb Wever <mc...@apache.org> wrote:
>>
>>> But layering in the ability to do it with docker is opposed to requiring
>>> the use of docker.  i.e. I don't think we should make docker mandatory
>>> inside `ant check`.
>>>
>>
>> I think if you don't have docker it could be reasonable not to build the
>> docs as part of ant check (the golang part of gen-doc could be skipped).
>> Docker is fairly standard build tooling these days since it obviates the
>> need to have the right versions of things on peoples systems (e.g. golang
>> or python).
>>
>>
>>> There are two possible paths forward
>>>  option 1) remove the auto-install, and just print the command the user
>>> needs to run to do the install, and/or how to skip that target.
>>>
>>  option 2) build on Maxim's contribution to rewrite it to
>>> python+virtualenv+jinja
>>>
>>
>> I don't object to #2, but once you start talking about virtualenv and
>> python libraries you may want to be building that via docker as well as
>> libraries often have minimum requirements of python versions.
>>
>>
>>> With option (2) i feel it would be a shame that the original work in
>>> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from
>>> our newly joined gocql community would be redone.   I can only hope they
>>> understand and appreciate this *is* their work being carried forward, and
>>> our legitimate reasons for the golang to python rewrite (i.e. there's
>>> nothing against golang in the community).  (I've cc'd them.)
>>>
>>> Otherwise, I'm entirely on the fence – all ears.  Slightly leaning
>>> towards option (1).
>>>
>>
>> Is there an option 3? gen-doc (specifically the part that uses golang)
>> requires docker, and if you don't have docker that component of the docs is
>> skipped? That seems to move towards a possible world where we could move
>> towards just one build requirement (docker) instead of 10? If you feel
>> strongly we should always try first with native and only ask people to use
>> docker if their system isn't running the right versions then 1 seems better
>> to me given we have existing functional code and the official builds would
>> have golang available.
>>
>> -Joey
>>
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Jordan West <jw...@apache.org>.
+1 to docker. It’s a relatively universal dependency at this point, allows
for more repeatable builds, and doesn’t require installing software in the
build. It’s not any more heavyweight that go or python and has less
environment nuances. It lets us pick what python (or go) we use among many
other benefits.

+1 to skipping parts of the build w an info/warning if docker is missing.
We should NEVER be installing software without promoting.

+1 to unifying generation on python. Absolutely appreciate the work done by
the go folks. Separate from that keeping the build simple to reason about
is important.

Jordan

On Fri, May 2, 2025 at 05:54 Joseph Lynch <jo...@gmail.com> wrote:

> On Fri, May 2, 2025 at 6:46 AM Mick Semb Wever <mc...@apache.org> wrote:
>
>> But layering in the ability to do it with docker is opposed to requiring
>> the use of docker.  i.e. I don't think we should make docker mandatory
>> inside `ant check`.
>>
>
> I think if you don't have docker it could be reasonable not to build the
> docs as part of ant check (the golang part of gen-doc could be skipped).
> Docker is fairly standard build tooling these days since it obviates the
> need to have the right versions of things on peoples systems (e.g. golang
> or python).
>
>
>> There are two possible paths forward
>>  option 1) remove the auto-install, and just print the command the user
>> needs to run to do the install, and/or how to skip that target.
>>
>  option 2) build on Maxim's contribution to rewrite it to
>> python+virtualenv+jinja
>>
>
> I don't object to #2, but once you start talking about virtualenv and
> python libraries you may want to be building that via docker as well as
> libraries often have minimum requirements of python versions.
>
>
>> With option (2) i feel it would be a shame that the original work in
>> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from
>> our newly joined gocql community would be redone.   I can only hope they
>> understand and appreciate this *is* their work being carried forward, and
>> our legitimate reasons for the golang to python rewrite (i.e. there's
>> nothing against golang in the community).  (I've cc'd them.)
>>
>> Otherwise, I'm entirely on the fence – all ears.  Slightly leaning
>> towards option (1).
>>
>
> Is there an option 3? gen-doc (specifically the part that uses golang)
> requires docker, and if you don't have docker that component of the docs is
> skipped? That seems to move towards a possible world where we could move
> towards just one build requirement (docker) instead of 10? If you feel
> strongly we should always try first with native and only ask people to use
> docker if their system isn't running the right versions then 1 seems better
> to me given we have existing functional code and the official builds would
> have golang available.
>
> -Joey
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Joseph Lynch <jo...@gmail.com>.
On Fri, May 2, 2025 at 6:46 AM Mick Semb Wever <mc...@apache.org> wrote:

> But layering in the ability to do it with docker is opposed to requiring
> the use of docker.  i.e. I don't think we should make docker mandatory
> inside `ant check`.
>

I think if you don't have docker it could be reasonable not to build the
docs as part of ant check (the golang part of gen-doc could be skipped).
Docker is fairly standard build tooling these days since it obviates the
need to have the right versions of things on peoples systems (e.g. golang
or python).


> There are two possible paths forward
>  option 1) remove the auto-install, and just print the command the user
> needs to run to do the install, and/or how to skip that target.
>
 option 2) build on Maxim's contribution to rewrite it to
> python+virtualenv+jinja
>

I don't object to #2, but once you start talking about virtualenv and
python libraries you may want to be building that via docker as well as
libraries often have minimum requirements of python versions.


> With option (2) i feel it would be a shame that the original work in
> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from
> our newly joined gocql community would be redone.   I can only hope they
> understand and appreciate this *is* their work being carried forward, and
> our legitimate reasons for the golang to python rewrite (i.e. there's
> nothing against golang in the community).  (I've cc'd them.)
>
> Otherwise, I'm entirely on the fence – all ears.  Slightly leaning towards
> option (1).
>

Is there an option 3? gen-doc (specifically the part that uses golang)
requires docker, and if you don't have docker that component of the docs is
skipped? That seems to move towards a possible world where we could move
towards just one build requirement (docker) instead of 10? If you feel
strongly we should always try first with native and only ask people to use
docker if their system isn't running the right versions then 1 seems better
to me given we have existing functional code and the official builds would
have golang available.

-Joey

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Mick Semb Wever <mc...@apache.org>.
This can be done now using the following command (instead of `ant check`):

.build/docker/check-code.sh


See readme under `.build/`

But layering in the ability to do it with docker is opposed to requiring
the use of docker.  i.e. I don't think we should make docker mandatory
inside `ant check`.


There are two possible paths forward
 option 1) remove the auto-install, and just print the command the user
needs to run to do the install, and/or how to skip that target.
 option 2) build on Maxim's contribution to rewrite it to
python+virtualenv+jinja


With option (2) i feel it would be a shame that the original work in
https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc from our
newly joined gocql community would be redone.   I can only hope they
understand and appreciate this *is* their work being carried forward, and
our legitimate reasons for the golang to python rewrite (i.e. there's
nothing against golang in the community).  (I've cc'd them.)

Otherwise, I'm entirely on the fence – all ears.  Slightly leaning towards
option (1).



On Wed, 30 Apr 2025 at 16:20, Joseph Lynch <jo...@gmail.com> wrote:

> I also feel like this is what docker is for? It should be relatively
> straightforward to start from a golang Dockerfile, add the files you need,
> generate the docs and copy the result back out?
>
> -Joey
>
> On Wed, Apr 30, 2025 at 10:14 AM Chris Lohfink <cl...@gmail.com>
> wrote:
>
>> Cassandra's rube goldberg build system is so incredibly painful to
>> integrate inside corporate CI environments already... maybe docker
>> containers so you dont actually install random tools on the host computer
>> it might not have privileges to do?
>>
>> On Wed, Apr 30, 2025 at 6:13 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>> So while it would be nice to keep things such that someone just runs ant
>>> and gets everything built, given this does not seem to be a standard method
>>> of dealing with a go install in build scripts, I would suggest we stop
>>> doing it.  It looks to be very simple to install  Go, so maybe switch to
>>> telling someone how to install it if it is not found, as well as giving
>>> them the setting to disable that artifact.
>>>
>>> +1 to Jeremiah's thoughts here.
>>>
>>> Passing thought - maybe introduce an "ant install-deps" target that'll
>>> install deps if not found?
>>>
>>> On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
>>>
>>> Hey,
>>>
>>> I've prepared a python script that generates the same docs (no go
>>> dependency). I use the jinja2 dependency, not sure if it's optimal
>>> because I had to google how to use it though (also not sure if it has
>>> to be run in docker).
>>> I haven't tested the generated files with the website, but I've
>>> compared the results with the same files in the trunk, and they look
>>> similar (almost).
>>>
>>>
>>> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
>>>
>>> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
>>> >
>>> > We should never download and install software via adhoc scripts
>>> without user consent. Was this ever discussed on this mailing list? If not,
>>> it’s a clear breach of policy (introducing a new dependency) and a severe
>>> one in my opinion, as it seems to introduce a new supply chain attack
>>> vector for all developers of Cassandra.
>>> >
>>> >
>>> >
>>> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
>>> >
>>> > 
>>> >
>>> >   .
>>> >
>>> >
>>> >>
>>> >> But that doesn’t seem to be the case here, the script checks for arm
>>> vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
>>> There is no verification of the download.  The only check is if curl
>>> returned non 0.
>>> >
>>> >
>>> >
>>> > Thanks for catching this, the sha256 check should always have been in
>>> place.  Adding this is just a one-liner, so that alone shouldn't force the
>>> decision.
>>> >
>>> >
>>> >
>>> >> It looks to be very simple to install  Go
>>> >
>>> >
>>> >
>>> > It takes a bit to ensure all build and CI systems are updated, and we
>>> never catch everything (esp what's downstream).
>>> >
>>> >
>>> > While it's "simple", multiplied by everyone (and every system) it adds
>>> up to be a significant time demand.
>>> >
>>> > Again, this too shouldn't be forcing the decision either way on what
>>> we want to do.
>>> >
>>> >
>>> >
>>>
>>>
>>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Patrick McFadin <pm...@gmail.com>.
That's how the website build works. It's kinda funny watching Docker build
fire up when you need to publish a web page, but it separates things
nicely.

I don't think it's a huge declaration, but there is definitely a case for
doing a "year of efficiency" type effort with the builds. It's got a lot of
tech debt built up as witnessed by "ant clean" "ant realclean" and "ant
nuke_it_from_space." Not looking forward to the religious wars this will
spark, but needs to be done. I would be happy to organize some scoping
documentation in Confluent to start.

Patrick

On Wed, Apr 30, 2025 at 7:20 AM Joseph Lynch <jo...@gmail.com> wrote:

> I also feel like this is what docker is for? It should be relatively
> straightforward to start from a golang Dockerfile, add the files you need,
> generate the docs and copy the result back out?
>
> -Joey
>
> On Wed, Apr 30, 2025 at 10:14 AM Chris Lohfink <cl...@gmail.com>
> wrote:
>
>> Cassandra's rube goldberg build system is so incredibly painful to
>> integrate inside corporate CI environments already... maybe docker
>> containers so you dont actually install random tools on the host computer
>> it might not have privileges to do?
>>
>> On Wed, Apr 30, 2025 at 6:13 AM Josh McKenzie <jm...@apache.org>
>> wrote:
>>
>>> So while it would be nice to keep things such that someone just runs ant
>>> and gets everything built, given this does not seem to be a standard method
>>> of dealing with a go install in build scripts, I would suggest we stop
>>> doing it.  It looks to be very simple to install  Go, so maybe switch to
>>> telling someone how to install it if it is not found, as well as giving
>>> them the setting to disable that artifact.
>>>
>>> +1 to Jeremiah's thoughts here.
>>>
>>> Passing thought - maybe introduce an "ant install-deps" target that'll
>>> install deps if not found?
>>>
>>> On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
>>>
>>> Hey,
>>>
>>> I've prepared a python script that generates the same docs (no go
>>> dependency). I use the jinja2 dependency, not sure if it's optimal
>>> because I had to google how to use it though (also not sure if it has
>>> to be run in docker).
>>> I haven't tested the generated files with the website, but I've
>>> compared the results with the same files in the trunk, and they look
>>> similar (almost).
>>>
>>>
>>> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
>>>
>>> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
>>> >
>>> > We should never download and install software via adhoc scripts
>>> without user consent. Was this ever discussed on this mailing list? If not,
>>> it’s a clear breach of policy (introducing a new dependency) and a severe
>>> one in my opinion, as it seems to introduce a new supply chain attack
>>> vector for all developers of Cassandra.
>>> >
>>> >
>>> >
>>> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
>>> >
>>> > 
>>> >
>>> >   .
>>> >
>>> >
>>> >>
>>> >> But that doesn’t seem to be the case here, the script checks for arm
>>> vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
>>> There is no verification of the download.  The only check is if curl
>>> returned non 0.
>>> >
>>> >
>>> >
>>> > Thanks for catching this, the sha256 check should always have been in
>>> place.  Adding this is just a one-liner, so that alone shouldn't force the
>>> decision.
>>> >
>>> >
>>> >
>>> >> It looks to be very simple to install  Go
>>> >
>>> >
>>> >
>>> > It takes a bit to ensure all build and CI systems are updated, and we
>>> never catch everything (esp what's downstream).
>>> >
>>> >
>>> > While it's "simple", multiplied by everyone (and every system) it adds
>>> up to be a significant time demand.
>>> >
>>> > Again, this too shouldn't be forcing the decision either way on what
>>> we want to do.
>>> >
>>> >
>>> >
>>>
>>>
>>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Joseph Lynch <jo...@gmail.com>.
I also feel like this is what docker is for? It should be relatively
straightforward to start from a golang Dockerfile, add the files you need,
generate the docs and copy the result back out?

-Joey

On Wed, Apr 30, 2025 at 10:14 AM Chris Lohfink <cl...@gmail.com> wrote:

> Cassandra's rube goldberg build system is so incredibly painful to
> integrate inside corporate CI environments already... maybe docker
> containers so you dont actually install random tools on the host computer
> it might not have privileges to do?
>
> On Wed, Apr 30, 2025 at 6:13 AM Josh McKenzie <jm...@apache.org>
> wrote:
>
>> So while it would be nice to keep things such that someone just runs ant
>> and gets everything built, given this does not seem to be a standard method
>> of dealing with a go install in build scripts, I would suggest we stop
>> doing it.  It looks to be very simple to install  Go, so maybe switch to
>> telling someone how to install it if it is not found, as well as giving
>> them the setting to disable that artifact.
>>
>> +1 to Jeremiah's thoughts here.
>>
>> Passing thought - maybe introduce an "ant install-deps" target that'll
>> install deps if not found?
>>
>> On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
>>
>> Hey,
>>
>> I've prepared a python script that generates the same docs (no go
>> dependency). I use the jinja2 dependency, not sure if it's optimal
>> because I had to google how to use it though (also not sure if it has
>> to be run in docker).
>> I haven't tested the generated files with the website, but I've
>> compared the results with the same files in the trunk, and they look
>> similar (almost).
>>
>>
>> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
>>
>> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
>> >
>> > We should never download and install software via adhoc scripts without
>> user consent. Was this ever discussed on this mailing list? If not, it’s a
>> clear breach of policy (introducing a new dependency) and a severe one in
>> my opinion, as it seems to introduce a new supply chain attack vector for
>> all developers of Cassandra.
>> >
>> >
>> >
>> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
>> >
>> > 
>> >
>> >   .
>> >
>> >
>> >>
>> >> But that doesn’t seem to be the case here, the script checks for arm
>> vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
>> There is no verification of the download.  The only check is if curl
>> returned non 0.
>> >
>> >
>> >
>> > Thanks for catching this, the sha256 check should always have been in
>> place.  Adding this is just a one-liner, so that alone shouldn't force the
>> decision.
>> >
>> >
>> >
>> >> It looks to be very simple to install  Go
>> >
>> >
>> >
>> > It takes a bit to ensure all build and CI systems are updated, and we
>> never catch everything (esp what's downstream).
>> >
>> >
>> > While it's "simple", multiplied by everyone (and every system) it adds
>> up to be a significant time demand.
>> >
>> > Again, this too shouldn't be forcing the decision either way on what we
>> want to do.
>> >
>> >
>> >
>>
>>
>>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Chris Lohfink <cl...@gmail.com>.
Cassandra's rube goldberg build system is so incredibly painful to
integrate inside corporate CI environments already... maybe docker
containers so you dont actually install random tools on the host computer
it might not have privileges to do?

On Wed, Apr 30, 2025 at 6:13 AM Josh McKenzie <jm...@apache.org> wrote:

> So while it would be nice to keep things such that someone just runs ant
> and gets everything built, given this does not seem to be a standard method
> of dealing with a go install in build scripts, I would suggest we stop
> doing it.  It looks to be very simple to install  Go, so maybe switch to
> telling someone how to install it if it is not found, as well as giving
> them the setting to disable that artifact.
>
> +1 to Jeremiah's thoughts here.
>
> Passing thought - maybe introduce an "ant install-deps" target that'll
> install deps if not found?
>
> On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
>
> Hey,
>
> I've prepared a python script that generates the same docs (no go
> dependency). I use the jinja2 dependency, not sure if it's optimal
> because I had to google how to use it though (also not sure if it has
> to be run in docker).
> I haven't tested the generated files with the website, but I've
> compared the results with the same files in the trunk, and they look
> similar (almost).
>
>
> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
>
> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
> >
> > We should never download and install software via adhoc scripts without
> user consent. Was this ever discussed on this mailing list? If not, it’s a
> clear breach of policy (introducing a new dependency) and a severe one in
> my opinion, as it seems to introduce a new supply chain attack vector for
> all developers of Cassandra.
> >
> >
> >
> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
> >
> > 
> >
> >   .
> >
> >
> >>
> >> But that doesn’t seem to be the case here, the script checks for arm vs
> amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
> There is no verification of the download.  The only check is if curl
> returned non 0.
> >
> >
> >
> > Thanks for catching this, the sha256 check should always have been in
> place.  Adding this is just a one-liner, so that alone shouldn't force the
> decision.
> >
> >
> >
> >> It looks to be very simple to install  Go
> >
> >
> >
> > It takes a bit to ensure all build and CI systems are updated, and we
> never catch everything (esp what's downstream).
> >
> >
> > While it's "simple", multiplied by everyone (and every system) it adds
> up to be a significant time demand.
> >
> > Again, this too shouldn't be forcing the decision either way on what we
> want to do.
> >
> >
> >
>
>
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Josh McKenzie <jm...@apache.org>.
> So while it would be nice to keep things such that someone just runs ant and gets everything built, given this does not seem to be a standard method of dealing with a go install in build scripts, I would suggest we stop doing it.  It looks to be very simple to install  Go, so maybe switch to telling someone how to install it if it is not found, as well as giving them the setting to disable that artifact.
+1 to Jeremiah's thoughts here.

Passing thought - maybe introduce an "ant install-deps" target that'll install deps if not found?

On Tue, Apr 29, 2025, at 7:30 AM, Maxim Muzafarov wrote:
> Hey,
> 
> I've prepared a python script that generates the same docs (no go
> dependency). I use the jinja2 dependency, not sure if it's optimal
> because I had to google how to use it though (also not sure if it has
> to be run in docker).
> I haven't tested the generated files with the website, but I've
> compared the results with the same files in the trunk, and they look
> similar (almost).
> 
> https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python
> 
> On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
> >
> > We should never download and install software via adhoc scripts without user consent. Was this ever discussed on this mailing list? If not, it’s a clear breach of policy (introducing a new dependency) and a severe one in my opinion, as it seems to introduce a new supply chain attack vector for all developers of Cassandra.
> >
> >
> >
> > On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
> >
> > 
> >
> >   .
> >
> >
> >>
> >> But that doesn’t seem to be the case here, the script checks for arm vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp. There is no verification of the download.  The only check is if curl returned non 0.
> >
> >
> >
> > Thanks for catching this, the sha256 check should always have been in place.  Adding this is just a one-liner, so that alone shouldn't force the decision.
> >
> >
> >
> >> It looks to be very simple to install  Go
> >
> >
> >
> > It takes a bit to ensure all build and CI systems are updated, and we never catch everything (esp what's downstream).
> >
> >
> > While it's "simple", multiplied by everyone (and every system) it adds up to be a significant time demand.
> >
> > Again, this too shouldn't be forcing the decision either way on what we want to do.
> >
> >
> >
> 

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Maxim Muzafarov <mm...@apache.org>.
Hey,

I've prepared a python script that generates the same docs (no go
dependency). I use the jinja2 dependency, not sure if it's optimal
because I had to google how to use it though (also not sure if it has
to be run in docker).
I haven't tested the generated files with the website, but I've
compared the results with the same files in the trunk, and they look
similar (almost).

https://github.com/apache/cassandra/compare/trunk...Mmuzaf:cassandra:generate-cqlprotodocs-python

On Tue, 29 Apr 2025 at 10:10, Benedict <be...@apache.org> wrote:
>
> We should never download and install software via adhoc scripts without user consent. Was this ever discussed on this mailing list? If not, it’s a clear breach of policy (introducing a new dependency) and a severe one in my opinion, as it seems to introduce a new supply chain attack vector for all developers of Cassandra.
>
>
>
> On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
>
> 
>
>   .
>
>
>>
>> But that doesn’t seem to be the case here, the script checks for arm vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp. There is no verification of the download.  The only check is if curl returned non 0.
>
>
>
> Thanks for catching this, the sha256 check should always have been in place.  Adding this is just a one-liner, so that alone shouldn't force the decision.
>
>
>
>> It looks to be very simple to install  Go
>
>
>
> It takes a bit to ensure all build and CI systems are updated, and we never catch everything (esp what's downstream).
>
>
> While it's "simple", multiplied by everyone (and every system) it adds up to be a significant time demand.
>
> Again, this too shouldn't be forcing the decision either way on what we want to do.
>
>
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Benedict <be...@apache.org>.
We should never download and install software via adhoc scripts without user consent. Was this ever discussed on this mailing list? If not, it’s a clear breach of policy (introducing a new dependency) and a severe one in my opinion, as it seems to introduce a new supply chain attack vector for all developers of Cassandra.



> On 29 Apr 2025, at 08:17, Mick Semb Wever <mc...@apache.org> wrote:
> 
> 
>   .
>   
>  
>> But that doesn’t seem to be the case here, the script checks for arm vs amd64, Linux vs Mac, and then fetches and untars the go distro into tmp. There is no verification of the download.  The only check is if curl returned non 0.
> 
> 
> 
> Thanks for catching this, the sha256 check should always have been in place.  Adding this is just a one-liner, so that alone shouldn't force the decision.
> 
> 
>> It looks to be very simple to install  Go
> 
> 
> It takes a bit to ensure all build and CI systems are updated, and we never catch everything (esp what's downstream).
> 
> While it's "simple", multiplied by everyone (and every system) it adds up to be a significant time demand.
> Again, this too shouldn't be forcing the decision either way on what we want to do.

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Mick Semb Wever <mc...@apache.org>.
  .



> But that doesn’t seem to be the case here, the script checks for arm vs
> amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
> There is no verification of the download.  The only check is if curl
> returned non 0.
>


Thanks for catching this, the sha256 check should always have been in
place.  Adding this is just a one-liner, so that alone shouldn't force the
decision.



It looks to be very simple to install  Go



It takes a bit to ensure all build and CI systems are updated, and we never
catch everything (esp what's downstream).


While it's "simple", multiplied by everyone (and every system) it adds up
to be a significant time demand.

Again, this too shouldn't be forcing the decision either way on what we
want to do.

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Jeremiah Jordan <je...@apache.org>.
When I first read this thread I assumed the go download was using some
standard tool similar to mvnw or gradlew which download and cache a copy of
the respective tool on use, with md5sum checks and such to verify the
download before use.

But that doesn’t seem to be the case here, the script checks for arm vs
amd64, Linux vs Mac, and then fetches and untars the go distro into tmp.
There is no verification of the download.  The only check is if curl
returned non 0.

I did some google searches and while
there does exist a “gow” tool, it does not seem to be maintained and also
seems to not be in widespread use within the go community.

So while it would be nice to keep things such that someone just runs ant
and gets everything built, given this does not seem to be a standard method
of dealing with a go install in build scripts, I would suggest we stop
doing it.  It looks to be very simple to install  Go, so maybe switch to
telling someone how to install it if it is not found, as well as giving
them the setting to disable that artifact.

-Jeremiah

On Mon, Apr 28, 2025 at 1:49 PM Brandon Williams <dr...@gmail.com> wrote:

> On Mon, Apr 28, 2025 at 12:19 PM Jon Haddad <jo...@rustyrazorblade.com>
> wrote:
> >
> > I strongly prefer we didn't install things on users systems they didn't
> ask for.
>
> I agree with you, and it also doesn't make sense to install go for one
> small thing, but depend on (not install) python for a lot more things.
> If we aren't willing to install python why are we installing go?
>
> Kind Regards,
> Brandon
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Apr 28, 2025 at 12:19 PM Jon Haddad <jo...@rustyrazorblade.com> wrote:
>
> I strongly prefer we didn't install things on users systems they didn't ask for.

I agree with you, and it also doesn't make sense to install go for one
small thing, but depend on (not install) python for a lot more things.
If we aren't willing to install python why are we installing go?

Kind Regards,
Brandon

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Jon Haddad <jo...@rustyrazorblade.com>.
I strongly prefer we didn't install things on users systems they didn't ask
for.  I think we're also wasting people's time by building the website by
default.

ant isn't a popular tool.  The more we ask people to know about it, the
more we turn people away from the project.  I'm personally unwilling to
invest the time in learning what's otherwise a dead technology, and I can't
be the only one.  We should by default do the thing that most people want
by default, which is building the server.

Jon

On Mon, Apr 28, 2025 at 9:54 AM Alex Petrov <al...@coffeenco.de> wrote:

> I’ve talked to some folks privately and looks like ant jar is a more
> common workflow than just ant. I also don’t mind disabling check style and
> doc gen for my purposes. But I would suggest to not auto install, or hide
> installation behind a flag and suggest user to opt in to this.
>
> One more reason to not install is that this way we commit to maintaining
> installing go.
>
> Ideally we would not pull in one more language for a small task but it’s
> hard for me to judge how difficult it would’ve been to implement this
> feature differently.
>
> On Sun, Apr 27, 2025, at 6:20 PM, Mick Semb Wever wrote:
>
> Pulling out to a separate thread, as requested.
>
> Should golang auto-install when it's not found, when executing `ant
> gen-doc` ?
>
> Our build system has a number of path dependencies: e.g. ant, maven,
> python, golang, gradle, git, grep, curl, docker, make.  Gradle and golang
> will auto-install if not found.
>
> Golang was recently introduced for CASSANDRA-19915, helping consolidate
> and improve the website documentation pages for our native protocol
> specification.  This was work that came over from the gocql community.
>
> https://cassandra.apache.org/doc/latest/cassandra/reference/native-protocol.html
>
> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc
>
> Both python and golang are dependencies for in-tree docs generation (`ant
> gen-doc`).  Python is also used for tests.
>
> While python is expected to be found, golang like gradle will be
> downloaded and installed if it is not.  But unlike gradle, golang is only
> temporarily installed into your $TMPDIR and only used for the native
> protocol doc page generation.
>
> https://github.com/apache/cassandra/blob/trunk/doc/scripts/process-native-protocol-specs-in-docker.sh
>
>
> If you don't want to be executing gen-doc (note that `ant jar` doesn't
> already), you can just add `-Dant.gen-doc.skip=true` to your ant
> cmdline/config.  `ant check` does invoke it, because code changes can break
> doc generation.
>
> Is this a problem ? Why ?  What can/should be done ?
>
>
>
>
> On Fri, 25 Apr 2025 at 13:13, Alex Petrov <al...@coffeenco.de> wrote:
>
>
> Thank you for input.
>
> I will disable gen-doc in addition to checkstyle locally, but I would
> still suggest that in short-term we should at very least remove/disable the
> part that auto-installs go (or at least have a discussion or vote about it,
> in case folks believe auto-installing is a good idea), but ideally in the
> long term we can find a solution that does not require go.
>
> On Thu, Apr 24, 2025, at 12:39 PM, Štefan Miklošovič wrote:
>
> I don't see it necessary to port Gradle stuff to Ant. I have noticed zero
> problems with it after CEP-15 merge. Basically a non-event.
>
> It just looks strange that we have Ant + Maven poms / resolver wired into
> it + submodule on Gradle. I mean ... wow. There is nothing wrong per se but
> ... strange. One would expect that more into the future we go, the more
> coherent / consistent everything related to the build system will be, not
> the other way around. But I totally get that this is just how things are
> for now and there is nobody to blame really.
>
> On Thu, Apr 24, 2025 at 10:39 AM Benedict <be...@apache.org> wrote:
>
>
> We should separate out any grade discussion. I’m happy to migrate accord
> to ant if that’s the project preference, but there’s continual discussion
> to begin modularising Cassandra (at least a little), and a proposal to use
> grade for the modules - which might be a happy medium for everyone’s
> competing priorities.
>
>
> On 24 Apr 2025, at 07:26, Mick Semb Wever <mc...@apache.org> wrote:
>
> 
> Does this also apply to gradle, which now gets downloaded and installed,
> and is the most recent addition ?
>
> The python requirement from gen-doc has been around for over three years
> now.
>
> I agree with the rationale that `ant` should default to `ant check`,
> keeping newcomers in mind while being more just a final pre-commit action
> for seasoned devs where `ant jar` is the typical dev loop.  I'm sure this
> has been discussed in the past.
>
>
>
> On Thu, 24 Apr 2025 at 08:16, Alex Petrov <al...@coffeenco.de> wrote:
>
>
> I didn’t say we should disable checks. I thought this was a purpose of CI
> : to check everything: there’s likely more to break semantically anyways,
> also harder to detect and fix. I’m also not proposing removing something
> that was long in place, these dependencies were introduced very recently.
>
> On Wed, Apr 23, 2025, at 11:51 PM, Jeremiah Jordan wrote:
>
> I think the default build should be to build and check everything.  I
> think that if someone is new it is better to have everything built and
> checked by default to flag issues.
>
> If someone knows what they are doing and wants to speed up the process it
> is very easy to add the right settings to the ant command so things are
> faster.
>
> -Jeremiah
>
> On Wed, Apr 23, 2025 at 4:36 PM Jordan West <jo...@gmail.com> wrote:
>
> Should we consider making that the default and then passing false
> explicitly in CI/builds? I agree with Alex it’s a bit surprising and
> shorter build times when developing would be helpful.
>
> Jordan
>
> On Wed, Apr 23, 2025 at 13:37 Mick Semb Wever <mc...@apache.org> wrote:
>
> Python and Go are used by the gen-doc target.
>
> Code changes can break these, hence it is part of `ant check`.
> It is not called by `ant jar`
>
> If you want to run check but skip it, it's to add `-Dant.gen-doc.skip=true`
>
>
>
> On Wed, 23 Apr 2025 at 22:06, Alex Petrov <al...@coffeenco.de> wrote:
>
>
> Hi folks,
>
> Building Cassandra jar has been getting increasingly slow, and now it
> looks like we depend not only on python3 (which was already not optimal),
> but also on go:
>
>     ant -Dno-checkstyle=true
>
> ...
>
>      [exec] python3 ./scripts/gen-nodetool-docs.py
>      [exec] python3 ./scripts/convert_yaml_to_adoc.py
> ../conf/cassandra.yaml
> ./modules/cassandra/pages/managing/configuration/cass_yaml_file.adoc
>      [exec] ./scripts/process-native-protocol-specs-in-docker.sh
>      [exec] Go env not found in your system, proceeding with installation.
>      [exec] Downloading Go 1.23.1...
>      [exec] Installing Go 1.23.1...
>      [exec] Building the cqlprotodoc...
>      [exec] Cloning into 'cassandra-website'...
>      [exec] Your branch is up to date with 'origin/trunk'.
>      [exec] go: downloading github.com/mvdan/xurls v1.1.0
>      [exec] Processing the .spec files...
>
> I personally consider this extremely dangerous, but also unnecessary. My
> current stance is that functionality introducing python3 and go should be
> moved to a separate task that only runs on demand / ci / release.  I
> welcome convincing arguments that would suggest otherwise.
>
> If you agree we should not require python and go to run `ant
> -Dno-checkstyle=true`, please also write a short message, this will be very
> helpful as well.
>
> Thank you,
> --Alex
>
>
>
>
>
>

Re: [DISCUSS] auto-installing golang in `ant gen-doc` (CASSANDRA-19915)

Posted by Alex Petrov <al...@coffeenco.de>.
I’ve talked to some folks privately and looks like ant jar is a more common workflow than just ant. I also don’t mind disabling check style and doc gen for my purposes. But I would suggest to not auto install, or hide installation behind a flag and suggest user to opt in to this. 

One more reason to not install is that this way we commit to maintaining installing go. 

Ideally we would not pull in one more language for a small task but it’s hard for me to judge how difficult it would’ve been to implement this feature differently.

On Sun, Apr 27, 2025, at 6:20 PM, Mick Semb Wever wrote:
> Pulling out to a separate thread, as requested.
> 
> Should golang auto-install when it's not found, when executing `ant gen-doc` ? 
> 
> Our build system has a number of path dependencies: e.g. ant, maven, python, golang, gradle, git, grep, curl, docker, make.  Gradle and golang will auto-install if not found.
> 
> Golang was recently introduced for CASSANDRA-19915, helping consolidate and improve the website documentation pages for our native protocol specification.  This was work that came over from the gocql community.
> https://cassandra.apache.org/doc/latest/cassandra/reference/native-protocol.html 
> https://github.com/apache/cassandra-website/tree/trunk/cqlprotodoc 
> 
> Both python and golang are dependencies for in-tree docs generation (`ant gen-doc`).  Python is also used for tests.
> 
> While python is expected to be found, golang like gradle will be downloaded and installed if it is not.  But unlike gradle, golang is only temporarily installed into your $TMPDIR and only used for the native protocol doc page generation.
> https://github.com/apache/cassandra/blob/trunk/doc/scripts/process-native-protocol-specs-in-docker.sh 
> 
> If you don't want to be executing gen-doc (note that `ant jar` doesn't already), you can just add `-Dant.gen-doc.skip=true` to your ant cmdline/config.  `ant check` does invoke it, because code changes can break doc generation.
> 
> Is this a problem ? Why ?  What can/should be done ?
> 
> 
> 
> 
> On Fri, 25 Apr 2025 at 13:13, Alex Petrov <al...@coffeenco.de> wrote:
>> __
>> Thank you for input.
>> 
>> I will disable gen-doc in addition to checkstyle locally, but I would still suggest that in short-term we should at very least remove/disable the part that auto-installs go (or at least have a discussion or vote about it, in case folks believe auto-installing is a good idea), but ideally in the long term we can find a solution that does not require go. 
>> 
>> On Thu, Apr 24, 2025, at 12:39 PM, Štefan Miklošovič wrote:
>>> I don't see it necessary to port Gradle stuff to Ant. I have noticed zero problems with it after CEP-15 merge. Basically a non-event. 
>>> 
>>> It just looks strange that we have Ant + Maven poms / resolver wired into it + submodule on Gradle. I mean ... wow. There is nothing wrong per se but ... strange. One would expect that more into the future we go, the more coherent / consistent everything related to the build system will be, not the other way around. But I totally get that this is just how things are for now and there is nobody to blame really. 
>>> 
>>> On Thu, Apr 24, 2025 at 10:39 AM Benedict <be...@apache.org> wrote:
>>>> 
>>>> We should separate out any grade discussion. I’m happy to migrate accord to ant if that’s the project preference, but there’s continual discussion to begin modularising Cassandra (at least a little), and a proposal to use grade for the modules - which might be a happy medium for everyone’s competing priorities.
>>>> 
>>>> 
>>>>> On 24 Apr 2025, at 07:26, Mick Semb Wever <mc...@apache.org> wrote:
>>>>> 
>>>>> Does this also apply to gradle, which now gets downloaded and installed, and is the most recent addition ? 
>>>>> 
>>>>> The python requirement from gen-doc has been around for over three years now.
>>>>> 
>>>>> I agree with the rationale that `ant` should default to `ant check`, keeping newcomers in mind while being more just a final pre-commit action for seasoned devs where `ant jar` is the typical dev loop.  I'm sure this has been discussed in the past.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Thu, 24 Apr 2025 at 08:16, Alex Petrov <al...@coffeenco.de> wrote:
>>>>>> __
>>>>>> I didn’t say we should disable checks. I thought this was a purpose of CI : to check everything: there’s likely more to break semantically anyways, also harder to detect and fix. I’m also not proposing removing something that was long in place, these dependencies were introduced very recently.
>>>>>> 
>>>>>> On Wed, Apr 23, 2025, at 11:51 PM, Jeremiah Jordan wrote:
>>>>>>> I think the default build should be to build and check everything.  I think that if someone is new it is better to have everything built and checked by default to flag issues.
>>>>>>> 
>>>>>>> If someone knows what they are doing and wants to speed up the process it is very easy to add the right settings to the ant command so things are faster.
>>>>>>> 
>>>>>>> -Jeremiah
>>>>>>> 
>>>>>>> On Wed, Apr 23, 2025 at 4:36 PM Jordan West <jo...@gmail.com> wrote:
>>>>>>>> Should we consider making that the default and then passing false explicitly in CI/builds? I agree with Alex it’s a bit surprising and shorter build times when developing would be helpful. 
>>>>>>>> 
>>>>>>>> Jordan 
>>>>>>>> 
>>>>>>>> On Wed, Apr 23, 2025 at 13:37 Mick Semb Wever <mc...@apache.org> wrote:
>>>>>>>>> Python and Go are used by the gen-doc target.
>>>>>>>>> 
>>>>>>>>> Code changes can break these, hence it is part of `ant check`.
>>>>>>>>> It is not called by `ant jar`
>>>>>>>>> 
>>>>>>>>> If you want to run check but skip it, it's to add `-Dant.gen-doc.skip=true`
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, 23 Apr 2025 at 22:06, Alex Petrov <al...@coffeenco.de> wrote:
>>>>>>>>>> __
>>>>>>>>>> Hi folks,
>>>>>>>>>> 
>>>>>>>>>> Building Cassandra jar has been getting increasingly slow, and now it looks like we depend not only on python3 (which was already not optimal), but also on go:
>>>>>>>>>> 
>>>>>>>>>>     ant -Dno-checkstyle=true
>>>>>>>>>> 
>>>>>>>>>> ...
>>>>>>>>>> 
>>>>>>>>>>      [exec] python3 ./scripts/gen-nodetool-docs.py
>>>>>>>>>>      [exec] python3 ./scripts/convert_yaml_to_adoc.py ../conf/cassandra.yaml ./modules/cassandra/pages/managing/configuration/cass_yaml_file.adoc
>>>>>>>>>>      [exec] ./scripts/process-native-protocol-specs-in-docker.sh
>>>>>>>>>>      [exec] Go env not found in your system, proceeding with installation.
>>>>>>>>>>      [exec] Downloading Go 1.23.1...
>>>>>>>>>>      [exec] Installing Go 1.23.1...
>>>>>>>>>>      [exec] Building the cqlprotodoc...
>>>>>>>>>>      [exec] Cloning into 'cassandra-website'...
>>>>>>>>>>      [exec] Your branch is up to date with 'origin/trunk'.
>>>>>>>>>>      [exec] go: downloading github.com/mvdan/xurls v1.1.0
>>>>>>>>>>      [exec] Processing the .spec files...
>>>>>>>>>> 
>>>>>>>>>> I personally consider this extremely dangerous, but also unnecessary. My current stance is that functionality introducing python3 and go should be moved to a separate task that only runs on demand / ci / release.  I welcome convincing arguments that would suggest otherwise.
>>>>>>>>>> 
>>>>>>>>>> If you agree we should not require python and go to run `ant -Dno-checkstyle=true`, please also write a short message, this will be very helpful as well.
>>>>>>>>>> 
>>>>>>>>>> Thank you,
>>>>>>>>>> --Alex
>>>>>> 
>> 
>