Re: [VOTE] Release Apache Druid 29.0.0 [RC1]

[Karan Kumar <ka...@gmail.com>]

Hi
Vote thread for druid 29.0.1 is up.
https://lists.apache.org/thread/6syof9fmnb7vbyjrpowrt7s73rh2bqy4

Thanks
Karan

On Wed, Feb 21, 2024 at 1:10 AM Vadim Ogievetsky <vo...@apache.org>
wrote:

> That sounds great to me. Thank you +1
>
> On 2024/02/20 18:58:35 Laksh Singla wrote:
> > Since the artifacts were already out at the time of the Vad's vote, I
> will
> > be going ahead with the release of Druid 29.0.0. However, I do echo with
> > the comments mentioned in the mail, therefore we can take the following
> > steps to prevent misfortune for the unexpecting users:
> >
> > For the current Druid 29.0.0 release:
> > 1. The docs (update, release, and general docs) will be updated with the
> > possible pitfalls arising out of console injecting the "arrayIngestMode"
> > automatically, and persisting it for the given tab.
> >
> > Further, we can have a patch release, 29.0.1, with the following patches
> > mentioned in the Vad's mail, that would have been a part of RC2, had the
> > vote not passed through at the time:
> > 1. https://github.com/apache/druid/pull/15927
> > 2. https://github.com/apache/druid/pull/15920
> >
> > This will address the concerns regarding the 29.0.0 release, and also
> give
> > enough time for the PRs to get reviewed without being hastened by the
> > release timeline. I'll volunteer to be the release manager for the same.
> > Thoughts?
> >
> > Regards
> > Laksh
> >
> >
> >
> > On Tue, 20 Feb 2024 at 10:50, Abhishek Agarwal <ab...@apache.org>
> wrote:
> >
> > > Vad, The vote was closed by Laksh, given that 72 hours had passed and
> > > artifacts had been released. Given we had the three binding +1s, we can
> > > proceed as usual. I suggest that we call out the caveat clearly in the
> > > release notes section.
> > >
> > > On Mon, Feb 19, 2024 at 11:53 PM Vadim Ogievetsky <
> vogievetsky@apache.org>
> > > wrote:
> > >
> > > > Thank you for all the work getting this release to this point. As the
> > > > author of https://github.com/apache/druid/pull/15588 I regretfully
> vote
> > > > -1 on this release. To echo Gian's point I think the chance of some
> user
> > > > injuring a production Datasource with an innocent query pasted into
> the
> > > > wrong web console tab is just too large. The fact that the
> > > > "arrayIngestMode" setting itself is not new in this release is
> immaterial
> > > > to my consideration as I am more worried about "what might happen by
> > > > accident" than "what is technically possible to mess up".
> > > >
> > > > I have made a PR (https://github.com/apache/druid/pull/15927) that
> > > > implements Gian's suggestion of: Making it obvious in the web console
> > > > whether a tab is in "arrayIngestMode: array" or "arrayIngestMode:
> mvd" or
> > > > "server default". Furthermore this PR also minimizes the use of
> > > > "arrayIngestMode: array" to only when needed while still generating
> the
> > > > "correct" form of the SQL queries at all times. If this PR was
> merged and
> > > > back-ported I would change my vote to +1.
> > > >
> > > > I think that Gian's validation PR (
> > > > https://github.com/apache/druid/pull/15920) is an even better
> solution,
> > > > as it would protect API usage and usage via other frontends also.
> > > Obviously
> > > > we could merge both.
> > > >
> > > > Another idea that would solve this issue would be have the ability
> to set
> > > > context flags via SQL perhaps with a PARAMETER keyword. If this was
> > > > possible the console could exclusively use that for setting query
> context
> > > > parameters on generated queries.
> > > >
> > > > Vadim
> > > >
> > > > On 2024/02/17 02:52:38 Gian Merlino wrote:
> > > > > Here's a patch with the validation idea:
> > > > https://github.com/apache/druid/pull/15920
> > > > >
> > > > > It adds validation for the most problematic case (mixing strings
> and
> > > > arrays), provides a way to override the validation, and makes the
> warning
> > > > log on the controller task when arrayIngestMode is 'mvd' more
> friendly
> > > and
> > > > explanatory.
> > > > >
> > > > > Depending on which direction you're going in, the errors look like
> > > > either:
> > > > >
> > > > >   Cannot write into field[flags] using type[VARCHAR ARRAY] and
> > > > arrayIngestMode[mvd], since the existing type is[VARCHAR ARRAY]. Try
> > > > setting arrayIngestMode to[array] to retain the SQL type[VARCHAR
> ARRAY]
> > > > >
> > > > > Or:
> > > > >
> > > > >   Cannot write into field[flags] using type[VARCHAR ARRAY] and
> > > > arrayIngestMode[array], since the existing type is[VARCHAR]. Try
> wrapping
> > > > this field using ARRAY_TO_MV(...) AS "flags"
> > > > >
> > > > > The "try" language pushes people towards the behavior we'd like
> for the
> > > > future: using arrayIngestMode[array] and wrapping MVDs in
> ARRAY_TO_MV.
> > > > >
> > > > > I'm changing my vote to a plain 0, given that _most_ of the changes
> > > > related to arrayIngestMode went out in Druid 28. However I do think
> it
> > > > would be nice to get a patch like this in, given that the Druid 29
> web
> > > > console is pushing more people to change their arrayIngestMode.
> > > > >
> > > > > Gian
> > > > >
> > > > > On 2024/02/16 22:24:23 Gian Merlino wrote:
> > > > > > I just learned that arrayIngestMode is not actually new, just
> > > > > > https://github.com/apache/druid/pull/15588 is. However this will
> > > > still make
> > > > > > it more likely that people accidentally break their tables, so I
> am
> > > > still
> > > > > > -0. Just, slightly less so. I still think it would be a good
> idea,
> > > for
> > > > > > Druid 29, to add string-to-array type validation to Druid 29's
> > > INSERT /
> > > > > > REPLACE handling to compensate for the new web console support
> for
> > > > > > arrayIngestMode, and the UI changes to push people towards
> setting it
> > > > to
> > > > > > "array".
> > > > > >
> > > > > > I could be convinced that it's ok to do that in a 29.0.1. I don't
> > > > think it
> > > > > > should wait for 30, given the impact that can happen if people
> end up
> > > > with
> > > > > > mixed types without planning for it.
> > > > > >
> > > > > > On Fri, Feb 16, 2024 at 2:16 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Thanks for managing this release!
> > > > > > >
> > > > > > > My vote is -0, let me explain why. I am concerned about
> usability
> > > > issues
> > > > > > > with the new arrayIngestMode feature. There are various issues
> when
> > > > mixing
> > > > > > > MVD strings and string arrays in the same column: as soon as
> arrays
> > > > show up
> > > > > > > in a column, various "classic MVD-style" queries will fail at
> > > > validation or
> > > > > > > at runtime. I believe that the new feature, and especially the
> > > > changes to
> > > > > > > the web console in https://github.com/apache/druid/pull/15588,
> > > will
> > > > make
> > > > > > > it more likely that people will do this by accident and
> experience
> > > > > > > brokenness.
> > > > > > >
> > > > > > > When this occurs, there is not an easy way to fix it; data
> needs to
> > > > be
> > > > > > > reingested or queries need to be adjusted. I believe that in
> some
> > > > cases,
> > > > > > > queries can't be adjusted without suffering from performance
> loss.
> > > > > > >
> > > > > > > This is something that could happen even before Druid 29, if
> you
> > > did
> > > > a
> > > > > > > Kafka ingest with auto types or useSchemaDiscovery, followed
> by a
> > > SQL
> > > > > > > REPLACE from that table into itself. In that case, arrays
> written
> > > by
> > > > Kafka
> > > > > > > ingest would get rewritten as MVDs. But with this Druid 29 RC,
> > > there
> > > > are
> > > > > > > additional pathways created that enable people to get into this
> > > > problematic
> > > > > > > scenario with increased likelihood. I suggest we make some
> > > > adjustments that
> > > > > > > prevent it, such as:
> > > > > > >
> > > > > > > - Validating that SQL INSERT and SQL REPLACE do not insert an
> array
> > > > type
> > > > > > > into a column that previously contained strings, or vice versa.
> > > > > > > - Making it obvious in the web console whether a tab is in
> > > > > > > "arrayIngestMode: array" or "arrayIngestMode: mvd" or "server
> > > > default".
> > > > > > >
> > > > > > > Thank you for considering this viewpoint.
> > > > > > >
> > > > > > > Gian
> > > > > > >
> > > > > > > On Tue, Feb 13, 2024 at 4:34 AM Laksh Singla <
> > > lakshsingla@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> I have created a build for Apache Druid 29.0.0, release
> > > > > > >> candidate 1.
> > > > > > >>
> > > > > > >> Thanks to everyone who has helped contribute to the release!
> You
> > > > can read
> > > > > > >> the proposed release notes here:
> > > > > > >> https://github.com/apache/druid/issues/15896
> > > > > > >>
> > > > > > >> The release candidate has been tagged in GitHub as
> > > > > > >> druid-29.0.0-rc1 (869bd3978f0c835ef8eb7c1f25c468e23472a81b),
> > > > > > >> available here:
> > > > > > >> https://github.com/apache/druid/tree/druid-29.0.0-rc1
> > > > > > >>
> > > > > > >> The artifacts to be voted on are located here:
> > > > > > >> https://dist.apache.org/repos/dist/dev/druid/29.0.0-rc1/
> > > > > > >>
> > > > > > >> A staged Maven repository is available for review at:
> > > > > > >>
> > > >
> https://repository.apache.org/content/repositories/orgapachedruid-1061/
> > > > > > >>
> > > > > > >> Staged druid.apache.org website documentation is available
> here:
> > > > > > >> https://druid.staged.apache.org/docs/29.0.0/design/
> > > > > > >>
> > > > > > >> A Docker image containing the binary of the release candidate
> can
> > > be
> > > > > > >> retrieved via:
> > > > > > >> docker pull apache/druid:29.0.0-rc1
> > > > > > >>
> > > > > > >> artifact checksums
> > > > > > >> src:
> > > > > > >>
> > > > > > >>
> > > >
> > >
> 1948fab4500f3571591f887a638631b6a05f040b88b35004406ca852e16884e5d2269a74e8d07c79c477d8b02a2489efec85776f8ef46f4e8defedce4efb9931
> > > > > > >> bin:
> > > > > > >>
> > > > > > >>
> > > >
> > >
> f2a11ddc71b59a648d01d7c220a6fae527c34d702d4f1e7aed954803a3576543c4cd7149f34e8948eccf89240435f09a8512db83b46991ac6e354eeba8cbada4
> > > > > > >> docker:
> > > > 2bddcf692f2137dc4094908b63d2043423fca895ba0f479f0db34ec8016c4472
> > > > > > >>
> > > > > > >> Release artifacts are signed with the following key:
> > > > > > >> https://people.apache.org/keys/committer/lakshsingla.asc
> > > > > > >>
> > > > > > >> This key and the key of other committers can also be found in
> the
> > > > > > >> project's
> > > > > > >> KEYS file here:
> > > > > > >> https://dist.apache.org/repos/dist/release/druid/KEYS
> > > > > > >>
> > > > > > >> (If you are a committer, please feel free to add your own key
> to
> > > > that file
> > > > > > >> by following the instructions in the file's header.)
> > > > > > >>
> > > > > > >>
> > > > > > >> Verify checksums:
> > > > > > >> diff <(shasum -a512 apache-druid-29.0.0-src.tar.gz | \
> > > > > > >> cut -d ' ' -f1) \
> > > > > > >> <(cat apache-druid-29.0.0-src.tar.gz.sha512 ; echo)
> > > > > > >>
> > > > > > >> diff <(shasum -a512 apache-druid-29.0.0-bin.tar.gz | \
> > > > > > >> cut -d ' ' -f1) \
> > > > > > >> <(cat apache-druid-29.0.0-bin.tar.gz.sha512 ; echo)
> > > > > > >>
> > > > > > >> Verify signatures:
> > > > > > >> gpg --verify apache-druid-29.0.0-src.tar.gz.asc \
> > > > > > >> apache-druid-29.0.0-src.tar.gz
> > > > > > >>
> > > > > > >> gpg --verify apache-druid-29.0.0-bin.tar.gz.asc \
> > > > > > >> apache-druid-29.0.0-bin.tar.gz
> > > > > > >>
> > > > > > >> Please review the proposed artifacts and vote. Note that
> Apache
> > > has
> > > > > > >> specific requirements that must be met before +1 binding
> votes can
> > > > be cast
> > > > > > >> by PMC members. Please refer to the policy at
> > > > > > >> http://www.apache.org/legal/release-policy.html#policy for
> more
> > > > details.
> > > > > > >>
> > > > > > >> As part of the validation process, the release artifacts can
> be
> > > > generated
> > > > > > >> from source by running:
> > > > > > >> mvn clean install -Papache-release,dist -Dgpg.skip
> > > > > > >>
> > > > > > >> The RAT license check can be run from source by:
> > > > > > >> mvn apache-rat:check -Prat
> > > > > > >>
> > > > > > >> This vote will be open for at least 72 hours. The vote will
> pass
> > > if
> > > > a
> > > > > > >> majority of at least three +1 PMC votes are cast.
> > > > > > >>
> > > > > > >> [ ] +1 Release this package as Apache Druid 29.0.0
> > > > > > >> [ ] 0 I don't feel strongly about it, but I'm okay with the
> > > release
> > > > > > >> [ ] -1 Do not release this package because...
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > > For additional commands, e-mail: dev-help@druid.apache.org
> > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

-- 
Thanks
Karan