You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Gian Merlino <gi...@apache.org> on 2023/06/28 06:42:08 UTC

[DISCUSS] Druid 28 dropping support for Hadoop 2

I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the very
next release (which I assume will be Druid 27) but the one after that,
likely late 2023 timeframe.

In 2021, we had a discussion about moving away from Hadoop 2:
https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For
various reasons, it didn't seem like the right time. However, I believe now
is the right time:

1) We didn't support Hadoop 3 in 2021, but we support it now. There is now
a Hadoop 3 build profile, as well as convenience binaries on
https://druid.apache.org/downloads.html.

2) We have SQL-based ingest with MSQ tasks, which provides a built-in /
scalable / robust alternative to using Hadoop at all.

3) It has been an additional two years. Hadoop 2 is that much older, that
much more time has passed since it was superseded by Hadoop 3, and people
have had that much more time to migrate.

4) The original main reason for wanting to move away from Hadoop 2 is still
relevant. It keeps us on various old dependencies, including an ancient
version of Guava, which in turn has been keeping us on an ancient version
of Calcite. The Calcite community has graciously decided to support this
old version of Guava for at least one release, but plans to drop support by
Calcite 1.36, leaving us back in the same position. Managing this situation
is time-consuming for both Druid and Calcite maintainers.

5) Other solutions beyond dropping Hadoop 2 support were proposed in 2021,
such as reworking Hadoop support to be purely extension based, and
reworking extensions to be more isolated from each other. However, these
are both substantially more complex than dropping support, and in the two
years since the original thread, these more complex solutions have not been
implemented. So, I think we need to move on with the simpler solution of
dropping support.

Gian

Re: [DISCUSS] Druid 28 dropping support for Hadoop 2

Posted by Gian Merlino <gi...@apache.org>.
Given the replies on this thread, I think it's appropriate that we do the following now:

1) Announce in the upcoming Druid 27 release that Hadoop 2 support is deprecated and will be removed in Druid 28

2) Remove Hadoop 2 support from the master branch, since Druid 27 has been branched off already, and the next release (28) is meant to not have it.

Does anyone have some spare cycles to do (2)?

Gian

On 2023/06/28 06:42:08 Gian Merlino wrote:
> I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the very
> next release (which I assume will be Druid 27) but the one after that,
> likely late 2023 timeframe.
> 
> In 2021, we had a discussion about moving away from Hadoop 2:
> https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For
> various reasons, it didn't seem like the right time. However, I believe now
> is the right time:
> 
> 1) We didn't support Hadoop 3 in 2021, but we support it now. There is now
> a Hadoop 3 build profile, as well as convenience binaries on
> https://druid.apache.org/downloads.html.
> 
> 2) We have SQL-based ingest with MSQ tasks, which provides a built-in /
> scalable / robust alternative to using Hadoop at all.
> 
> 3) It has been an additional two years. Hadoop 2 is that much older, that
> much more time has passed since it was superseded by Hadoop 3, and people
> have had that much more time to migrate.
> 
> 4) The original main reason for wanting to move away from Hadoop 2 is still
> relevant. It keeps us on various old dependencies, including an ancient
> version of Guava, which in turn has been keeping us on an ancient version
> of Calcite. The Calcite community has graciously decided to support this
> old version of Guava for at least one release, but plans to drop support by
> Calcite 1.36, leaving us back in the same position. Managing this situation
> is time-consuming for both Druid and Calcite maintainers.
> 
> 5) Other solutions beyond dropping Hadoop 2 support were proposed in 2021,
> such as reworking Hadoop support to be purely extension based, and
> reworking extensions to be more isolated from each other. However, these
> are both substantially more complex than dropping support, and in the two
> years since the original thread, these more complex solutions have not been
> implemented. So, I think we need to move on with the simpler solution of
> dropping support.
> 
> Gian
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org


Re: [DISCUSS] Druid 28 dropping support for Hadoop 2

Posted by Gian Merlino <gi...@apache.org>.
Yes, I think it would make sense to deprecate it in Druid 27 if we're planning to remove the support in Druid 28.

I haven't looked into what it would take to make the Hadoop integration into an optional extension. That would be really nice though. Has anyone on this list looked into it?

Gian

On 2023/06/29 19:49:42 Xavier Léauté wrote:
> +1, does this mean we would mark Hadoop 2 deprecated in Druid 27?
> 
> Also, do we have a broader plan to remove Hadoop in general from core
> dependencies and make a an optional extension?
> 
> On Tue, Jun 27, 2023 at 11:53 PM Karan Kumar <ka...@gmail.com>
> wrote:
> 
> > In favour of dropping hadoop 2 support . Another point is the lack of
> > security and vulnerability fixes in hadoop2.
> >
> >
> >
> > On Wed, Jun 28, 2023 at 12:17 PM Clint Wylie <cw...@apache.org> wrote:
> >
> > > obvious +1 from me
> > >
> > > On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the
> > > very
> > > > next release (which I assume will be Druid 27) but the one after that,
> > > > likely late 2023 timeframe.
> > > >
> > > > In 2021, we had a discussion about moving away from Hadoop 2:
> > > > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For
> > > > various reasons, it didn't seem like the right time. However, I believe
> > > now
> > > > is the right time:
> > > >
> > > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There is
> > > now
> > > > a Hadoop 3 build profile, as well as convenience binaries on
> > > > https://druid.apache.org/downloads.html.
> > > >
> > > > 2) We have SQL-based ingest with MSQ tasks, which provides a built-in /
> > > > scalable / robust alternative to using Hadoop at all.
> > > >
> > > > 3) It has been an additional two years. Hadoop 2 is that much older,
> > that
> > > > much more time has passed since it was superseded by Hadoop 3, and
> > people
> > > > have had that much more time to migrate.
> > > >
> > > > 4) The original main reason for wanting to move away from Hadoop 2 is
> > > still
> > > > relevant. It keeps us on various old dependencies, including an ancient
> > > > version of Guava, which in turn has been keeping us on an ancient
> > version
> > > > of Calcite. The Calcite community has graciously decided to support
> > this
> > > > old version of Guava for at least one release, but plans to drop
> > support
> > > by
> > > > Calcite 1.36, leaving us back in the same position. Managing this
> > > situation
> > > > is time-consuming for both Druid and Calcite maintainers.
> > > >
> > > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in
> > > 2021,
> > > > such as reworking Hadoop support to be purely extension based, and
> > > > reworking extensions to be more isolated from each other. However,
> > these
> > > > are both substantially more complex than dropping support, and in the
> > two
> > > > years since the original thread, these more complex solutions have not
> > > been
> > > > implemented. So, I think we need to move on with the simpler solution
> > of
> > > > dropping support.
> > > >
> > > > Gian
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >
> > >
> >
> > --
> > Thanks
> > Karan
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org


Re: [DISCUSS] Druid 28 dropping support for Hadoop 2

Posted by rahul gidwani <ch...@apache.org>.
+1 drop hadoop 2.

On Thu, Jun 29, 2023 at 12:50 PM Xavier Léauté <xa...@confluent.io.invalid>
wrote:

> +1, does this mean we would mark Hadoop 2 deprecated in Druid 27?
>
> Also, do we have a broader plan to remove Hadoop in general from core
> dependencies and make a an optional extension?
>
> On Tue, Jun 27, 2023 at 11:53 PM Karan Kumar <ka...@gmail.com>
> wrote:
>
> > In favour of dropping hadoop 2 support . Another point is the lack of
> > security and vulnerability fixes in hadoop2.
> >
> >
> >
> > On Wed, Jun 28, 2023 at 12:17 PM Clint Wylie <cw...@apache.org> wrote:
> >
> > > obvious +1 from me
> > >
> > > On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not
> the
> > > very
> > > > next release (which I assume will be Druid 27) but the one after
> that,
> > > > likely late 2023 timeframe.
> > > >
> > > > In 2021, we had a discussion about moving away from Hadoop 2:
> > > > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4.
> For
> > > > various reasons, it didn't seem like the right time. However, I
> believe
> > > now
> > > > is the right time:
> > > >
> > > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There
> is
> > > now
> > > > a Hadoop 3 build profile, as well as convenience binaries on
> > > > https://druid.apache.org/downloads.html.
> > > >
> > > > 2) We have SQL-based ingest with MSQ tasks, which provides a
> built-in /
> > > > scalable / robust alternative to using Hadoop at all.
> > > >
> > > > 3) It has been an additional two years. Hadoop 2 is that much older,
> > that
> > > > much more time has passed since it was superseded by Hadoop 3, and
> > people
> > > > have had that much more time to migrate.
> > > >
> > > > 4) The original main reason for wanting to move away from Hadoop 2 is
> > > still
> > > > relevant. It keeps us on various old dependencies, including an
> ancient
> > > > version of Guava, which in turn has been keeping us on an ancient
> > version
> > > > of Calcite. The Calcite community has graciously decided to support
> > this
> > > > old version of Guava for at least one release, but plans to drop
> > support
> > > by
> > > > Calcite 1.36, leaving us back in the same position. Managing this
> > > situation
> > > > is time-consuming for both Druid and Calcite maintainers.
> > > >
> > > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in
> > > 2021,
> > > > such as reworking Hadoop support to be purely extension based, and
> > > > reworking extensions to be more isolated from each other. However,
> > these
> > > > are both substantially more complex than dropping support, and in the
> > two
> > > > years since the original thread, these more complex solutions have
> not
> > > been
> > > > implemented. So, I think we need to move on with the simpler solution
> > of
> > > > dropping support.
> > > >
> > > > Gian
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > > For additional commands, e-mail: dev-help@druid.apache.org
> > >
> > >
> >
> > --
> > Thanks
> > Karan
> >
>

Re: [DISCUSS] Druid 28 dropping support for Hadoop 2

Posted by Xavier Léauté <xa...@confluent.io.INVALID>.
+1, does this mean we would mark Hadoop 2 deprecated in Druid 27?

Also, do we have a broader plan to remove Hadoop in general from core
dependencies and make a an optional extension?

On Tue, Jun 27, 2023 at 11:53 PM Karan Kumar <ka...@gmail.com>
wrote:

> In favour of dropping hadoop 2 support . Another point is the lack of
> security and vulnerability fixes in hadoop2.
>
>
>
> On Wed, Jun 28, 2023 at 12:17 PM Clint Wylie <cw...@apache.org> wrote:
>
> > obvious +1 from me
> >
> > On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the
> > very
> > > next release (which I assume will be Druid 27) but the one after that,
> > > likely late 2023 timeframe.
> > >
> > > In 2021, we had a discussion about moving away from Hadoop 2:
> > > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For
> > > various reasons, it didn't seem like the right time. However, I believe
> > now
> > > is the right time:
> > >
> > > 1) We didn't support Hadoop 3 in 2021, but we support it now. There is
> > now
> > > a Hadoop 3 build profile, as well as convenience binaries on
> > > https://druid.apache.org/downloads.html.
> > >
> > > 2) We have SQL-based ingest with MSQ tasks, which provides a built-in /
> > > scalable / robust alternative to using Hadoop at all.
> > >
> > > 3) It has been an additional two years. Hadoop 2 is that much older,
> that
> > > much more time has passed since it was superseded by Hadoop 3, and
> people
> > > have had that much more time to migrate.
> > >
> > > 4) The original main reason for wanting to move away from Hadoop 2 is
> > still
> > > relevant. It keeps us on various old dependencies, including an ancient
> > > version of Guava, which in turn has been keeping us on an ancient
> version
> > > of Calcite. The Calcite community has graciously decided to support
> this
> > > old version of Guava for at least one release, but plans to drop
> support
> > by
> > > Calcite 1.36, leaving us back in the same position. Managing this
> > situation
> > > is time-consuming for both Druid and Calcite maintainers.
> > >
> > > 5) Other solutions beyond dropping Hadoop 2 support were proposed in
> > 2021,
> > > such as reworking Hadoop support to be purely extension based, and
> > > reworking extensions to be more isolated from each other. However,
> these
> > > are both substantially more complex than dropping support, and in the
> two
> > > years since the original thread, these more complex solutions have not
> > been
> > > implemented. So, I think we need to move on with the simpler solution
> of
> > > dropping support.
> > >
> > > Gian
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
> >
> >
>
> --
> Thanks
> Karan
>

Re: [DISCUSS] Druid 28 dropping support for Hadoop 2

Posted by Karan Kumar <ka...@gmail.com>.
In favour of dropping hadoop 2 support . Another point is the lack of
security and vulnerability fixes in hadoop2.



On Wed, Jun 28, 2023 at 12:17 PM Clint Wylie <cw...@apache.org> wrote:

> obvious +1 from me
>
> On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the
> very
> > next release (which I assume will be Druid 27) but the one after that,
> > likely late 2023 timeframe.
> >
> > In 2021, we had a discussion about moving away from Hadoop 2:
> > https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For
> > various reasons, it didn't seem like the right time. However, I believe
> now
> > is the right time:
> >
> > 1) We didn't support Hadoop 3 in 2021, but we support it now. There is
> now
> > a Hadoop 3 build profile, as well as convenience binaries on
> > https://druid.apache.org/downloads.html.
> >
> > 2) We have SQL-based ingest with MSQ tasks, which provides a built-in /
> > scalable / robust alternative to using Hadoop at all.
> >
> > 3) It has been an additional two years. Hadoop 2 is that much older, that
> > much more time has passed since it was superseded by Hadoop 3, and people
> > have had that much more time to migrate.
> >
> > 4) The original main reason for wanting to move away from Hadoop 2 is
> still
> > relevant. It keeps us on various old dependencies, including an ancient
> > version of Guava, which in turn has been keeping us on an ancient version
> > of Calcite. The Calcite community has graciously decided to support this
> > old version of Guava for at least one release, but plans to drop support
> by
> > Calcite 1.36, leaving us back in the same position. Managing this
> situation
> > is time-consuming for both Druid and Calcite maintainers.
> >
> > 5) Other solutions beyond dropping Hadoop 2 support were proposed in
> 2021,
> > such as reworking Hadoop support to be purely extension based, and
> > reworking extensions to be more isolated from each other. However, these
> > are both substantially more complex than dropping support, and in the two
> > years since the original thread, these more complex solutions have not
> been
> > implemented. So, I think we need to move on with the simpler solution of
> > dropping support.
> >
> > Gian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

-- 
Thanks
Karan

Re: [DISCUSS] Druid 28 dropping support for Hadoop 2

Posted by Clint Wylie <cw...@apache.org>.
obvious +1 from me

On Tue, Jun 27, 2023 at 11:42 PM Gian Merlino <gi...@apache.org> wrote:
>
> I'd like to propose dropping support for Hadoop 2 in Druid 28. Not the very
> next release (which I assume will be Druid 27) but the one after that,
> likely late 2023 timeframe.
>
> In 2021, we had a discussion about moving away from Hadoop 2:
> https://lists.apache.org/thread/zmc389trnkh6x444so8mdb2h0x0noqq4. For
> various reasons, it didn't seem like the right time. However, I believe now
> is the right time:
>
> 1) We didn't support Hadoop 3 in 2021, but we support it now. There is now
> a Hadoop 3 build profile, as well as convenience binaries on
> https://druid.apache.org/downloads.html.
>
> 2) We have SQL-based ingest with MSQ tasks, which provides a built-in /
> scalable / robust alternative to using Hadoop at all.
>
> 3) It has been an additional two years. Hadoop 2 is that much older, that
> much more time has passed since it was superseded by Hadoop 3, and people
> have had that much more time to migrate.
>
> 4) The original main reason for wanting to move away from Hadoop 2 is still
> relevant. It keeps us on various old dependencies, including an ancient
> version of Guava, which in turn has been keeping us on an ancient version
> of Calcite. The Calcite community has graciously decided to support this
> old version of Guava for at least one release, but plans to drop support by
> Calcite 1.36, leaving us back in the same position. Managing this situation
> is time-consuming for both Druid and Calcite maintainers.
>
> 5) Other solutions beyond dropping Hadoop 2 support were proposed in 2021,
> such as reworking Hadoop support to be purely extension based, and
> reworking extensions to be more isolated from each other. However, these
> are both substantially more complex than dropping support, and in the two
> years since the original thread, these more complex solutions have not been
> implemented. So, I think we need to move on with the simpler solution of
> dropping support.
>
> Gian

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org