You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "LINZ, Arnaud" <AL...@bouyguestelecom.fr> on 2020/03/03 17:02:55 UTC
RE: Building with Hadoop 3
Hello,
Have you shared it somewhere on the web already?
Best,
Arnaud
De : vino yang <ya...@gmail.com>
Envoyé : mercredi 4 décembre 2019 11:55
À : Márton Balassi <ba...@gmail.com>
Cc : Chesnay Schepler <ch...@apache.org>; Foster, Craig <fo...@amazon.com>; user@flink.apache.org; dev@flink.apache.org
Objet : Re: Building with Hadoop 3
Hi Marton,
Thanks for your explanation. Personally, I look forward to your contribution!
Best,
Vino
Márton Balassi <ba...@gmail.com>> 于2019年12月4日周三 下午5:15写道：
Wearing my Cloudera hat I can tell you that we have done this exercise for our distros of the 3.0 and 3.1 Hadoop versions. We have not contributed these back just yet, but we are open to do so. If the community is interested we can contribute those changes back to flink-shaded and suggest the necessay changes to flink too. The task was not overly complex, but it certainly involved a bit of dependency hell. :-)
Right now we are focused on internal timelines, but we could invest into contributing this back in the end of January timeframe if the community deems this a worthwhile effort.
Best,
Marton
On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <ch...@apache.org>> wrote:
There's no JIRA and no one actively working on it. I'm not aware of any investigations on the matter; hence the first step would be to just try it out.
A flink-shaded artifact isn't a hard requirement; Flink will work with any 2.X hadoop distribution (provided that there aren't any dependency clashes).
On 03/12/2019 18:22, Foster, Craig wrote:
Hi:
I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here from a year ago that no one is looking into Hadoop 3 support [1]. Is there a document or JIRA that now exists which would point to what needs to be done to support Hadoop 3? Right now builds with Hadoop 3 don’t work obviously because there’s no flink-shaded-hadoop-3 artifacts.
Thanks!
Craig
[1] https://issues.apache.org/jira/browse/FLINK-11086
________________________________
L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.
The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Re: Building with Hadoop 3
Posted by Stephan Ewen <se...@apache.org>.
Have you tried to just export Hadoop 3's classpath to `HADOOP_CLASSPATH`
and see if that works out of the box?
If the main use case is HDFS access, then there is a fair chance it might
just work, because Flink uses only a small subset of the Hadoop FS API
which is stable between 2.x and 3.x, as far as I tried it out a while back.
On Tue, Mar 3, 2020 at 6:03 PM LINZ, Arnaud <AL...@bouyguestelecom.fr>
wrote:
> Hello,
>
> Have you shared it somewhere on the web already?
>
> Best,
>
> Arnaud
>
>
>
> *De :* vino yang <ya...@gmail.com>
> *Envoyé :* mercredi 4 décembre 2019 11:55
> *À :* Márton Balassi <ba...@gmail.com>
> *Cc :* Chesnay Schepler <ch...@apache.org>; Foster, Craig <
> foscraig@amazon.com>; user@flink.apache.org; dev@flink.apache.org
> *Objet :* Re: Building with Hadoop 3
>
>
>
> Hi Marton,
>
>
>
> Thanks for your explanation. Personally, I look forward to your
> contribution!
>
>
>
> Best,
>
> Vino
>
>
>
> Márton Balassi <ba...@gmail.com> 于2019年12月4日周三 下午5:15写道：
>
> Wearing my Cloudera hat I can tell you that we have done this exercise for
> our distros of the 3.0 and 3.1 Hadoop versions. We have not contributed
> these back just yet, but we are open to do so. If the community is
> interested we can contribute those changes back to flink-shaded and suggest
> the necessay changes to flink too. The task was not overly complex, but it
> certainly involved a bit of dependency hell. :-)
>
>
>
> Right now we are focused on internal timelines, but we could invest into
> contributing this back in the end of January timeframe if the community
> deems this a worthwhile effort.
>
>
>
> Best,
>
> Marton
>
>
>
> On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> There's no JIRA and no one actively working on it. I'm not aware of any
> investigations on the matter; hence the first step would be to just try it
> out.
>
>
>
> A flink-shaded artifact isn't a hard requirement; Flink will work with any
> 2.X hadoop distribution (provided that there aren't any dependency clashes).
>
>
>
> On 03/12/2019 18:22, Foster, Craig wrote:
>
> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
> a document or JIRA that now exists which would point to what needs to be
> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
>
>
> Thanks!
>
> Craig
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>
>
>
>
>
>
> ------------------------------
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>
Re: Building with Hadoop 3
Posted by Stephan Ewen <se...@apache.org>.
Have you tried to just export Hadoop 3's classpath to `HADOOP_CLASSPATH`
and see if that works out of the box?
If the main use case is HDFS access, then there is a fair chance it might
just work, because Flink uses only a small subset of the Hadoop FS API
which is stable between 2.x and 3.x, as far as I tried it out a while back.
On Tue, Mar 3, 2020 at 6:03 PM LINZ, Arnaud <AL...@bouyguestelecom.fr>
wrote:
> Hello,
>
> Have you shared it somewhere on the web already?
>
> Best,
>
> Arnaud
>
>
>
> *De :* vino yang <ya...@gmail.com>
> *Envoyé :* mercredi 4 décembre 2019 11:55
> *À :* Márton Balassi <ba...@gmail.com>
> *Cc :* Chesnay Schepler <ch...@apache.org>; Foster, Craig <
> foscraig@amazon.com>; user@flink.apache.org; dev@flink.apache.org
> *Objet :* Re: Building with Hadoop 3
>
>
>
> Hi Marton,
>
>
>
> Thanks for your explanation. Personally, I look forward to your
> contribution!
>
>
>
> Best,
>
> Vino
>
>
>
> Márton Balassi <ba...@gmail.com> 于2019年12月4日周三 下午5:15写道：
>
> Wearing my Cloudera hat I can tell you that we have done this exercise for
> our distros of the 3.0 and 3.1 Hadoop versions. We have not contributed
> these back just yet, but we are open to do so. If the community is
> interested we can contribute those changes back to flink-shaded and suggest
> the necessay changes to flink too. The task was not overly complex, but it
> certainly involved a bit of dependency hell. :-)
>
>
>
> Right now we are focused on internal timelines, but we could invest into
> contributing this back in the end of January timeframe if the community
> deems this a worthwhile effort.
>
>
>
> Best,
>
> Marton
>
>
>
> On Wed, Dec 4, 2019 at 10:00 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> There's no JIRA and no one actively working on it. I'm not aware of any
> investigations on the matter; hence the first step would be to just try it
> out.
>
>
>
> A flink-shaded artifact isn't a hard requirement; Flink will work with any
> 2.X hadoop distribution (provided that there aren't any dependency clashes).
>
>
>
> On 03/12/2019 18:22, Foster, Craig wrote:
>
> Hi:
>
> I don’t see a JIRA for Hadoop 3 support. I see a comment on a JIRA here
> from a year ago that no one is looking into Hadoop 3 support [1]. Is there
> a document or JIRA that now exists which would point to what needs to be
> done to support Hadoop 3? Right now builds with Hadoop 3 don’t work
> obviously because there’s no flink-shaded-hadoop-3 artifacts.
>
>
>
> Thanks!
>
> Craig
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086
>
>
>
>
>
>
> ------------------------------
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>