You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Odon Copon <od...@gmail.com> on 2019/03/22 10:10:40 UTC
Re: Hadoop version
Hi Bosco,
Just noticed I never updated the thread with the final results. Well, it
worked in the end with Ranger latest version and Hadoop >= 2.7.
What you said about the interfaces and the versions was the key. After
that, with newer versions, it worked fine.
At the moment, I'm testing Ranger + Hive. Do you know if versions are being
affected here as well between Hive and Ranger?
Thanks!
On Wed, 16 Jan 2019 at 22:05, Don Bosco Durai <bo...@apache.org> wrote:
> > Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin,
> because saw some info logs regarding Ranger.
>
> This might work. I don’t think the interface changed within Hadoop 2.7 and
> Hadoop 3x. If the interface changed, you would have got runtime error and
> namenode would have not come up. You can look at the Ranger Admin portal
> and see if the Audit->Plugin got any request from the HDFS plugin.
>
>
>
> > If I pick and old version, should I modify also the pom or that doesn't
> influence?
>
> Sorry, just the changing the pom will not work. It is entirely different
> implementation.
>
>
>
> Bosco
>
>
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 1:53 PM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks, that makes sense.
>
> Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin,
> because saw some info logs regarding Ranger. Does that make sense, or will
> load but not work eventually due to the old Hadoop version?
>
> Will try tomorrow with 2.6 and an old Ranger version.
>
> If I pick and old version, should I modify also the pom or that doesn't
> influence?
>
> On Wed, 16 Jan 2019, 21:44 Don Bosco Durai <bosco@apache.org wrote:
>
> This might explain why the latest Ranger is not working for you. During
> Hadoop 2.6, we didn’t have official interface from the Hadoop team, so we
> implemented using code injection at runtime. So for Hadoop 2.6 you might
> have to go back to Ranger release 0.4. Just recompiling the latest with the
> older Hadoop version will not work…
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 12:17 PM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks Bosco,
>
> I'm using Hadoop 2.6.0 at the moment, and thinking on upgrading to 2.7.1
> at some point soon.
>
> What version should I try while I'm using 2.6?
>
> Should I modify the pom as well to reflect the exact same Hadoop version I
> have install in my system when building Ranger? Because I see 2.7 in the
> pom for almost all Ranger releases.
>
>
>
> On Wed, 16 Jan 2019 at 19:56, Don Bosco Durai <bo...@apache.org> wrote:
>
> If you are using Hadoop 2.7, then Ranger 0.7 is the best bet. Let us know
> which version you are using…
>
>
>
> As Loïc mentioned, where possible we take effort to provide backward
> compatibility. But sometimes, the plugin interface from component/service
> might change dramatically between releases. In those cases, we might not be
> able to provide backward compatibility. But we ensured Ranger Admin API
> level backward compatibility, so that you old plugins can communicate with
> new version of Ranger Admin.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 8:26 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks.
>
> Knowing which Hadoop version supports each Ranger version would be
> amazing. Because at the moment I'm having troubles with the installation
> and I'm starting thinking it must be due to versioning issues.
>
>
>
> On Wed, 16 Jan 2019 at 15:33, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Yes, this is just for Hortonworks distribution but you can assume that as
> far as Ranger is concerned some backward compliancy have been implemented,
> so Ranger 0.6.0 supports Hadoop 2.7.3 and older versions of Hadoop, and
> Ranger 1.0 supports Hadoop 3.1.0 and older ... but Bosco know this better
> than I do so I'll let him confirm.
>
> Best regards,
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 14:58, Odon Copon <od...@gmail.com> a écrit :
>
> Oh true, didn't see that Hadoop column.
>
> But this is just for Hortonworks distribution, right? I can see per
> example, in Ranger POM file on 1.2.0 release that contains a reference to
> Hadoop 2.7.1, so I'm assuming Apache Ranger 1.2.0 supports Hadoop 2.7.1.
>
>
>
> On Wed, 16 Jan 2019 at 13:49, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Hi,
>
>
>
> I think there was a misunderstanding. What you see on the left border is
> the HDP version. As HDP is a stack (a product packaged by Hortonworks), it
> comes with a lot of components (Hive, HBase, Spark ...) including Ranger
> and Hadoop (in the HDP core column, on the left side of the picture).
>
> Therefore, you can see on the picture that Hadoop 2.7.3 is supported by
> both Ranger 0.6.0 and Ranger 0.7.0 versions, as they are packaged in the
> same HDP stack.
>
>
>
> Hope this helps.
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 13:38, Odon Copon <od...@gmail.com> a écrit :
>
> Thanks Loic!
>
> That's definitely really helpful. But let me confirm and clarify what I'm
> seeing there:
>
> - ranger 0.6 just works with hadoop 2.5.*
>
> - ranger 0.7 just works with hadoop 2.6.4 and 2.6.5
>
> - ranger 1.1.0 just works with hadoop 3.0.0
>
> - ranger 1.2.0 just works with hadoop 3.1.*
>
>
>
> Is this correct?
> So Hadoop 2.7.* is not supported?
>
>
>
>
>
> On Wed, 16 Jan 2019 at 12:07, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Hi Odon,
>
>
>
> Here it is :
>
> *Error! Filename not specified.*
>
> (https://fr.hortonworks.com/products/data-platforms/hdp/ )
>
>
>
> Best regards,
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 11:53, Odon Copon <od...@gmail.com> a écrit :
>
> Thanks!
>
> Do you know if there's a table with Hadoop versions and the corresponding
> Ranger version?
>
>
>
> On Wed, 16 Jan 2019 at 10:41, Don Bosco Durai <bo...@apache.org> wrote:
>
> Ranger implements the interface published by the service component. So
> every Ranger version can only support the components which it is built
> against.
>
>
>
> For older version of components, you will have to use older version of
> Ranger which supports it.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 2:03 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Hadoop version
>
>
>
> Hi,
>
> Is there any requirement on the Hadoop version supported by Apache Ranger?
> I can see the pom.xml contains a 2.7.1 pinned version for Hadoop, but my
> question is, would it work on older versions like 2.6? Or is a hard
> requirement to be >= 2.7.1?
>
> Thanks.
>
>
Re: Hadoop version
Posted by Odon Copon <od...@gmail.com>.
For me it would be really helpful, and I'm sure it would be also for others.
Happy to help.
Thanks again.
On Tue, 26 Mar 2019, 21:26 Don Bosco Durai, <bo...@apache.org> wrote:
> Hi Odon, sorry I am not aware of it. Generally I check the pom file to see
> the dependencies. I feel we should document it as part of the release ☺
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Tuesday, March 26, 2019 at 8:42 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Hi Bosco,
>
> Do you know if there's any kind of table with all compatible version
> between Hive and Ranger?
>
>
>
> Thanks
>
>
>
> On Sat, 23 Mar 2019 at 16:18, Don Bosco Durai <bo...@apache.org> wrote:
>
> Hi Odon
>
>
>
> Good it worked.
>
>
>
> Hive is has been more stable recently. In Hive 1.2, the API to get remote
> address is not there, so you would get runtime error, which you can capture
> and ignore them.
>
>
>
> Hive team have added more hook points in recent versions. Some of them are
> for row level filtering and column masking. So if you are not going to use
> them, then you should be fine.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Friday, March 22, 2019 at 11:11 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Hi Bosco,
>
> Just noticed I never updated the thread with the final results. Well, it
> worked in the end with Ranger latest version and Hadoop >= 2.7.
>
> What you said about the interfaces and the versions was the key. After
> that, with newer versions, it worked fine.
>
>
>
> At the moment, I'm testing Ranger + Hive. Do you know if versions are
> being affected here as well between Hive and Ranger?
>
>
>
> Thanks!
>
>
>
>
>
> On Wed, 16 Jan 2019 at 22:05, Don Bosco Durai <bo...@apache.org> wrote:
>
> > Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin,
> because saw some info logs regarding Ranger.
>
> This might work. I don’t think the interface changed within Hadoop 2.7 and
> Hadoop 3x. If the interface changed, you would have got runtime error and
> namenode would have not come up. You can look at the Ranger Admin portal
> and see if the Audit->Plugin got any request from the HDFS plugin.
>
>
>
> > If I pick and old version, should I modify also the pom or that doesn't
> influence?
>
> Sorry, just the changing the pom will not work. It is entirely different
> implementation.
>
>
>
> Bosco
>
>
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 1:53 PM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks, that makes sense.
>
> Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin,
> because saw some info logs regarding Ranger. Does that make sense, or will
> load but not work eventually due to the old Hadoop version?
>
> Will try tomorrow with 2.6 and an old Ranger version.
>
> If I pick and old version, should I modify also the pom or that doesn't
> influence?
>
> On Wed, 16 Jan 2019, 21:44 Don Bosco Durai <bosco@apache.org wrote:
>
> This might explain why the latest Ranger is not working for you. During
> Hadoop 2.6, we didn’t have official interface from the Hadoop team, so we
> implemented using code injection at runtime. So for Hadoop 2.6 you might
> have to go back to Ranger release 0.4. Just recompiling the latest with the
> older Hadoop version will not work…
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 12:17 PM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks Bosco,
>
> I'm using Hadoop 2.6.0 at the moment, and thinking on upgrading to 2.7.1
> at some point soon.
>
> What version should I try while I'm using 2.6?
>
> Should I modify the pom as well to reflect the exact same Hadoop version I
> have install in my system when building Ranger? Because I see 2.7 in the
> pom for almost all Ranger releases.
>
>
>
> On Wed, 16 Jan 2019 at 19:56, Don Bosco Durai <bo...@apache.org> wrote:
>
> If you are using Hadoop 2.7, then Ranger 0.7 is the best bet. Let us know
> which version you are using…
>
>
>
> As Loïc mentioned, where possible we take effort to provide backward
> compatibility. But sometimes, the plugin interface from component/service
> might change dramatically between releases. In those cases, we might not be
> able to provide backward compatibility. But we ensured Ranger Admin API
> level backward compatibility, so that you old plugins can communicate with
> new version of Ranger Admin.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 8:26 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks.
>
> Knowing which Hadoop version supports each Ranger version would be
> amazing. Because at the moment I'm having troubles with the installation
> and I'm starting thinking it must be due to versioning issues.
>
>
>
> On Wed, 16 Jan 2019 at 15:33, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Yes, this is just for Hortonworks distribution but you can assume that as
> far as Ranger is concerned some backward compliancy have been implemented,
> so Ranger 0.6.0 supports Hadoop 2.7.3 and older versions of Hadoop, and
> Ranger 1.0 supports Hadoop 3.1.0 and older ... but Bosco know this better
> than I do so I'll let him confirm.
>
> Best regards,
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 14:58, Odon Copon <od...@gmail.com> a écrit :
>
> Oh true, didn't see that Hadoop column.
>
> But this is just for Hortonworks distribution, right? I can see per
> example, in Ranger POM file on 1.2.0 release that contains a reference to
> Hadoop 2.7.1, so I'm assuming Apache Ranger 1.2.0 supports Hadoop 2.7.1.
>
>
>
> On Wed, 16 Jan 2019 at 13:49, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Hi,
>
>
>
> I think there was a misunderstanding. What you see on the left border is
> the HDP version. As HDP is a stack (a product packaged by Hortonworks), it
> comes with a lot of components (Hive, HBase, Spark ...) including Ranger
> and Hadoop (in the HDP core column, on the left side of the picture).
>
> Therefore, you can see on the picture that Hadoop 2.7.3 is supported by
> both Ranger 0.6.0 and Ranger 0.7.0 versions, as they are packaged in the
> same HDP stack.
>
>
>
> Hope this helps.
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 13:38, Odon Copon <od...@gmail.com> a écrit :
>
> Thanks Loic!
>
> That's definitely really helpful. But let me confirm and clarify what I'm
> seeing there:
>
> - ranger 0.6 just works with hadoop 2.5.*
>
> - ranger 0.7 just works with hadoop 2.6.4 and 2.6.5
>
> - ranger 1.1.0 just works with hadoop 3.0.0
>
> - ranger 1.2.0 just works with hadoop 3.1.*
>
>
>
> Is this correct?
> So Hadoop 2.7.* is not supported?
>
>
>
>
>
> On Wed, 16 Jan 2019 at 12:07, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Hi Odon,
>
>
>
> Here it is :
>
> *Error! Filename not specified.*
>
> (https://fr.hortonworks.com/products/data-platforms/hdp/ )
>
>
>
> Best regards,
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 11:53, Odon Copon <od...@gmail.com> a écrit :
>
> Thanks!
>
> Do you know if there's a table with Hadoop versions and the corresponding
> Ranger version?
>
>
>
> On Wed, 16 Jan 2019 at 10:41, Don Bosco Durai <bo...@apache.org> wrote:
>
> Ranger implements the interface published by the service component. So
> every Ranger version can only support the components which it is built
> against.
>
>
>
> For older version of components, you will have to use older version of
> Ranger which supports it.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 2:03 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Hadoop version
>
>
>
> Hi,
>
> Is there any requirement on the Hadoop version supported by Apache Ranger?
> I can see the pom.xml contains a 2.7.1 pinned version for Hadoop, but my
> question is, would it work on older versions like 2.6? Or is a hard
> requirement to be >= 2.7.1?
>
> Thanks.
>
>
Re: Hadoop version
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Odon, sorry I am not aware of it. Generally I check the pom file to see the dependencies. I feel we should document it as part of the release ☺
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Tuesday, March 26, 2019 at 8:42 AM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Hi Bosco,
Do you know if there's any kind of table with all compatible version between Hive and Ranger?
Thanks
On Sat, 23 Mar 2019 at 16:18, Don Bosco Durai <bo...@apache.org> wrote:
Hi Odon
Good it worked.
Hive is has been more stable recently. In Hive 1.2, the API to get remote address is not there, so you would get runtime error, which you can capture and ignore them.
Hive team have added more hook points in recent versions. Some of them are for row level filtering and column masking. So if you are not going to use them, then you should be fine.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Friday, March 22, 2019 at 11:11 AM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Hi Bosco,
Just noticed I never updated the thread with the final results. Well, it worked in the end with Ranger latest version and Hadoop >= 2.7.
What you said about the interfaces and the versions was the key. After that, with newer versions, it worked fine.
At the moment, I'm testing Ranger + Hive. Do you know if versions are being affected here as well between Hive and Ranger?
Thanks!
On Wed, 16 Jan 2019 at 22:05, Don Bosco Durai <bo...@apache.org> wrote:
> Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin, because saw some info logs regarding Ranger.
This might work. I don’t think the interface changed within Hadoop 2.7 and Hadoop 3x. If the interface changed, you would have got runtime error and namenode would have not come up. You can look at the Ranger Admin portal and see if the Audit->Plugin got any request from the HDFS plugin.
> If I pick and old version, should I modify also the pom or that doesn't influence?
Sorry, just the changing the pom will not work. It is entirely different implementation.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 1:53 PM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Thanks, that makes sense.
Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin, because saw some info logs regarding Ranger. Does that make sense, or will load but not work eventually due to the old Hadoop version?
Will try tomorrow with 2.6 and an old Ranger version.
If I pick and old version, should I modify also the pom or that doesn't influence?
On Wed, 16 Jan 2019, 21:44 Don Bosco Durai <bosco@apache.org wrote:
This might explain why the latest Ranger is not working for you. During Hadoop 2.6, we didn’t have official interface from the Hadoop team, so we implemented using code injection at runtime. So for Hadoop 2.6 you might have to go back to Ranger release 0.4. Just recompiling the latest with the older Hadoop version will not work…
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 12:17 PM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Thanks Bosco,
I'm using Hadoop 2.6.0 at the moment, and thinking on upgrading to 2.7.1 at some point soon.
What version should I try while I'm using 2.6?
Should I modify the pom as well to reflect the exact same Hadoop version I have install in my system when building Ranger? Because I see 2.7 in the pom for almost all Ranger releases.
On Wed, 16 Jan 2019 at 19:56, Don Bosco Durai <bo...@apache.org> wrote:
If you are using Hadoop 2.7, then Ranger 0.7 is the best bet. Let us know which version you are using…
As Loïc mentioned, where possible we take effort to provide backward compatibility. But sometimes, the plugin interface from component/service might change dramatically between releases. In those cases, we might not be able to provide backward compatibility. But we ensured Ranger Admin API level backward compatibility, so that you old plugins can communicate with new version of Ranger Admin.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 8:26 AM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Thanks.
Knowing which Hadoop version supports each Ranger version would be amazing. Because at the moment I'm having troubles with the installation and I'm starting thinking it must be due to versioning issues.
On Wed, 16 Jan 2019 at 15:33, Loïc Chanel <lo...@telecomnancy.net> wrote:
Yes, this is just for Hortonworks distribution but you can assume that as far as Ranger is concerned some backward compliancy have been implemented, so Ranger 0.6.0 supports Hadoop 2.7.3 and older versions of Hadoop, and Ranger 1.0 supports Hadoop 3.1.0 and older ... but Bosco know this better than I do so I'll let him confirm.
Best regards,
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le mer. 16 janv. 2019 à 14:58, Odon Copon <od...@gmail.com> a écrit :
Oh true, didn't see that Hadoop column.
But this is just for Hortonworks distribution, right? I can see per example, in Ranger POM file on 1.2.0 release that contains a reference to Hadoop 2.7.1, so I'm assuming Apache Ranger 1.2.0 supports Hadoop 2.7.1.
On Wed, 16 Jan 2019 at 13:49, Loïc Chanel <lo...@telecomnancy.net> wrote:
Hi,
I think there was a misunderstanding. What you see on the left border is the HDP version. As HDP is a stack (a product packaged by Hortonworks), it comes with a lot of components (Hive, HBase, Spark ...) including Ranger and Hadoop (in the HDP core column, on the left side of the picture).
Therefore, you can see on the picture that Hadoop 2.7.3 is supported by both Ranger 0.6.0 and Ranger 0.7.0 versions, as they are packaged in the same HDP stack.
Hope this helps.
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le mer. 16 janv. 2019 à 13:38, Odon Copon <od...@gmail.com> a écrit :
Thanks Loic!
That's definitely really helpful. But let me confirm and clarify what I'm seeing there:
- ranger 0.6 just works with hadoop 2.5.*
- ranger 0.7 just works with hadoop 2.6.4 and 2.6.5
- ranger 1.1.0 just works with hadoop 3.0.0
- ranger 1.2.0 just works with hadoop 3.1.*
Is this correct?
So Hadoop 2.7.* is not supported?
On Wed, 16 Jan 2019 at 12:07, Loïc Chanel <lo...@telecomnancy.net> wrote:
Hi Odon,
Here it is :
Error! Filename not specified.
(https://fr.hortonworks.com/products/data-platforms/hdp/ )
Best regards,
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le mer. 16 janv. 2019 à 11:53, Odon Copon <od...@gmail.com> a écrit :
Thanks!
Do you know if there's a table with Hadoop versions and the corresponding Ranger version?
On Wed, 16 Jan 2019 at 10:41, Don Bosco Durai <bo...@apache.org> wrote:
Ranger implements the interface published by the service component. So every Ranger version can only support the components which it is built against.
For older version of components, you will have to use older version of Ranger which supports it.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 2:03 AM
To: <us...@ranger.apache.org>
Subject: Hadoop version
Hi,
Is there any requirement on the Hadoop version supported by Apache Ranger?
I can see the pom.xml contains a 2.7.1 pinned version for Hadoop, but my question is, would it work on older versions like 2.6? Or is a hard requirement to be >= 2.7.1?
Thanks.
Re: Hadoop version
Posted by Odon Copon <od...@gmail.com>.
Hi Bosco,
Do you know if there's any kind of table with all compatible version
between Hive and Ranger?
Thanks
On Sat, 23 Mar 2019 at 16:18, Don Bosco Durai <bo...@apache.org> wrote:
> Hi Odon
>
>
>
> Good it worked.
>
>
>
> Hive is has been more stable recently. In Hive 1.2, the API to get remote
> address is not there, so you would get runtime error, which you can capture
> and ignore them.
>
>
>
> Hive team have added more hook points in recent versions. Some of them are
> for row level filtering and column masking. So if you are not going to use
> them, then you should be fine.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Friday, March 22, 2019 at 11:11 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Hi Bosco,
>
> Just noticed I never updated the thread with the final results. Well, it
> worked in the end with Ranger latest version and Hadoop >= 2.7.
>
> What you said about the interfaces and the versions was the key. After
> that, with newer versions, it worked fine.
>
>
>
> At the moment, I'm testing Ranger + Hive. Do you know if versions are
> being affected here as well between Hive and Ranger?
>
>
>
> Thanks!
>
>
>
>
>
> On Wed, 16 Jan 2019 at 22:05, Don Bosco Durai <bo...@apache.org> wrote:
>
> > Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin,
> because saw some info logs regarding Ranger.
>
> This might work. I don’t think the interface changed within Hadoop 2.7 and
> Hadoop 3x. If the interface changed, you would have got runtime error and
> namenode would have not come up. You can look at the Ranger Admin portal
> and see if the Audit->Plugin got any request from the HDFS plugin.
>
>
>
> > If I pick and old version, should I modify also the pom or that doesn't
> influence?
>
> Sorry, just the changing the pom will not work. It is entirely different
> implementation.
>
>
>
> Bosco
>
>
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 1:53 PM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks, that makes sense.
>
> Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin,
> because saw some info logs regarding Ranger. Does that make sense, or will
> load but not work eventually due to the old Hadoop version?
>
> Will try tomorrow with 2.6 and an old Ranger version.
>
> If I pick and old version, should I modify also the pom or that doesn't
> influence?
>
> On Wed, 16 Jan 2019, 21:44 Don Bosco Durai <bosco@apache.org wrote:
>
> This might explain why the latest Ranger is not working for you. During
> Hadoop 2.6, we didn’t have official interface from the Hadoop team, so we
> implemented using code injection at runtime. So for Hadoop 2.6 you might
> have to go back to Ranger release 0.4. Just recompiling the latest with the
> older Hadoop version will not work…
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 12:17 PM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks Bosco,
>
> I'm using Hadoop 2.6.0 at the moment, and thinking on upgrading to 2.7.1
> at some point soon.
>
> What version should I try while I'm using 2.6?
>
> Should I modify the pom as well to reflect the exact same Hadoop version I
> have install in my system when building Ranger? Because I see 2.7 in the
> pom for almost all Ranger releases.
>
>
>
> On Wed, 16 Jan 2019 at 19:56, Don Bosco Durai <bo...@apache.org> wrote:
>
> If you are using Hadoop 2.7, then Ranger 0.7 is the best bet. Let us know
> which version you are using…
>
>
>
> As Loïc mentioned, where possible we take effort to provide backward
> compatibility. But sometimes, the plugin interface from component/service
> might change dramatically between releases. In those cases, we might not be
> able to provide backward compatibility. But we ensured Ranger Admin API
> level backward compatibility, so that you old plugins can communicate with
> new version of Ranger Admin.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 8:26 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Re: Hadoop version
>
>
>
> Thanks.
>
> Knowing which Hadoop version supports each Ranger version would be
> amazing. Because at the moment I'm having troubles with the installation
> and I'm starting thinking it must be due to versioning issues.
>
>
>
> On Wed, 16 Jan 2019 at 15:33, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Yes, this is just for Hortonworks distribution but you can assume that as
> far as Ranger is concerned some backward compliancy have been implemented,
> so Ranger 0.6.0 supports Hadoop 2.7.3 and older versions of Hadoop, and
> Ranger 1.0 supports Hadoop 3.1.0 and older ... but Bosco know this better
> than I do so I'll let him confirm.
>
> Best regards,
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 14:58, Odon Copon <od...@gmail.com> a écrit :
>
> Oh true, didn't see that Hadoop column.
>
> But this is just for Hortonworks distribution, right? I can see per
> example, in Ranger POM file on 1.2.0 release that contains a reference to
> Hadoop 2.7.1, so I'm assuming Apache Ranger 1.2.0 supports Hadoop 2.7.1.
>
>
>
> On Wed, 16 Jan 2019 at 13:49, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Hi,
>
>
>
> I think there was a misunderstanding. What you see on the left border is
> the HDP version. As HDP is a stack (a product packaged by Hortonworks), it
> comes with a lot of components (Hive, HBase, Spark ...) including Ranger
> and Hadoop (in the HDP core column, on the left side of the picture).
>
> Therefore, you can see on the picture that Hadoop 2.7.3 is supported by
> both Ranger 0.6.0 and Ranger 0.7.0 versions, as they are packaged in the
> same HDP stack.
>
>
>
> Hope this helps.
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 13:38, Odon Copon <od...@gmail.com> a écrit :
>
> Thanks Loic!
>
> That's definitely really helpful. But let me confirm and clarify what I'm
> seeing there:
>
> - ranger 0.6 just works with hadoop 2.5.*
>
> - ranger 0.7 just works with hadoop 2.6.4 and 2.6.5
>
> - ranger 1.1.0 just works with hadoop 3.0.0
>
> - ranger 1.2.0 just works with hadoop 3.1.*
>
>
>
> Is this correct?
> So Hadoop 2.7.* is not supported?
>
>
>
>
>
> On Wed, 16 Jan 2019 at 12:07, Loïc Chanel <lo...@telecomnancy.net>
> wrote:
>
> Hi Odon,
>
>
>
> Here it is :
>
> *Error! Filename not specified.*
>
> (https://fr.hortonworks.com/products/data-platforms/hdp/ )
>
>
>
> Best regards,
>
>
>
>
> Loïc CHANEL
> System Big Data engineer
> Vision 360 Degrés (Lyon, France)
>
>
>
>
>
> Le mer. 16 janv. 2019 à 11:53, Odon Copon <od...@gmail.com> a écrit :
>
> Thanks!
>
> Do you know if there's a table with Hadoop versions and the corresponding
> Ranger version?
>
>
>
> On Wed, 16 Jan 2019 at 10:41, Don Bosco Durai <bo...@apache.org> wrote:
>
> Ranger implements the interface published by the service component. So
> every Ranger version can only support the components which it is built
> against.
>
>
>
> For older version of components, you will have to use older version of
> Ranger which supports it.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Odon Copon <od...@gmail.com>
> *Reply-To: *<us...@ranger.apache.org>
> *Date: *Wednesday, January 16, 2019 at 2:03 AM
> *To: *<us...@ranger.apache.org>
> *Subject: *Hadoop version
>
>
>
> Hi,
>
> Is there any requirement on the Hadoop version supported by Apache Ranger?
> I can see the pom.xml contains a 2.7.1 pinned version for Hadoop, but my
> question is, would it work on older versions like 2.6? Or is a hard
> requirement to be >= 2.7.1?
>
> Thanks.
>
>
Re: Hadoop version
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Odon
Good it worked.
Hive is has been more stable recently. In Hive 1.2, the API to get remote address is not there, so you would get runtime error, which you can capture and ignore them.
Hive team have added more hook points in recent versions. Some of them are for row level filtering and column masking. So if you are not going to use them, then you should be fine.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Friday, March 22, 2019 at 11:11 AM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Hi Bosco,
Just noticed I never updated the thread with the final results. Well, it worked in the end with Ranger latest version and Hadoop >= 2.7.
What you said about the interfaces and the versions was the key. After that, with newer versions, it worked fine.
At the moment, I'm testing Ranger + Hive. Do you know if versions are being affected here as well between Hive and Ranger?
Thanks!
On Wed, 16 Jan 2019 at 22:05, Don Bosco Durai <bo...@apache.org> wrote:
> Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin, because saw some info logs regarding Ranger.
This might work. I don’t think the interface changed within Hadoop 2.7 and Hadoop 3x. If the interface changed, you would have got runtime error and namenode would have not come up. You can look at the Ranger Admin portal and see if the Audit->Plugin got any request from the HDFS plugin.
> If I pick and old version, should I modify also the pom or that doesn't influence?
Sorry, just the changing the pom will not work. It is entirely different implementation.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 1:53 PM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Thanks, that makes sense.
Tried with Ranger 1.2.0 and Hadoop 2.7 and seems that loads the plugin, because saw some info logs regarding Ranger. Does that make sense, or will load but not work eventually due to the old Hadoop version?
Will try tomorrow with 2.6 and an old Ranger version.
If I pick and old version, should I modify also the pom or that doesn't influence?
On Wed, 16 Jan 2019, 21:44 Don Bosco Durai <bosco@apache.org wrote:
This might explain why the latest Ranger is not working for you. During Hadoop 2.6, we didn’t have official interface from the Hadoop team, so we implemented using code injection at runtime. So for Hadoop 2.6 you might have to go back to Ranger release 0.4. Just recompiling the latest with the older Hadoop version will not work…
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 12:17 PM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Thanks Bosco,
I'm using Hadoop 2.6.0 at the moment, and thinking on upgrading to 2.7.1 at some point soon.
What version should I try while I'm using 2.6?
Should I modify the pom as well to reflect the exact same Hadoop version I have install in my system when building Ranger? Because I see 2.7 in the pom for almost all Ranger releases.
On Wed, 16 Jan 2019 at 19:56, Don Bosco Durai <bo...@apache.org> wrote:
If you are using Hadoop 2.7, then Ranger 0.7 is the best bet. Let us know which version you are using…
As Loïc mentioned, where possible we take effort to provide backward compatibility. But sometimes, the plugin interface from component/service might change dramatically between releases. In those cases, we might not be able to provide backward compatibility. But we ensured Ranger Admin API level backward compatibility, so that you old plugins can communicate with new version of Ranger Admin.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 8:26 AM
To: <us...@ranger.apache.org>
Subject: Re: Hadoop version
Thanks.
Knowing which Hadoop version supports each Ranger version would be amazing. Because at the moment I'm having troubles with the installation and I'm starting thinking it must be due to versioning issues.
On Wed, 16 Jan 2019 at 15:33, Loïc Chanel <lo...@telecomnancy.net> wrote:
Yes, this is just for Hortonworks distribution but you can assume that as far as Ranger is concerned some backward compliancy have been implemented, so Ranger 0.6.0 supports Hadoop 2.7.3 and older versions of Hadoop, and Ranger 1.0 supports Hadoop 3.1.0 and older ... but Bosco know this better than I do so I'll let him confirm.
Best regards,
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le mer. 16 janv. 2019 à 14:58, Odon Copon <od...@gmail.com> a écrit :
Oh true, didn't see that Hadoop column.
But this is just for Hortonworks distribution, right? I can see per example, in Ranger POM file on 1.2.0 release that contains a reference to Hadoop 2.7.1, so I'm assuming Apache Ranger 1.2.0 supports Hadoop 2.7.1.
On Wed, 16 Jan 2019 at 13:49, Loïc Chanel <lo...@telecomnancy.net> wrote:
Hi,
I think there was a misunderstanding. What you see on the left border is the HDP version. As HDP is a stack (a product packaged by Hortonworks), it comes with a lot of components (Hive, HBase, Spark ...) including Ranger and Hadoop (in the HDP core column, on the left side of the picture).
Therefore, you can see on the picture that Hadoop 2.7.3 is supported by both Ranger 0.6.0 and Ranger 0.7.0 versions, as they are packaged in the same HDP stack.
Hope this helps.
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le mer. 16 janv. 2019 à 13:38, Odon Copon <od...@gmail.com> a écrit :
Thanks Loic!
That's definitely really helpful. But let me confirm and clarify what I'm seeing there:
- ranger 0.6 just works with hadoop 2.5.*
- ranger 0.7 just works with hadoop 2.6.4 and 2.6.5
- ranger 1.1.0 just works with hadoop 3.0.0
- ranger 1.2.0 just works with hadoop 3.1.*
Is this correct?
So Hadoop 2.7.* is not supported?
On Wed, 16 Jan 2019 at 12:07, Loïc Chanel <lo...@telecomnancy.net> wrote:
Hi Odon,
Here it is :
Error! Filename not specified.
(https://fr.hortonworks.com/products/data-platforms/hdp/ )
Best regards,
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le mer. 16 janv. 2019 à 11:53, Odon Copon <od...@gmail.com> a écrit :
Thanks!
Do you know if there's a table with Hadoop versions and the corresponding Ranger version?
On Wed, 16 Jan 2019 at 10:41, Don Bosco Durai <bo...@apache.org> wrote:
Ranger implements the interface published by the service component. So every Ranger version can only support the components which it is built against.
For older version of components, you will have to use older version of Ranger which supports it.
Bosco
From: Odon Copon <od...@gmail.com>
Reply-To: <us...@ranger.apache.org>
Date: Wednesday, January 16, 2019 at 2:03 AM
To: <us...@ranger.apache.org>
Subject: Hadoop version
Hi,
Is there any requirement on the Hadoop version supported by Apache Ranger?
I can see the pom.xml contains a 2.7.1 pinned version for Hadoop, but my question is, would it work on older versions like 2.6? Or is a hard requirement to be >= 2.7.1?
Thanks.