You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Piotr Szuberski <pi...@polidea.com> on 2020/10/22 07:35:47 UTC
Problem with updating Hadoop and Hcatalog dependencies
I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've encountered a problem - the Hadoop related checks seem to work without any further changes but Hcatalog requires to be bumped to 3.x.y versions as well (2.x.y versions require Hadoop 2.x.y).
When I use Hadoop 3.3.0 there is a guava jar versions related exception [1] which I tried to solve by enforcing Guava 27.0-jre which is used by Hadoop 3.3.0 - without success.
Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive 3.1.2. I also replaced hive-site.xml with the recent default one from Hive's master. Then 4 tests from io/hcatalog are failing:
testWriteThenReadSuccess - with exception [2]
testWriteThenUnboundedReadSuccess - with the same exception.
As far as I deduced it's a bit misleading because setOutput indeed is called in HCatalogIO.Write's writerContext = masterWriter.prepareWrite() - which under the hood tries to call setOutput and fails.
The probable cause could be Hcatalog configuration. But I definitely lack knowledge how to set it up, especially the Hcatalog's version 3.x documentation really doesn't help.
Do we have anyone with some knowledge about HCatalog that could help me with this?
[1] NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V
[2] org.apache.beam.sdk.util.UserCodeException: org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.lang.NullPointerException
Re: Problem with updating Hadoop and Hcatalog dependencies
Posted by Piotr Szuberski <pi...@polidea.com>.
The NPE comes from AuthorizationMetaStoreFilterHook#getFilteredObjects at:
SessionState ss = SessionState.get(); // returns null
HiveAuthzContext.Builder authzContextBuilder = new HiveAuthzContext.Builder();
authzContextBuilder.setUserIpAddress(ss.getUserIpAddress());
Stacktrace: [3]
[3]
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook.getFilteredObjects(AuthorizationMetaStoreFilterHook.java:78)
at org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook.filterDatabases(AuthorizationMetaStoreFilterHook.java:55)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:1345)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:1340)
at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.isOpen(HiveClientCache.java:471)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
at com.sun.proxy.$Proxy88.isOpen(Unknown Source)
at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:270)
at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:569)
at org.apache.hive.hcatalog.mapreduce.HCatOutputFormat.setOutput(HCatOutputFormat.java:90)
Re: Problem with updating Hadoop and Hcatalog dependencies
Posted by Piotr Szuberski <pi...@polidea.com>.
I've tried to use the configuration from tests from Hive master [1], but without luck.
[1] https://github.com/apache/hive/blob/3f9cf3809617aa3405e732b569b95a33ce14424a/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatBaseTest.java#L47
Re: Problem with updating Hadoop and Hcatalog dependencies
Posted by Piotr Szuberski <pi...@polidea.com>.
Okay! Together with the update to 2.10.1 I'll make the tests for 2.7.7 and later for 3.2.1.
On 2020/10/30 14:49:51, Ismaël Mejía <ie...@gmail.com> wrote:
> 2.x changes are usually compatible but still we have to test it a bit.
>
> The target in Hadoop dependencies is always the oldest because the full
> ecosystem is still tight to those e.g. Spark/Flink.
>
> It would be awesome if we could get tests to be run with earlier versions too at
> least with 2.7.x the oldest target at the present.
>
> On Fri, Oct 30, 2020 at 12:21 PM Piotr Szuberski
> <pi...@polidea.com> wrote:
> >
> > Ok I see. I'll update Hadoop to the recent 2.10.1.
> >
> > On 2020/10/28 16:48:57, Tyson Hamilton <ty...@google.com> wrote:
> > >
> > >
> > > On 2020/10/22 07:35:47, Piotr Szuberski <pi...@polidea.com> wrote:
> > > > I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've encountered a problem - the Hadoop related checks seem to work without any further changes but Hcatalog requires to be bumped to 3.x.y versions as well (2.x.y versions require Hadoop 2.x.y).
> > >
> > > Are there any backwards incompatible changes with the Hadoop major version bump? If so it may be better to push this off until the next major Beam release and just to a minor version bump for Hadoop.
> > >
> > > >
> > > > When I use Hadoop 3.3.0 there is a guava jar versions related exception [1] which I tried to solve by enforcing Guava 27.0-jre which is used by Hadoop 3.3.0 - without success.
> > > >
> > > > Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive 3.1.2. I also replaced hive-site.xml with the recent default one from Hive's master. Then 4 tests from io/hcatalog are failing:
> > > > testWriteThenReadSuccess - with exception [2]
> > > > testWriteThenUnboundedReadSuccess - with the same exception.
> > > >
> > > > As far as I deduced it's a bit misleading because setOutput indeed is called in HCatalogIO.Write's writerContext = masterWriter.prepareWrite() - which under the hood tries to call setOutput and fails.
> > > >
> > > > The probable cause could be Hcatalog configuration. But I definitely lack knowledge how to set it up, especially the Hcatalog's version 3.x documentation really doesn't help.
> > > >
> > > > Do we have anyone with some knowledge about HCatalog that could help me with this?
> > >
> > > timrobertson100@ is listed as the owner of this dependency [1].
> > >
> > > [1] https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml
> > >
> > > >
> > > >
> > > > [1] NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V
> > > > [2] org.apache.beam.sdk.util.UserCodeException: org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.lang.NullPointerException
> > > >
> > >
>
Re: Problem with updating Hadoop and Hcatalog dependencies
Posted by Ismaël Mejía <ie...@gmail.com>.
2.x changes are usually compatible but still we have to test it a bit.
The target in Hadoop dependencies is always the oldest because the full
ecosystem is still tight to those e.g. Spark/Flink.
It would be awesome if we could get tests to be run with earlier versions too at
least with 2.7.x the oldest target at the present.
On Fri, Oct 30, 2020 at 12:21 PM Piotr Szuberski
<pi...@polidea.com> wrote:
>
> Ok I see. I'll update Hadoop to the recent 2.10.1.
>
> On 2020/10/28 16:48:57, Tyson Hamilton <ty...@google.com> wrote:
> >
> >
> > On 2020/10/22 07:35:47, Piotr Szuberski <pi...@polidea.com> wrote:
> > > I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've encountered a problem - the Hadoop related checks seem to work without any further changes but Hcatalog requires to be bumped to 3.x.y versions as well (2.x.y versions require Hadoop 2.x.y).
> >
> > Are there any backwards incompatible changes with the Hadoop major version bump? If so it may be better to push this off until the next major Beam release and just to a minor version bump for Hadoop.
> >
> > >
> > > When I use Hadoop 3.3.0 there is a guava jar versions related exception [1] which I tried to solve by enforcing Guava 27.0-jre which is used by Hadoop 3.3.0 - without success.
> > >
> > > Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive 3.1.2. I also replaced hive-site.xml with the recent default one from Hive's master. Then 4 tests from io/hcatalog are failing:
> > > testWriteThenReadSuccess - with exception [2]
> > > testWriteThenUnboundedReadSuccess - with the same exception.
> > >
> > > As far as I deduced it's a bit misleading because setOutput indeed is called in HCatalogIO.Write's writerContext = masterWriter.prepareWrite() - which under the hood tries to call setOutput and fails.
> > >
> > > The probable cause could be Hcatalog configuration. But I definitely lack knowledge how to set it up, especially the Hcatalog's version 3.x documentation really doesn't help.
> > >
> > > Do we have anyone with some knowledge about HCatalog that could help me with this?
> >
> > timrobertson100@ is listed as the owner of this dependency [1].
> >
> > [1] https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml
> >
> > >
> > >
> > > [1] NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V
> > > [2] org.apache.beam.sdk.util.UserCodeException: org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.lang.NullPointerException
> > >
> >
Re: Problem with updating Hadoop and Hcatalog dependencies
Posted by Piotr Szuberski <pi...@polidea.com>.
Ok I see. I'll update Hadoop to the recent 2.10.1.
On 2020/10/28 16:48:57, Tyson Hamilton <ty...@google.com> wrote:
>
>
> On 2020/10/22 07:35:47, Piotr Szuberski <pi...@polidea.com> wrote:
> > I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've encountered a problem - the Hadoop related checks seem to work without any further changes but Hcatalog requires to be bumped to 3.x.y versions as well (2.x.y versions require Hadoop 2.x.y).
>
> Are there any backwards incompatible changes with the Hadoop major version bump? If so it may be better to push this off until the next major Beam release and just to a minor version bump for Hadoop.
>
> >
> > When I use Hadoop 3.3.0 there is a guava jar versions related exception [1] which I tried to solve by enforcing Guava 27.0-jre which is used by Hadoop 3.3.0 - without success.
> >
> > Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive 3.1.2. I also replaced hive-site.xml with the recent default one from Hive's master. Then 4 tests from io/hcatalog are failing:
> > testWriteThenReadSuccess - with exception [2]
> > testWriteThenUnboundedReadSuccess - with the same exception.
> >
> > As far as I deduced it's a bit misleading because setOutput indeed is called in HCatalogIO.Write's writerContext = masterWriter.prepareWrite() - which under the hood tries to call setOutput and fails.
> >
> > The probable cause could be Hcatalog configuration. But I definitely lack knowledge how to set it up, especially the Hcatalog's version 3.x documentation really doesn't help.
> >
> > Do we have anyone with some knowledge about HCatalog that could help me with this?
>
> timrobertson100@ is listed as the owner of this dependency [1].
>
> [1] https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml
>
> >
> >
> > [1] NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V
> > [2] org.apache.beam.sdk.util.UserCodeException: org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.lang.NullPointerException
> >
>
Re: Problem with updating Hadoop and Hcatalog dependencies
Posted by Tyson Hamilton <ty...@google.com>.
On 2020/10/22 07:35:47, Piotr Szuberski <pi...@polidea.com> wrote:
> I'm trying to update Hadoop dependencies to the recent 3.3.0 and I've encountered a problem - the Hadoop related checks seem to work without any further changes but Hcatalog requires to be bumped to 3.x.y versions as well (2.x.y versions require Hadoop 2.x.y).
Are there any backwards incompatible changes with the Hadoop major version bump? If so it may be better to push this off until the next major Beam release and just to a minor version bump for Hadoop.
>
> When I use Hadoop 3.3.0 there is a guava jar versions related exception [1] which I tried to solve by enforcing Guava 27.0-jre which is used by Hadoop 3.3.0 - without success.
>
> Then I used Hadoop 3.2.0 which doesn't have guava updated and Hive 3.1.2. I also replaced hive-site.xml with the recent default one from Hive's master. Then 4 tests from io/hcatalog are failing:
> testWriteThenReadSuccess - with exception [2]
> testWriteThenUnboundedReadSuccess - with the same exception.
>
> As far as I deduced it's a bit misleading because setOutput indeed is called in HCatalogIO.Write's writerContext = masterWriter.prepareWrite() - which under the hood tries to call setOutput and fails.
>
> The probable cause could be Hcatalog configuration. But I definitely lack knowledge how to set it up, especially the Hcatalog's version 3.x documentation really doesn't help.
>
> Do we have anyone with some knowledge about HCatalog that could help me with this?
timrobertson100@ is listed as the owner of this dependency [1].
[1] https://github.com/apache/beam/blob/master/ownership/JAVA_DEPENDENCY_OWNERS.yaml
>
>
> [1] NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;J)V
> [2] org.apache.beam.sdk.util.UserCodeException: org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : java.lang.NullPointerException
>