You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by FABIAN Juan-antonio <ju...@alstomgroup.com> on 2020/01/24 10:24:30 UTC
Hive expection: Class org.apache.hadoop.fs.adl.AdlFileSystem not
found
Hello,
I'm getting an exception when running a NiFi flow, using the PutHiveQL processor:
10:12:58 UTC ERROR 5f9d00fe-e6dd-3c05-8698-50cab74ee142 nifi-node-0:8080
PutHiveQL[id=5f9d00fe-e6dd-3c05-8698-50cab74ee142] Failed to process session due to Failed to process StandardFlowFileRecord[uuid=8bd48767-dbb3-42c1-8492-4063e1aee6ec,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1579790709540-2, container=default, section=2], offset=448724, length=182],offset=0,name=3613831121596506,size=182] due to java.sql.SQLException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found);: org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=8bd48767-dbb3-42c1-8492-4063e1aee6ec,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1579790709540-2, container=default, section=2], offset=448724, length=182],offset=0,name=3613831121596506,size=182] due to java.sql.SQLException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found);
I've posted this question on the NiFi users list, but maybe you can help me a bit (I'm new both to Hive and NiFi).
I had a NiFi cluster with a flow using PutHDFS, and after that, I used PutHiveQL to addPartition. Now I've setup another NiFi cluster, with git as the flow provider, and we replaced HDFS with Minio, so instead using PutParquet/PutHDFS, I'm using PutS3Object.
AFAIK, PutHiveQL is talking to our Thrift server, so no need for Hadoop or AzureDataLake dependencies. However, I'm getting this. Both NiFi clusters are deployed on a custom Kubernetes installation on Azure.
I'm kind of lost here, so any help is much appreciated.
Thanks
Juan A. Fabián Simón
Data Engineer
Alstom
Calle Martínez Villergas 49, ed. V - 28027 Madrid - Spain
Office: +34 91 384 89 00
Email: <ma...@alstomgroup.com> juan-antonio.fabian@alstomgroup.com
www.alstom.com<http://www.alstom.com>
[cid:850c3903-2713-45c3-9040-c2f64d22bc89] [cid:3539e2fe-0e83-445a-be92-2063029e4590] <https://twitter.com/Alstom> [cid:fa891413-1d00-4c29-8789-4fe7149b2bef] <https://www.linkedin.com/company/alstom/> [cid:34c4d7b6-85be-4793-8910-9805a338456b] <https://www.facebook.com/ALSTOM/> [cid:c542d5f3-d10b-4fda-8c03-fdc0d155e987] <https://www.instagram.com/alstom> [cid:f6b0d063-0c70-4343-85ef-57dc6b720b3d] <https://www.youtube.com/user/Alstom>
________________________________
CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium.
RE: Hive expection: Class org.apache.hadoop.fs.adl.AdlFileSystem not
found
Posted by FABIAN Juan-antonio <ju...@alstomgroup.com>.
Hello,
just an update. I finally got it working. There were two issues:
* Even we're not using HDFS, we need the core-site.xml, where we define the storage layer (minio in our case).
* Apart from that, before the PutHiveQL, the queue still contained some elements pointing to an "adl://..." instead "s3a://...", so each time PutHiveQL picked one, it failed.
After referencing the core-site.xml and empty the queues, everything works.
Best,
________________________________
De: FABIAN Juan-antonio <ju...@alstomgroup.com>
Enviado: viernes, 24 de enero de 2020 11:24
Para: user@hive.apache.org <us...@hive.apache.org>
Asunto: Hive expection: Class org.apache.hadoop.fs.adl.AdlFileSystem not found
Hello,
I'm getting an exception when running a NiFi flow, using the PutHiveQL processor:
10:12:58 UTC ERROR 5f9d00fe-e6dd-3c05-8698-50cab74ee142 nifi-node-0:8080
PutHiveQL[id=5f9d00fe-e6dd-3c05-8698-50cab74ee142] Failed to process session due to Failed to process StandardFlowFileRecord[uuid=8bd48767-dbb3-42c1-8492-4063e1aee6ec,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1579790709540-2, container=default, section=2], offset=448724, length=182],offset=0,name=3613831121596506,size=182] due to java.sql.SQLException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found);: org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=8bd48767-dbb3-42c1-8492-4063e1aee6ec,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1579790709540-2, container=default, section=2], offset=448724, length=182],offset=0,name=3613831121596506,size=182] due to java.sql.SQLException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found);
I've posted this question on the NiFi users list, but maybe you can help me a bit (I'm new both to Hive and NiFi).
I had a NiFi cluster with a flow using PutHDFS, and after that, I used PutHiveQL to addPartition. Now I've setup another NiFi cluster, with git as the flow provider, and we replaced HDFS with Minio, so instead using PutParquet/PutHDFS, I'm using PutS3Object.
AFAIK, PutHiveQL is talking to our Thrift server, so no need for Hadoop or AzureDataLake dependencies. However, I'm getting this. Both NiFi clusters are deployed on a custom Kubernetes installation on Azure.
I'm kind of lost here, so any help is much appreciated.
Thanks
Juan A. Fabián Simón
Data Engineer
Alstom
Calle Martínez Villergas 49, ed. V - 28027 Madrid - Spain
Office: +34 91 384 89 00
Email: <ma...@alstomgroup.com> juan-antonio.fabian@alstomgroup.com
www.alstom.com<http://www.alstom.com>
[cid:850c3903-2713-45c3-9040-c2f64d22bc89] [cid:3539e2fe-0e83-445a-be92-2063029e4590] <https://twitter.com/Alstom> [cid:fa891413-1d00-4c29-8789-4fe7149b2bef] <https://www.linkedin.com/company/alstom/> [cid:34c4d7b6-85be-4793-8910-9805a338456b] <https://www.facebook.com/ALSTOM/> [cid:c542d5f3-d10b-4fda-8c03-fdc0d155e987] <https://www.instagram.com/alstom> [cid:f6b0d063-0c70-4343-85ef-57dc6b720b3d] <[The%20URL%20sent%20was%20deleted%20due%20to%20the%20non-respect%20of%20Alstom%20policies]>
________________________________
CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium.
________________________________
Some URLs of this mail have been deactivated due to non-respect of Alstom s policies
________________________________
CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium.