You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Srinivas M <sm...@gmail.com> on 2018/09/24 15:28:02 UTC
Re: Exception while writing a Parquet File in a Kerberos Secured Cluster
Issues while writing Parquet files.
On Mon, Sep 24, 2018 at 8:51 PM Srinivas M <sm...@gmail.com> wrote:
> Hi
>
> We have an application that writes parquet files. I am using the
> AvroParquetWriter to write parquet files. While this piece of code works
> fine in a Kerberos environment, it is failing when SSL is enabled in the
> Hadoop cluster. So, I had modified the code to use the swebhdfs protocol
> instead of the webhdfs and it is still failing with the following exception.
>
> {
> conf.set("hadoop.security.authentication", "kerberos");
> UserGroupInformation.setConfiguration(conf);
> ugi =
> UserGroupInformation.loginUserFromKeytabAndReturnUGI(_user,_keytab) ;
>
> try
> {
> ugi.doAs(new PrivilegedExceptionAction<Object>()
> {
> public Object run() throws IOException
> {
> fs = FileSystem.get(hdfsuri,conf) ;
>
> if (_fileExistsAction == VAL_FILEEXISTS_OVERWRITE)
> fs.delete(new Path(_fileName),false) ;
>
> _writer = new AvroParquetWriter(new Path(hdfsuri),
> _schema, _ParquetCompressionCodec, _ParquetBlockSize, _ParquetPageSize);
> return null ;
> }
> }
> );
>
> hdfsuri in this case is of the format "swebhdfs://"+_host+":"+_port + "/"
> + _fileName"
>
> *The application is failing with the following exception :*
> *===========================================*
> Error :
> Caused by
> org.apache.hadoop.ipc.RemoteException(javax.ws.rs.WebApplicationException):
> null
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:124)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:420)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:108)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:596)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:674)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:524)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:554)
> at
> java.security.AccessController.doPrivileged(AccessController.java:488)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:550)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.create(WebHdfsFileSystem.java:1257)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:926)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:225)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:302)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:253)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:219)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:153)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:119)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:92)
> at parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:66)
> at parquet.avro.AvroParquetWriter.<init>(AvroParquetWriter.java:54)
> at
> com.ibm.iis.jis.utilities.parquet.ParquetBuilder$1.run(ParquetBuilder.java:191)
> at
> java.security.AccessController.doPrivileged(AccessController.java:488)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> com.ibm.iis.jis.utilities.parquet.ParquetBuilder.open(ParquetBuilder.java:182)
> at
> com.ibm.iis.jis.utilities.dochandler.impl.OutputBuilder.<init>(OutputBuilder.java:83)
> at
> com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder(Registrar.java:331)
> at
> com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder(Registrar.java:293)
> at
> com.ibm.iis.cc.filesystem.FileSystem.getBuilder(FileSystem.java:2177)
> at
> com.ibm.iis.cc.filesystem.FileSystem.writeDelimitedFiles(FileSystem.java:1168)
> at com.ibm.iis.cc.filesystem.FileSystem.writeFiles(FileSystem.java:922)
> at com.ibm.iis.cc.filesystem.FileSystem.process(FileSystem.java:763)
> at
> com.ibm.is.cc.javastage.connector.CC_JavaAdapter.run(CC_JavaAdapter.java:443)
>
>
>
> Any reason why the writing is failing with security exception ? I have
> checked the Kerberos and SSL Debug logs, but there is no indication why it
> is failing with security exception..
>
>
> --
> Srinivas
> (*-*)
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> You have to grow from the inside out. None can teach you, none can make
> you spiritual.
> -Narendra Nath Dutta(Swamy Vivekananda)
>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
--
Srinivas
(*-*)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
You have to grow from the inside out. None can teach you, none can make you
spiritual.
-Narendra Nath Dutta(Swamy Vivekananda)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------