You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel Haviv <da...@veracity-group.com> on 2015/07/09 18:12:41 UTC

DataFrame insertInto fails, saveAsTable works (Azure HDInsight)

Hi,
I'm running Spark 1.4 on Azure.
DataFrame's insertInto fails, but when saveAsTable works.
It seems like some issue with accessing Azure's blob storage but that
doesn't explain why one type of write works and the other doesn't.

This is the stack trace:

Caused by: org.apache.hadoop.fs.azure.AzureException:
org.apache.hadoop.fs.azure.KeyProviderException: Unable to load key
provider class.

        at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:938)

        at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:438)

        at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1048)

        at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)

        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)

        at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)

        at
org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2618)

        at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:417)

        at
org.apache.hadoop.hive.shims.Hadoop23Shims.getNonCachedFileSystem(Hadoop23Shims.java:574)

        at
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3424)

        at
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3396)

        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)

        ... 59 more

Caused by: org.apache.hadoop.fs.azure.KeyProviderException: Unable to load
key provider class.

        at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:829)

        at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:917)

        ... 70 more

Caused by: java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider not found

        at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)

        at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:826)

        ... 71 more


Thanks,

Daniel

Re: DataFrame insertInto fails, saveAsTable works (Azure HDInsight)

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
It seems an issue with the azure, there was a discussion over here
https://azure.microsoft.com/en-in/documentation/articles/hdinsight-hadoop-spark-install/

Thanks
Best Regards

On Thu, Jul 9, 2015 at 9:42 PM, Daniel Haviv <
daniel.haviv@veracity-group.com> wrote:

> Hi,
> I'm running Spark 1.4 on Azure.
> DataFrame's insertInto fails, but when saveAsTable works.
> It seems like some issue with accessing Azure's blob storage but that
> doesn't explain why one type of write works and the other doesn't.
>
> This is the stack trace:
>
> Caused by: org.apache.hadoop.fs.azure.AzureException:
> org.apache.hadoop.fs.azure.KeyProviderException: Unable to load key
> provider class.
>
>         at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:938)
>
>         at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:438)
>
>         at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1048)
>
>         at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
>
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>
>         at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>
>         at
> org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2618)
>
>         at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:417)
>
>         at
> org.apache.hadoop.hive.shims.Hadoop23Shims.getNonCachedFileSystem(Hadoop23Shims.java:574)
>
>         at
> org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3424)
>
>         at
> org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3396)
>
>         at
> org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
>
>         ... 59 more
>
> Caused by: org.apache.hadoop.fs.azure.KeyProviderException: Unable to load
> key provider class.
>
>         at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:829)
>
>         at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:917)
>
>         ... 70 more
>
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider not found
>
>         at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
>
>         at
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:826)
>
>         ... 71 more
>
>
> Thanks,
>
> Daniel
>