[jira] [Created] (HADOOP-19247) Authentification failed in Azure Kubernetes with HTTP1.1 and Chunked transfer encoding

["Emeric (Jira)" <ji...@apache.org>]

Emeric created HADOOP-19247:
-------------------------------

             Summary: Authentification failed in Azure Kubernetes with HTTP1.1 and Chunked transfer encoding
                 Key: HADOOP-19247
                 URL: https://issues.apache.org/jira/browse/HADOOP-19247
             Project: Hadoop Common
          Issue Type: Bug
          Components: auth
    Affects Versions: 3.3.6, 3.3.4, 3.4.0, 3.5.0
         Environment: Azure Kubernetes Services

Azure Entra ID

Azure Metadata Service

Spark 3.3
            Reporter: Emeric
         Attachments: CodeResponse.png, TokenKO.png, TokenOK.png

 

The problem is related to Azure authentication on Kubernetes.

When I run my Spark program, I have this error when I try to authenticate the pod :

 
{code:java}
java.lang.NullPointerException
    at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.consumeInputStream(AzureADAuthenticator.java:340)
    at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenSingleCall(AzureADAuthenticator.java:270)
    at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenCall(AzureADAuthenticator.java:211)
    at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenFromMsi(AzureADAuthenticator.java:137)
    at org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider.refreshToken(MsiTokenProvider.java:45)
    at org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider.getToken(AccessTokenProvider.java:50)
    at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAccessToken(AbfsClient.java:554)
    at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:151)
    at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:125)
    at org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:181)
    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:569)
    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:536)
    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:359) {code}
 

 

My configuration is a spark-driver deployed on Azure kubernetes with managed identity.

I used [this method|https://medium.com/datamindedbe/running-spark-3-on-aks-with-azure-ad-integration-c1fc0032c550] with aad-pod-identity.

 


There are two different scenarios we can observe when trying to authenticate on Kubernetes to Azure Instance Metadata Service :
 * The returned token is short and its size is less than 2048 chars. The Token have all headers and explicitly the "Content-Length" header

!TokenOK.png!
 * The returned token is long and its size is more than 2048 chars. The Token have the HTTP1.1 capacity with transfer encoding property in Response and don't have the "Content-length" header due to Chunked transfer encoding mechanism.

!TokenKO.png!

 

NB : I run a curl command in pod to generate these sceenshots according to the [Azure Documentation|https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=linux]

In a GitHub repository I found my "AzureADAuthenticator.java" and this piece of code :

!CodeResponse.png!

The "Content-length" property is mandatory when the returned HTTP code is 200 and it's not compatible with the HTTP1.1 Chunked transfer encoding fonctionality.

Is it possible to update this authentification to support this mechanism implemented by Microsoft on kubernetes (and may be in virtual machine).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org