You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sfc-gh-mrojas (via GitHub)" <gi...@apache.org> on 2025/03/06 15:56:59 UTC
[I] Writing to Iceberg Tables in Polaris not working [iceberg]
sfc-gh-mrojas opened a new issue, #12469:
URL: https://github.com/apache/iceberg/issues/12469
### Apache Iceberg version
1.8.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
When using polaris catalog to write to Iceberg from Spark, it stopped working after 1.7.1.
It does not work for 1.8.0 and 1.8.1
This setup fails
```
import findspark
findspark.init()
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when
EXTRA_CLASS_PATH="/Users/mrojas/icebert_tutorial/iceberg-azure-bundle-1.8.1.jar:/Users/mrojas/icebert_tutorial/iceberg-spark-runtime-3.5_2.12-1.8.1.jar"
spark = SparkSession.builder.appName('iceberg_lab')\
.config('spark.driver.extraClassPath', EXTRA_CLASS_PATH) \
.config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
.config('spark.sql.defaultCatalog', 'polaris') \
.config('spark.sql.catalog.polaris', 'org.apache.iceberg.spark.SparkCatalog') \
.config('spark.sql.catalog.polaris.type', 'rest') \
.config('spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation', 'true') \
.config('spark.sql.catalog.polaris.uri',f'https://{POLARIS_CATALOG_ACCOUNT_IDENTIFIER}.snowflakecomputing.com/polaris/api/catalog') \
.config('spark.sql.catalog.polaris.oauth2-server-uri',f'https://{POLARIS_CATALOG_ACCOUNT_IDENTIFIER}.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens') \
.config('spark.sql.catalog.polaris.credential',f'{CLIENT_ID}:{CLIENT_SECRET}') \
.config('spark.sql.catalog.polaris.warehouse','AZURE_CATALOG') \
.config('spark.sql.catalog.polaris.scope',f'PRINCIPAL_ROLE:ALL')
.getOrCreate()
```
Using old jars it works
EXTRA_CLASS_PATH="/Users/mrojas/icebert_tutorial/iceberg-azure-bundle-1.6.1.jar:/Users/mrojas/icebert_tutorial/iceberg-spark-runtime-3.5_2.12-1.7.1.jar"
It retuns an error related to authentation:
```
com.azure.identity.CredentialUnavailableException: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/environmentcredential/troubleshoot
WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/java/identity/workloadidentitycredential/troubleshoot
Managed Identity authentication is not available.
SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
IntelliJ Authentication not available. Please log in with Azure Tools for IntelliJ plugin in the IDE. Fore more details refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/intellijcredential/troubleshoot
AzureCliCredential authentication unavailable. Azure CLI not installed.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/azclicredential/troubleshoot
Az.Account module with version >= 2.2.0 is not installed. It needs to be installed to use Azure PowerShell Credential.
AzureDeveloperCliCredential authentication unavailable. Azure Developer CLI not installed.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/azdevclicredential/troubleshootTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azure-identity-java-default-azure-credential-troubleshoot
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Writing to Iceberg Tables in Polaris not working [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed issue #12469: Writing to Iceberg Tables in Polaris not working
URL: https://github.com/apache/iceberg/issues/12469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Writing to Iceberg Tables in Polaris not working [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #12469:
URL: https://github.com/apache/iceberg/issues/12469#issuecomment-2706231209
Please open the issue at https://github.com/apache/polaris as the Iceberg community doesn't maintain Polaris
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org