You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "abharath9 (via GitHub)" <gi...@apache.org> on 2024/01/16 20:41:58 UTC
[I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
abharath9 opened a new issue, #9488:
URL: https://github.com/apache/iceberg/issues/9488
Throwing following error when trying into insert data into the Iceberg table with not-null columns constraints.
**_Cannot write nullable values to non-null column 'id' exception_**
Here is a sample code to reproduce this issue.
`from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, concat, when, countDistinct, lit, size
from pyspark.sql.types import StringType
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, ArrayType
conf = (
SparkConf()
.setAppName("analyst-hero-spark-write-iceberg")
.set("spark.sql.adaptive.coalescePartitions.enabled","true")
.set("spark.sql.parquet.filterPushdown","true")
.set("spark.hadoop.fs.s3a.impl.disable.cache","false")
.set("spark.sql.catalog.spark_catalog", 'org.apache.iceberg.spark.SparkSessionCatalog')
.set("spark.sql.extensions", 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
.set("spark.hadoop.iceberg.hive.lock-timeout-ms", '180')
.set("spark.hadoop.iceberg.delete-files-on-table-drop", 'true')
.set("spark.hadoop.iceberg.mr.commit.table.thread.pool.size", '1')
.set("spark.hadoop.iceberg.mr.commit.file.thread.pool.size", '1')
.set("spark.hadoop.iceberg.mr.commit.retry.num-retries", '300')
.set("spark.sql.iceberg.check-nullability", 'false')
)
spark = SparkSession.builder.enableHiveSupport().config(conf=conf).getOrCreate()
spark.sql("CREATE TABLE IF NOT EXISTS sandbox.iceberg_table_mandatory_column_test (id string not null,name string,nationality string) USING iceberg")
spark.sql("show create table sandbox.iceberg_table_mandatory_column_test").select("createtab_stmt").show(truncate=False)
from pyspark.sql import Row
flatData = spark.createDataFrame([Row(id='100', name='bob', nationality='welsh'),Row(id='200', name='john', nationality='british')])
flatData.show(truncate=False)`
Tried to insert the data using sql-insert into and also using Dataframe API insertInto both are returning **_Cannot write nullable values to non-null column 'id' exception_**
`flatData.createOrReplaceTempView("test_data")
spark.sql("insert into sandbox.iceberg_table_mandatory_column_test select * from test_data")
or
flatData.write.insertInto("sandbox.iceberg_table_mandatory_column_test")`
Complete stack trace in the attachment.
[error-iceberg-test.txt](https://github.com/apache/iceberg/files/13955530/error-iceberg-test.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "aagamasjain (via GitHub)" <gi...@apache.org>.
aagamasjain commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-2837639922
Hello @1316147945 may I know if COALESCE(column,0) is applied on dataFrame or on spark.sql view?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-3449054848
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9488: Cannot write nullable values to non-null column in the Iceberg Table
URL: https://github.com/apache/iceberg/issues/9488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-2408755005
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-1895861177
@abharath9 it's likely that `select * from test_data` contains null values, which you are then trying to insert into `id` which is defined as `non-null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "abharath9 (via GitHub)" <gi...@apache.org>.
abharath9 commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-1896331390
@nastra Yes i am aware of that. How do i write optional fields data to the mandatory fields data. It is mentioned in this issue that it is possible by setting "spark.sql.iceberg.check-nullability", 'false' in spark config but it is not working. I wonder if there's anything that i am missing or any other config to add.
https://github.com/apache/iceberg/pull/514
Just FYI, the above code is working fine if i upgrade to spark 3.5 and iceberg 1.4.0. I want to get this working with Spark 3.2.0 and Iceberg 0.13.0.
Thanks and appreciate any help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-3514445603
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-1898787154
Sorry it wasn't clear from the description what the goal was and I overlooked the usage of `spark.sql.iceberg.check-nullability`.
It's difficult to tell why it doesn't work with Iceberg 0.13.0 at this point (because numerous features/enhancements have been released in the meantime). We typically would recommend to use a newer Spark/Iceberg version that is [maintained](https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Cannot write nullable values to non-null column in the Iceberg Table [iceberg]
Posted by "1316147945 (via GitHub)" <gi...@apache.org>.
1316147945 commented on issue #9488:
URL: https://github.com/apache/iceberg/issues/9488#issuecomment-2434044363
COALESCE(column,0) maybe useful, and SET `spark.sql.iceberg.check-nullability`=`false` + COALESCE(column,0) can solve problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org