You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhengshubin (via GitHub)" <gi...@apache.org> on 2023/04/28 00:46:47 UTC
[GitHub] [incubator-paimon] zhengshubin opened a new issue, #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
zhengshubin opened a new issue, #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
0.4-SNAPSHOT
### Compute Engine
spark3.3
### Minimal reproduce step
create table paimon.default.t1 (
pkey BIGINT ,
dt string)
TBLPROPERTIES(
'primary-key' = 'pkey',
'bucket' = '4',
'bucket-key' = 'pkey'
)
insert into paimon.default.t1 select pkey , date_format(last_tx_dt,'yyyy-MM-dd') dt from t
error:
write incompatible data to table 't1':
- Cannot write nullable values to non-null column 'pkey'
### What doesn't meet your expectations?
It can not insert success even though pk is not null.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1548971429
An inelegant way is adding a option: `spark.sql.catalog.paimon.force-nullable=true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhengshubin commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "zhengshubin (via GitHub)" <gi...@apache.org>.
zhengshubin commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1564092215
> Spark sql when executed, it will validate the column and table attribute like: if (queryExpr.nullable && !tableAttr.nullable) { addError(s"Cannot write nullable values to non-null column '${colPath.quoted}'") None } when queryExpr = "nvl(a, 1)", queryExpr.nullable will return false, so in spark-sql, nvl seems could resolve this question
Thanks. It is usefully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] leaves12138 commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1562383543
Spark sql when executed, it will validate the column and table attribute like:
if (queryExpr.nullable && !tableAttr.nullable) {
addError(s"Cannot write nullable values to non-null column '${colPath.quoted}'")
None
}
when queryExpr = "nvl(a, 1)", queryExpr.nullable will return false, so in spark-sql, nvl seems could resolve this question
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] smokeriu commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "smokeriu (via GitHub)" <gi...@apache.org>.
smokeriu commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1547341644
I'm not a paimon developer, but I think this is an issue caused by spark's detection.
spark will check the nullable property of the field, you set the pkey as the primary key, which requires it to be non-null, and the nullable property in your data may be true, in spark, you can use df.printSchema to see it.
Details: [TableOutputResolver](https://github.com/apache/spark/blob/5103e00c4ce5fcc4264ca9c4df12295d42557af6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala#L287)
> In V2-API, `StoreAssignmentPolicy.LEGACY` is already disable, so you can't use it.
> log: org.apache.spark.sql.AnalysisException: LEGACY store assignment policy is disallowed in Spark data source V2.
You can use `where pkey is not null` to ensure that its data is not null, and then use `AssertNotNull` expression to modify the nullable property.This applies to both batch and stream scenarios.
Details: [expressions.objects](https://github.com/apache/spark/blob/5103e00c4ce5fcc4264ca9c4df12295d42557af6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala#L1790)
If you are only using batch, you can choose to modify the nullable property by `sparkSession.createDataFrame(df.rdd, newSchema)`.
Details: [change-nullable-property-of-column-in-spark-dataframe](https://stackoverflow.com/questions/33193958/change-nullable-property-of-column-in-spark-dataframe)
Perhaps there are other ways to modify spark's nullable more elegantly, if so, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhengshubin commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "zhengshubin (via GitHub)" <gi...@apache.org>.
zhengshubin commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1564094090
> Can we use Spark `nvl` or `Coalesce`?
OK. It is useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi closed issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
URL: https://github.com/apache/incubator-paimon/issues/1048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] ftutor commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "ftutor (via GitHub)" <gi...@apache.org>.
ftutor commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1557046776
hive table can't be insert into paimon table if it has primary key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhengshubin closed issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "zhengshubin (via GitHub)" <gi...@apache.org>.
zhengshubin closed issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
URL: https://github.com/apache/incubator-paimon/issues/1048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhengshubin commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "zhengshubin (via GitHub)" <gi...@apache.org>.
zhengshubin commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1549077179
> I'm not a paimon developer, but I think this is an issue caused by spark's detection.
>
> spark will check the nullable property of the field, you set the pkey as the primary key, which requires it to be non-null, and the nullable property in your data may be true, in spark, you can use df.printSchema to see it. Details: [TableOutputResolver](https://github.com/apache/spark/blob/5103e00c4ce5fcc4264ca9c4df12295d42557af6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala#L287)
>
> > In V2-API, `StoreAssignmentPolicy.LEGACY` is already disable, so you can't use it.
> > log: org.apache.spark.sql.AnalysisException: LEGACY store assignment policy is disallowed in Spark data source V2.
>
> You can use `where pkey is not null` to ensure that its data is not null, and then use `AssertNotNull` expression to modify the nullable property.This applies to both batch and stream scenarios. Details: [expressions.objects](https://github.com/apache/spark/blob/5103e00c4ce5fcc4264ca9c4df12295d42557af6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala#L1790)
>
> If you are only using batch, you can choose to modify the nullable property by `sparkSession.createDataFrame(df.rdd, newSchema)`. Details: [change-nullable-property-of-column-in-spark-dataframe](https://stackoverflow.com/questions/33193958/change-nullable-property-of-column-in-spark-dataframe)
>
> Perhaps there are other ways to modify spark's nullable more elegantly, if so, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] zhengshubin commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "zhengshubin (via GitHub)" <gi...@apache.org>.
zhengshubin commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1560566440
> Can we use Spark `nvl` or `Coalesce`?
It make same error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] JingsongLi commented on issue #1048: [spark][Bug] Cannot write nullable values to non-null column when use insert into select
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #1048:
URL: https://github.com/apache/incubator-paimon/issues/1048#issuecomment-1560556261
Can we use Spark `nvl` or `Coalesce`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org