You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ymZhao1001 (via GitHub)" <gi...@apache.org> on 2023/02/23 01:30:58 UTC
[GitHub] [iceberg] ymZhao1001 opened a new issue, #6914: change partition led to query bug
ymZhao1001 opened a new issue, #6914:
URL: https://github.com/apache/iceberg/issues/6914
### Apache Iceberg version
main (development)
### Query engine
Flink
### Please describe the bug 🐞
CREATE CATALOG c1WITH (
'type'='iceberg',
'catalog-type'='hive',
'uri'='thrift://***:9083',
'clients'='5',
'property-version'='1',
'warehouse'='hdfs://***:8020/iceberg/'
);
use catalog c1;
create database d1;
use d1;
CREATE TABLE t1(
ss_create_time TIMESTAMP(3),
ss_update_time TIMESTAMP(3),
ss_user_name STRING,
ss_product_name STRING,
ss_shop_name STRING,
ss_city_name STRING,
ss_country_name STRING,
ss_sold_date_sk BIGINT,
ss_sold_time_sk BIGINT,
ss_item_sk BIGINT,
ss_customer_sk BIGINT,
ss_cdemo_sk BIGINT,
ss_hdemo_sk BIGINT,
ss_addr_sk BIGINT,
ss_store_sk BIGINT,
ss_promo_sk BIGINT,
ss_ticket_number BIGINT,
PRIMARY KEY(ss_ticket_number,ss_sold_date_sk,ss_sold_time_sk) NOT ENFORCED
) PARTITIONED BY (ss_sold_date_sk) WITH ('format-version'='2','write.upsert.enabled'='true');
insert into t1 select * from default_catalog.default_database.datagen_src;
then spark engine
ALTER TABLE t1 ADD PARTITION FIELD ss_sold_time_sk;
flink query
select * from t1;
exception：
org.apache.flink.table.api.TableException: Column 'ss_sold_time_sk' is NOT NULL, however, a null value is being written into it. You can set job configuration 'table.exec.sink.not-null-enforcer'='drop' to suppress this exception and drop such records silently.
at org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:56)
at org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:30)
at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:38)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6914: change partition led to query bug
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #6914:
URL: https://github.com/apache/iceberg/issues/6914#issuecomment-1443029409
It seems that you're inserting null values for `ss_sold_time_sk`, then making it a partition field and Flink errors about it (which is the correct thing to do). Flink even says that you can set `table.exec.sink.not-null-enforcer'='drop'` to silently drop those null values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6914: change partition led to query bug
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6914: change partition led to query bug
URL: https://github.com/apache/iceberg/issues/6914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ymZhao1001 commented on issue #6914: change partition led to query bug
Posted by "ymZhao1001 (via GitHub)" <gi...@apache.org>.
ymZhao1001 commented on issue #6914:
URL: https://github.com/apache/iceberg/issues/6914#issuecomment-1441125031
#6915 https://github.com/apache/iceberg/pull/6915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6914: change partition led to query bug
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6914:
URL: https://github.com/apache/iceberg/issues/6914#issuecomment-1712351531
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ymZhao1001 commented on issue #6914: change partition led to query bug
Posted by "ymZhao1001 (via GitHub)" <gi...@apache.org>.
ymZhao1001 commented on issue #6914:
URL: https://github.com/apache/iceberg/issues/6914#issuecomment-1443200882
@nastra in fact , there is no null values for ss_sold_time_sk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6914: change partition led to query bug
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6914:
URL: https://github.com/apache/iceberg/issues/6914#issuecomment-1690799948
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org