You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/29 06:32:15 UTC
[GitHub] [iceberg] hemanth-saal opened a new issue, #6502: Iceberg 1.1.0 - Spark Insert overwrite - Facing issue if Data frame columns order is not as same as order of columns in table.
hemanth-saal opened a new issue, #6502:
URL: https://github.com/apache/iceberg/issues/6502
### Feature Request / Improvement
Iceberg 1.0.0, spark.sql(INSERT OVERWRITE),if Data frame columns order is not as same as order of columns in table, it was working fine.
When we changed the iceberg to 1.1.0 , we are facing the issue. In 1.1.0, It appears, its not mapping data frame columns with tables columns with by name , instead it is mapping by order. Later i changed the order of columns as same as table, then it worked fine in 1.1.0.
Issue details: vendor_published_at( time stamp column) and source_name (string column) order is not same in DF and Table.
Spark DF -columns order - ['year', 'month', 'region_id', 'region_name', 'country_id', 'country_name', 'product_id', 'product_name', 'value', 'scenario_id', 'units', 'data_quality_id', 'data_quality_name', 'vendor_published_at', 'source_name', 'ingested_at']
Table -columns order - ['year', 'month', 'region_id', 'region_name', 'country_id', 'country_name', 'product_id', 'product_name', 'value', 'scenario_id', 'units', 'data_quality_id', 'data_quality_name','source_name', 'vendor_published_at', 'ingested_at']
spark.sql("Insert Overwrite Statement: INSERT OVERWRITE datawarehouse.<schema>.<table> SELECT * from new_data")
22/12/29 05:56:36 INFO BaseMetastoreCatalog: Table loaded by catalog: datawarehouse.<schema>.<table
22/12/29 05:56:36 ERROR Cannot write incompatible data to table 'datawarehouse.<schema>.<table':
- Cannot safely cast 'vendor_published_at': string to timestamp
-
### Query engine
Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6502: Iceberg 1.1.0 - Spark Insert overwrite - Facing issue if Data frame columns order is not as same as order of columns in table.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6502:
URL: https://github.com/apache/iceberg/issues/6502#issuecomment-1627872807
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6502: Iceberg 1.1.0 - Spark Insert overwrite - Facing issue if Data frame columns order is not as same as order of columns in table.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6502:
URL: https://github.com/apache/iceberg/issues/6502#issuecomment-1650732401
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] akshayjain3450 commented on issue #6502: Iceberg 1.1.0 - Spark Insert overwrite - Facing issue if Data frame columns order is not as same as order of columns in table.
Posted by GitBox <gi...@apache.org>.
akshayjain3450 commented on issue #6502:
URL: https://github.com/apache/iceberg/issues/6502#issuecomment-1376795819
Hi guys, is there any update regarding this issue for 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6502: Iceberg 1.1.0 - Spark Insert overwrite - Facing issue if Data frame columns order is not as same as order of columns in table.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6502: Iceberg 1.1.0 - Spark Insert overwrite - Facing issue if Data frame columns order is not as same as order of columns in table.
URL: https://github.com/apache/iceberg/issues/6502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org