You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "saimigo (via GitHub)" <gi...@apache.org> on 2023/05/08 08:57:53 UTC
[GitHub] [iceberg] saimigo opened a new issue, #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
saimigo opened a new issue, #7554:
URL: https://github.com/apache/iceberg/issues/7554
### Query engine
Spark
### Question
Assuming that there are two rows of data as follows, we think that these two data are the same data, so how to avoid duplication of data in the Iceberg table when using spark to write data？
("2023-04-25 08:00:00.000000", "Name 1", "Note 1")
("2023-04-25 08:00:00.000000", “Name 1", "Note 2")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1538012396
@saimigo Are you appending the data, or overwriting? In the case of the first, you can call [`.dropDuplicates` using Spark](https://spark.apache.org/docs/3.1.2/api/python/reference/api/pyspark.sql.DataFrame.dropDuplicates.html). In the case of incremental data, you need to perform a [merge operation](https://iceberg.apache.org/docs/latest/spark-writes/#merge-into).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to ensure the data is not repeated when using spark to write to the iceberg table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1802943769
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to ensure the data is not repeated when using spark to write to the iceberg table [iceberg]
Posted by "sinban04 (via GitHub)" <gi...@apache.org>.
sinban04 commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-2795618433
It seems MERGE INTO is supported as Spark DataFrame v2 API since Spark 4.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jhchee commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "jhchee (via GitHub)" <gi...@apache.org>.
jhchee commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1538358229
@saimigo Merge into command is executed based on certain condition. A common use case is to use it for `UPSERT` operation. E.g.
```sql
MERGE INTO target t -- a target table
USING source s -- your source (can be a table)
ON t.id = s.id -- condition to find updates for target rows (can be your unique identifiers)
WHEN MATCHED UPDATE *
WHEN NOT MATCHED INSERT *
```
To address few questions,
1. You don't have to delete the table.
2. Be in mind, Spark engine will perform join based on condition you define. This means it requires to read from a table to find the matching row, so the target table must exist in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] saimigo commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "saimigo (via GitHub)" <gi...@apache.org>.
saimigo commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1538033546
> @saimigo Are you appending the data, or overwriting? In the case of the first, you can call [`.dropDuplicates` using Spark](https://spark.apache.org/docs/3.1.2/api/python/reference/api/pyspark.sql.DataFrame.dropDuplicates.html). In the case of incremental data, you need to perform a [merge operation](https://iceberg.apache.org/docs/latest/spark-writes/#merge-into).
Appending the data, If I use the merge into operation, do I need to create a new table every time, and then delete this table after merging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1544923713
I am not sure if merge into can be done using dataframe API, same with deletion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] saimigo commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "saimigo (via GitHub)" <gi...@apache.org>.
saimigo commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1539324853
> @saimigo Merge into command is executed based on certain condition. A common use case is to use it for `UPSERT` operation. E.g.
>
> ```sql
> MERGE INTO target t -- a target table
> USING source s -- your source (can be a table)
> ON t.id = s.id -- condition to find updates for target rows (can be your unique identifiers)
> WHEN MATCHED UPDATE *
> WHEN NOT MATCHED INSERT *
> ```
>
> To address few questions,
>
> 1. You can to guarantee uniqueness of your data by dropping duplicates in the source's Spark Dataframe or by doing UPSERT against the target table.
> 2. You don't have to delete the table when doing UPSERT.
> 3. Be in mind, Spark engine will perform join based on condition you define. This means it requires to read from a table to find the matching row, so the target table must exist in the first place.
Please, Is there any example or document of dataframe api related to merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to ensure the data is not repeated when using spark to write to the iceberg table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1823681773
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] saimigo commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "saimigo (via GitHub)" <gi...@apache.org>.
saimigo commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1539248767
> @saimigo Merge into command is executed based on certain condition. A common use case is to use it for `UPSERT` operation. E.g.
>
> ```sql
> MERGE INTO target t -- a target table
> USING source s -- your source (can be a table)
> ON t.id = s.id -- condition to find updates for target rows (can be your unique identifiers)
> WHEN MATCHED UPDATE *
> WHEN NOT MATCHED INSERT *
> ```
>
> To address few questions,
>
> 1. You can to guarantee uniqueness of your data by dropping duplicates in the source's Spark Dataframe or by doing UPSERT against the target table.
> 2. You don't have to delete the table when doing UPSERT.
> 3. Be in mind, Spark engine will perform join based on condition you define. This means it requires to read from a table to find the matching row, so the target table must exist in the first place.
Thank you very much, I need to learn more about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jhchee commented on issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
Posted by "jhchee (via GitHub)" <gi...@apache.org>.
jhchee commented on issue #7554:
URL: https://github.com/apache/iceberg/issues/7554#issuecomment-1540219979
It is usually run as a Spark SQL. There is an example from AWS
https://github.com/aws-samples/emr-on-eks-hudi-iceberg-delta/blob/main/iceberg/iceberg_scd_script.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to ensure the data is not repeated when using spark to write to the iceberg table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7554: How to ensure the data is not repeated when using spark to write to the iceberg table
URL: https://github.com/apache/iceberg/issues/7554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org