You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/09/02 18:10:03 UTC
[GitHub] [iceberg] flyrain opened a new issue #3064: `AppendFile` can add duplicated data files
flyrain opened a new issue #3064:
URL: https://github.com/apache/iceberg/issues/3064
The following Java APIs add a duplicated data file object(newDataFile1)
```
Transaction t = table.newTransaction();
t.newAppend().appendFile(newDataFile1).appendFile(newDataFile1).commit();
```
The manifest file will have two identical data file items after the commit. Also, any row in the data file will be doubled while reading back from the table.
Looks like a bug. Can we fix it in core? Two classes are involved `FastAppend` and `MergeAppend`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer edited a comment on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-911936631
@szehon-ho This is similar to the import data files fix that you added, I'm wondering if we should do the same thing here, or just always outlaw this ...
My only worry about always disabling this in the underlying code is to actually prevent duplicates we need to read the entire table datafiles listing before comitting. This could be expensive on a large table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-912087388
Yea it makes sense to just add the documentation , if it adds little value to do a whole spark job. If we really want to do something, we can do a quick in memory comparison on the Merge/FastAppend?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-911936631
@szehon-ho This is similar to the import data files fix that you added, I'm wondering if we should do the same thing here, or just always outlaw this ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho edited a comment on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
szehon-ho edited a comment on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-912087388
Yea it makes sense to just add the documentation , if it adds little value to do a whole spark job. If we really want to do something simple, we can do a quick in memory comparison on the Merge/FastAppend, but doesnt seem that worth it either?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho edited a comment on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
szehon-ho edited a comment on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-912087388
Yea it makes sense to just add the documentation , if it adds little value to do a whole spark job. If we want to do something simple, we can do a quick in memory comparison on the Merge/FastAppend to check for any duplicates within the transaction, but doesnt seem that worth it either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
flyrain commented on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-911945208
It is not a big deal if the compute engine like Spark has done extra work to remove the duplication. However, the APIs are considered public Java APIs, https://iceberg.apache.org/api/#transactions. It would be nice to let user be aware of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho edited a comment on issue #3064: `AppendFile` can add duplicated data files
Posted by GitBox <gi...@apache.org>.
szehon-ho edited a comment on issue #3064:
URL: https://github.com/apache/iceberg/issues/3064#issuecomment-912087388
Yea it makes sense to just add the documentation , if it adds little value to do a whole spark job. If we really want to do something simple, we can do a quick in memory comparison on the Merge/FastAppend?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org