You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2025/06/16 22:35:08 UTC
[PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
CTTY opened a new pull request, #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Related Issues:
- #1382 [EPIC]
## What changes are included in this PR?
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
- Implement `TransactionAction` for `FastAppendAction`
- Changed `SnapshotProduceAction` to `SnapshotProducer`
- The idea is to make `SnapshotProduceAction` an _immutable_ helper class (as discussed [here](https://github.com/apache/iceberg-rust/pull/1400#discussion_r2131851242))
## Are these changes tested?
Added unit tests
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2151569377
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -87,76 +169,45 @@ impl FastAppendAction {
/// Specifically, schema compatibility checks and support for adding to partitioned tables
/// have not yet been implemented.
#[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
+ async fn add_parquet_files(self, table: &Table, file_path: Vec<String>) -> Result<Self> {
Review Comment:
Yes that sounds good. We should make sure to make this clear in documentation and provide an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2151121670
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -87,76 +169,45 @@ impl FastAppendAction {
/// Specifically, schema compatibility checks and support for adding to partitioned tables
/// have not yet been implemented.
#[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
+ async fn add_parquet_files(self, table: &Table, file_path: Vec<String>) -> Result<Self> {
Review Comment:
Because `FastAppendAction` no longer holds `Transaction` or `Table`, we have to pass `&Table` to this method to convert parquet files to data files. I think 1) no transaction action API should allow users to pass `Table` in to keep APIs clear and simple, and 2) Transaction is probably a wrong place to do this file conversion
So I'm thinking of isolating the path-to-DataFile conversion logic out to make the API simpler. like this
```
let tx = Transaction::new()
let fast_append = tx.fast_append()
let data_files = to_data_files(vec![parquet_file_path]) // add a new helper to convert parquet files to DataFile
let tx = fast_append.add_data_files(data_files).apply()
tx.commit
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2151101515
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -87,76 +169,45 @@ impl FastAppendAction {
/// Specifically, schema compatibility checks and support for adding to partitioned tables
/// have not yet been implemented.
#[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
+ async fn add_parquet_files(self, table: &Table, file_path: Vec<String>) -> Result<Self> {
Review Comment:
It was originally implemented as such to allow the user to be able to call this function directly. Are you saying that we should batch data files then call `add_data_files` later all in one go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2151080053
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -87,76 +169,45 @@ impl FastAppendAction {
/// Specifically, schema compatibility checks and support for adding to partitioned tables
/// have not yet been implemented.
#[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
+ async fn add_parquet_files(self, table: &Table, file_path: Vec<String>) -> Result<Self> {
Review Comment:
I'm wondering if we actually need this as an API. I feel like it's cleaner to provide a helper method to convert parquet files to `DataFile`s, which can be passed to `add_data_files` directly later.
Would appreciate any input here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2162620617
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -62,101 +59,51 @@ impl FastAppendAction {
}
/// Add data files to the snapshot.
- pub fn add_data_files(
- &mut self,
- data_files: impl IntoIterator<Item = DataFile>,
- ) -> Result<&mut Self> {
- self.snapshot_produce_action.add_data_files(data_files)?;
- Ok(self)
+ pub fn add_data_files(mut self, data_files: impl IntoIterator<Item = DataFile>) -> Self {
+ self.added_data_files.extend(data_files);
+ self
}
- /// Set snapshot summary properties.
- pub fn set_snapshot_properties(
- &mut self,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<&mut Self> {
- self.snapshot_produce_action
- .set_snapshot_properties(snapshot_properties)?;
- Ok(self)
+ /// Set commit UUID for the snapshot.
+ pub fn set_commit_uuid(mut self, commit_uuid: Uuid) -> Self {
+ self.commit_uuid = Some(commit_uuid);
+ self
}
- /// Adds existing parquet files
- ///
- /// Note: This API is not yet fully supported in version 0.5.x.
- /// It is currently incomplete and should not be used in production.
- /// Specifically, schema compatibility checks and support for adding to partitioned tables
- /// have not yet been implemented.
- #[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
- return Err(Error::new(
- ErrorKind::FeatureUnsupported,
- "Appending to partitioned tables is not supported",
- ));
- }
-
- let table_metadata = self.snapshot_produce_action.tx.current_table.metadata();
-
- let data_files = ParquetWriter::parquet_files_to_data_files(
- self.snapshot_produce_action.tx.current_table.file_io(),
- file_path,
- table_metadata,
- )
- .await?;
-
- self.add_data_files(data_files)?;
+ /// Set key metadata for manifest files.
+ pub fn set_key_metadata(mut self, key_metadata: Vec<u8>) -> Self {
+ self.key_metadata = Some(key_metadata);
+ self
+ }
- self.apply().await
+ /// Set snapshot summary properties.
+ pub fn set_snapshot_properties(mut self, snapshot_properties: HashMap<String, String>) -> Self {
+ self.snapshot_properties = snapshot_properties;
+ self
}
+}
+
+#[async_trait]
+impl TransactionAction for FastAppendAction {
+ async fn commit(self: Arc<Self>, table: &Table) -> Result<ActionCommit> {
+ // validate added files
+ SnapshotProducer::validate_added_data_files(table, &self.added_data_files)?;
- /// Finished building the action and apply it to the transaction.
- pub async fn apply(self) -> Result<Transaction> {
Review Comment:
Thank you!
##########
crates/iceberg/src/spec/manifest/writer.rs:
##########
@@ -127,7 +127,7 @@ impl ManifestWriter {
pub(crate) fn new(
output: OutputFile,
snapshot_id: Option<i64>,
- key_metadata: Vec<u8>,
+ key_metadata: Option<Vec<u8>>,
Review Comment:
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2162616750
##########
crates/iceberg/src/spec/manifest/writer.rs:
##########
@@ -127,7 +127,7 @@ impl ManifestWriter {
pub(crate) fn new(
output: OutputFile,
snapshot_id: Option<i64>,
- key_metadata: Vec<u8>,
+ key_metadata: Option<Vec<u8>>,
Review Comment:
Sorry about that! I've updated the PR description.
cc @liurenjie1024 probably related to the release process: Is there a "staging release notes" that we can start putting changes like this into as breaking changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2160577732
##########
crates/iceberg/src/spec/manifest/writer.rs:
##########
@@ -127,7 +127,7 @@ impl ManifestWriter {
pub(crate) fn new(
output: OutputFile,
snapshot_id: Option<i64>,
- key_metadata: Vec<u8>,
+ key_metadata: Option<Vec<u8>>,
Review Comment:
This is an API change, would appreciate if we could call it out at PR description and release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2162823355
##########
crates/iceberg/src/spec/manifest/writer.rs:
##########
@@ -127,7 +127,7 @@ impl ManifestWriter {
pub(crate) fn new(
output: OutputFile,
snapshot_id: Option<i64>,
- key_metadata: Vec<u8>,
+ key_metadata: Option<Vec<u8>>,
Review Comment:
I'm thinking about adding a `breaking` label to pr so that it standout when we generate release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2153527451
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -16,43 +16,132 @@
// under the License.
use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use async_trait::async_trait;
use uuid::Uuid;
use crate::error::Result;
-use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation};
-use crate::transaction::Transaction;
+use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation, Struct, StructType};
+use crate::table::Table;
use crate::transaction::snapshot::{
- DefaultManifestProcess, SnapshotProduceAction, SnapshotProduceOperation,
+ DefaultManifestProcess, SnapshotProduceOperation, SnapshotProducer,
};
+use crate::transaction::{ActionCommit, TransactionAction};
use crate::writer::file_writer::ParquetWriter;
use crate::{Error, ErrorKind};
/// FastAppendAction is a transaction action for fast append data files to the table.
pub struct FastAppendAction {
- snapshot_produce_action: SnapshotProduceAction,
check_duplicate: bool,
+ // below are properties used to create SnapshotProducer when commit
+ snapshot_id: i64,
+ commit_uuid: Uuid,
+ key_metadata: Vec<u8>,
+ snapshot_properties: HashMap<String, String>,
+ added_data_files: Vec<DataFile>,
}
impl FastAppendAction {
- #[allow(clippy::too_many_arguments)]
- pub(crate) fn new(
- tx: Transaction,
- snapshot_id: i64,
- commit_uuid: Uuid,
- key_metadata: Vec<u8>,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<Self> {
- Ok(Self {
- snapshot_produce_action: SnapshotProduceAction::new(
- tx,
- snapshot_id,
- key_metadata,
- commit_uuid,
- snapshot_properties,
- )?,
+ pub(crate) fn new(snapshot_id: i64, commit_uuid: Uuid, key_metadata: Vec<u8>) -> Self {
Review Comment:
I think an `Option` has clearer semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2153519378
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -16,43 +16,132 @@
// under the License.
use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use async_trait::async_trait;
use uuid::Uuid;
use crate::error::Result;
-use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation};
-use crate::transaction::Transaction;
+use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation, Struct, StructType};
+use crate::table::Table;
use crate::transaction::snapshot::{
- DefaultManifestProcess, SnapshotProduceAction, SnapshotProduceOperation,
+ DefaultManifestProcess, SnapshotProduceOperation, SnapshotProducer,
};
+use crate::transaction::{ActionCommit, TransactionAction};
use crate::writer::file_writer::ParquetWriter;
use crate::{Error, ErrorKind};
/// FastAppendAction is a transaction action for fast append data files to the table.
pub struct FastAppendAction {
- snapshot_produce_action: SnapshotProduceAction,
check_duplicate: bool,
+ // below are properties used to create SnapshotProducer when commit
+ snapshot_id: i64,
+ commit_uuid: Uuid,
+ key_metadata: Vec<u8>,
+ snapshot_properties: HashMap<String, String>,
+ added_data_files: Vec<DataFile>,
}
impl FastAppendAction {
- #[allow(clippy::too_many_arguments)]
- pub(crate) fn new(
- tx: Transaction,
- snapshot_id: i64,
- commit_uuid: Uuid,
- key_metadata: Vec<u8>,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<Self> {
- Ok(Self {
- snapshot_produce_action: SnapshotProduceAction::new(
- tx,
- snapshot_id,
- key_metadata,
- commit_uuid,
- snapshot_properties,
- )?,
+ pub(crate) fn new(snapshot_id: i64, commit_uuid: Uuid, key_metadata: Vec<u8>) -> Self {
Review Comment:
I agree, I think something like this makes more sense:
```
pub fn fast_append(&self) -> FastAppendAction {
FastAppendAction::new(self.generate_unique_snapshot_id())
}
---
tx.fast_append().set_commit_uuid(...).set_key_metadata(...)
```
I also believe `key_metadata` should be an `Option`? In the existing code it seems that it's required, even though you can provide an empty vector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2153519378
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -16,43 +16,132 @@
// under the License.
use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use async_trait::async_trait;
use uuid::Uuid;
use crate::error::Result;
-use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation};
-use crate::transaction::Transaction;
+use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation, Struct, StructType};
+use crate::table::Table;
use crate::transaction::snapshot::{
- DefaultManifestProcess, SnapshotProduceAction, SnapshotProduceOperation,
+ DefaultManifestProcess, SnapshotProduceOperation, SnapshotProducer,
};
+use crate::transaction::{ActionCommit, TransactionAction};
use crate::writer::file_writer::ParquetWriter;
use crate::{Error, ErrorKind};
/// FastAppendAction is a transaction action for fast append data files to the table.
pub struct FastAppendAction {
- snapshot_produce_action: SnapshotProduceAction,
check_duplicate: bool,
+ // below are properties used to create SnapshotProducer when commit
+ snapshot_id: i64,
+ commit_uuid: Uuid,
+ key_metadata: Vec<u8>,
+ snapshot_properties: HashMap<String, String>,
+ added_data_files: Vec<DataFile>,
}
impl FastAppendAction {
- #[allow(clippy::too_many_arguments)]
- pub(crate) fn new(
- tx: Transaction,
- snapshot_id: i64,
- commit_uuid: Uuid,
- key_metadata: Vec<u8>,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<Self> {
- Ok(Self {
- snapshot_produce_action: SnapshotProduceAction::new(
- tx,
- snapshot_id,
- key_metadata,
- commit_uuid,
- snapshot_properties,
- )?,
+ pub(crate) fn new(snapshot_id: i64, commit_uuid: Uuid, key_metadata: Vec<u8>) -> Self {
Review Comment:
I agree, something like this makes more sense:
```
pub fn fast_append(&self) -> FastAppendAction {
FastAppendAction::new(self.generate_unique_snapshot_id())
}
---
tx.fast_append().set_commit_uuid(...).set_key_metadata(...)
```
I also believe `key_metadata` should be an `Option`? In the existing code it seems that it's required, even though you can provide an empty vector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2162617409
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -62,101 +59,51 @@ impl FastAppendAction {
}
/// Add data files to the snapshot.
- pub fn add_data_files(
- &mut self,
- data_files: impl IntoIterator<Item = DataFile>,
- ) -> Result<&mut Self> {
- self.snapshot_produce_action.add_data_files(data_files)?;
- Ok(self)
+ pub fn add_data_files(mut self, data_files: impl IntoIterator<Item = DataFile>) -> Self {
+ self.added_data_files.extend(data_files);
+ self
}
- /// Set snapshot summary properties.
- pub fn set_snapshot_properties(
- &mut self,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<&mut Self> {
- self.snapshot_produce_action
- .set_snapshot_properties(snapshot_properties)?;
- Ok(self)
+ /// Set commit UUID for the snapshot.
+ pub fn set_commit_uuid(mut self, commit_uuid: Uuid) -> Self {
+ self.commit_uuid = Some(commit_uuid);
+ self
}
- /// Adds existing parquet files
- ///
- /// Note: This API is not yet fully supported in version 0.5.x.
- /// It is currently incomplete and should not be used in production.
- /// Specifically, schema compatibility checks and support for adding to partitioned tables
- /// have not yet been implemented.
- #[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
- return Err(Error::new(
- ErrorKind::FeatureUnsupported,
- "Appending to partitioned tables is not supported",
- ));
- }
-
- let table_metadata = self.snapshot_produce_action.tx.current_table.metadata();
-
- let data_files = ParquetWriter::parquet_files_to_data_files(
- self.snapshot_produce_action.tx.current_table.file_io(),
- file_path,
- table_metadata,
- )
- .await?;
-
- self.add_data_files(data_files)?;
+ /// Set key metadata for manifest files.
+ pub fn set_key_metadata(mut self, key_metadata: Vec<u8>) -> Self {
+ self.key_metadata = Some(key_metadata);
+ self
+ }
- self.apply().await
+ /// Set snapshot summary properties.
+ pub fn set_snapshot_properties(mut self, snapshot_properties: HashMap<String, String>) -> Self {
+ self.snapshot_properties = snapshot_properties;
+ self
}
+}
+
+#[async_trait]
+impl TransactionAction for FastAppendAction {
+ async fn commit(self: Arc<Self>, table: &Table) -> Result<ActionCommit> {
+ // validate added files
+ SnapshotProducer::validate_added_data_files(table, &self.added_data_files)?;
- /// Finished building the action and apply it to the transaction.
- pub async fn apply(self) -> Result<Transaction> {
Review Comment:
Updated the PR description accordingly as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2151820818
##########
crates/iceberg/src/transaction/mod.rs:
##########
@@ -143,17 +142,15 @@ impl Transaction {
/// Creates a fast append action.
pub fn fast_append(
- self,
+ &self,
Review Comment:
How about removing these two parameters: `commit_uuid`, `key_metadata`
##########
crates/iceberg/src/transaction/snapshot.rs:
##########
@@ -59,126 +60,50 @@ pub(crate) trait ManifestProcess: Send + Sync {
fn process_manifests(&self, manifests: Vec<ManifestFile>) -> Vec<ManifestFile>;
}
-pub(crate) struct SnapshotProduceAction {
- pub tx: Transaction,
+pub(crate) struct SnapshotProducer {
snapshot_id: i64,
- key_metadata: Vec<u8>,
commit_uuid: Uuid,
+ key_metadata: Vec<u8>,
snapshot_properties: HashMap<String, String>,
- pub added_data_files: Vec<DataFile>,
+ added_data_files: Vec<DataFile>,
// A counter used to generate unique manifest file names.
// It starts from 0 and increments for each new manifest file.
// Note: This counter is limited to the range of (0..u64::MAX).
manifest_counter: RangeFrom<u64>,
}
-impl SnapshotProduceAction {
+impl SnapshotProducer {
pub(crate) fn new(
- tx: Transaction,
snapshot_id: i64,
- key_metadata: Vec<u8>,
commit_uuid: Uuid,
+ key_metadata: Vec<u8>,
snapshot_properties: HashMap<String, String>,
- ) -> Result<Self> {
- Ok(Self {
- tx,
+ added_data_files: Vec<DataFile>,
+ ) -> Self {
+ Self {
snapshot_id,
commit_uuid,
+ key_metadata,
snapshot_properties,
- added_data_files: vec![],
+ added_data_files,
manifest_counter: (0..),
- key_metadata,
- })
- }
-
- // Check if the partition value is compatible with the partition type.
- fn validate_partition_value(
Review Comment:
We should not move these methods to `FastAppendAction`, they will be shared by other actions like row delta.
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -87,76 +169,45 @@ impl FastAppendAction {
/// Specifically, schema compatibility checks and support for adding to partitioned tables
/// have not yet been implemented.
#[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
+ async fn add_parquet_files(self, table: &Table, file_path: Vec<String>) -> Result<Self> {
Review Comment:
+1, I think we could remove this method for now. This is in fact an incomplete api, which doesn't support partitioned table for now.
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -16,43 +16,132 @@
// under the License.
use std::collections::{HashMap, HashSet};
+use std::sync::Arc;
+use async_trait::async_trait;
use uuid::Uuid;
use crate::error::Result;
-use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation};
-use crate::transaction::Transaction;
+use crate::spec::{DataFile, ManifestEntry, ManifestFile, Operation, Struct, StructType};
+use crate::table::Table;
use crate::transaction::snapshot::{
- DefaultManifestProcess, SnapshotProduceAction, SnapshotProduceOperation,
+ DefaultManifestProcess, SnapshotProduceOperation, SnapshotProducer,
};
+use crate::transaction::{ActionCommit, TransactionAction};
use crate::writer::file_writer::ParquetWriter;
use crate::{Error, ErrorKind};
/// FastAppendAction is a transaction action for fast append data files to the table.
pub struct FastAppendAction {
- snapshot_produce_action: SnapshotProduceAction,
check_duplicate: bool,
+ // below are properties used to create SnapshotProducer when commit
+ snapshot_id: i64,
+ commit_uuid: Uuid,
+ key_metadata: Vec<u8>,
+ snapshot_properties: HashMap<String, String>,
+ added_data_files: Vec<DataFile>,
}
impl FastAppendAction {
- #[allow(clippy::too_many_arguments)]
- pub(crate) fn new(
- tx: Transaction,
- snapshot_id: i64,
- commit_uuid: Uuid,
- key_metadata: Vec<u8>,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<Self> {
- Ok(Self {
- snapshot_produce_action: SnapshotProduceAction::new(
- tx,
- snapshot_id,
- key_metadata,
- commit_uuid,
- snapshot_properties,
- )?,
+ pub(crate) fn new(snapshot_id: i64, commit_uuid: Uuid, key_metadata: Vec<u8>) -> Self {
Review Comment:
I'm thinking maybe we don't need a constructor? With the new transaction api, now the action class is more like a builder, so these parameters all could be passed by user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(transaction): Implement TransactionAction for FastAppendAction [iceberg-rust]
Posted by "dentiny (via GitHub)" <gi...@apache.org>.
dentiny commented on code in PR #1448:
URL: https://github.com/apache/iceberg-rust/pull/1448#discussion_r2160578316
##########
crates/iceberg/src/transaction/append.rs:
##########
@@ -62,101 +59,51 @@ impl FastAppendAction {
}
/// Add data files to the snapshot.
- pub fn add_data_files(
- &mut self,
- data_files: impl IntoIterator<Item = DataFile>,
- ) -> Result<&mut Self> {
- self.snapshot_produce_action.add_data_files(data_files)?;
- Ok(self)
+ pub fn add_data_files(mut self, data_files: impl IntoIterator<Item = DataFile>) -> Self {
+ self.added_data_files.extend(data_files);
+ self
}
- /// Set snapshot summary properties.
- pub fn set_snapshot_properties(
- &mut self,
- snapshot_properties: HashMap<String, String>,
- ) -> Result<&mut Self> {
- self.snapshot_produce_action
- .set_snapshot_properties(snapshot_properties)?;
- Ok(self)
+ /// Set commit UUID for the snapshot.
+ pub fn set_commit_uuid(mut self, commit_uuid: Uuid) -> Self {
+ self.commit_uuid = Some(commit_uuid);
+ self
}
- /// Adds existing parquet files
- ///
- /// Note: This API is not yet fully supported in version 0.5.x.
- /// It is currently incomplete and should not be used in production.
- /// Specifically, schema compatibility checks and support for adding to partitioned tables
- /// have not yet been implemented.
- #[allow(dead_code)]
- async fn add_parquet_files(mut self, file_path: Vec<String>) -> Result<Transaction> {
- if !self
- .snapshot_produce_action
- .tx
- .current_table
- .metadata()
- .default_spec
- .is_unpartitioned()
- {
- return Err(Error::new(
- ErrorKind::FeatureUnsupported,
- "Appending to partitioned tables is not supported",
- ));
- }
-
- let table_metadata = self.snapshot_produce_action.tx.current_table.metadata();
-
- let data_files = ParquetWriter::parquet_files_to_data_files(
- self.snapshot_produce_action.tx.current_table.file_io(),
- file_path,
- table_metadata,
- )
- .await?;
-
- self.add_data_files(data_files)?;
+ /// Set key metadata for manifest files.
+ pub fn set_key_metadata(mut self, key_metadata: Vec<u8>) -> Self {
+ self.key_metadata = Some(key_metadata);
+ self
+ }
- self.apply().await
+ /// Set snapshot summary properties.
+ pub fn set_snapshot_properties(mut self, snapshot_properties: HashMap<String, String>) -> Self {
+ self.snapshot_properties = snapshot_properties;
+ self
}
+}
+
+#[async_trait]
+impl TransactionAction for FastAppendAction {
+ async fn commit(self: Arc<Self>, table: &Table) -> Result<ActionCommit> {
+ // validate added files
+ SnapshotProducer::validate_added_data_files(table, &self.added_data_files)?;
- /// Finished building the action and apply it to the transaction.
- pub async fn apply(self) -> Result<Transaction> {
Review Comment:
This is an API change, would appreciate PR description and release notes, even provide a small example for no-op migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org