You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "leekeiabstraction (via GitHub)" <gi...@apache.org> on 2026/01/13 13:13:52 UTC
[PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
leekeiabstraction opened a new pull request, #158:
URL: https://github.com/apache/fluss-rust/pull/158
Related: #66
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690218129
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
Why do we return Option if we have default and unwrap?
Should it match get_datalake_format logic or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690218129
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
Why do we return Option if we have default and unwrap?
Should it match get_datalake_format logic or not?
Would smth like this work:
```rust
const DEFAULT: &str = "COMPACTED";
let raw = self.properties.get("table.kv.format").map(String::as_str).unwrap_or(DEFAULT);
raw.parse().map_err(Into::into)
```
Then the type would be `Result<KvFormat>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #158:
URL: https://github.com/apache/fluss-rust/pull/158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690218129
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
Why do we return Option if we have default and unwrap?
Should it match get_datalake_format logic or not?
Would smth like this work:
```rust
const DEFAULT: &str = "COMPACTED";
let raw = self.properties.get("table.kv.format").map(String::as_str).unwrap_or(DEFAULT);
raw.parse().map_err(Into::into)
```
Then the type would be Result<KvFormat>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690240218
##########
crates/fluss/src/client/write/write_format.rs:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::error::Error::IllegalArgument;
+use crate::error::Result;
+use crate::metadata::KvFormat;
+use std::fmt::Display;
+
+pub enum WriteFormat {
+ ArrowLog,
+ IndexedLog,
+ CompactedLog,
+ IndexedKv,
+ CompactedKv,
+}
+
+impl WriteFormat {
+ pub fn is_log(&self) -> bool {
Review Comment:
nit:
```rust
pub const fn is_log(self) -> bool {
matches!(self, Self::ArrowLog | Self::IndexedLog | Self::CompactedLog)
}
```
##########
crates/fluss/src/metadata/datatype.rs:
##########
@@ -857,6 +859,24 @@ impl RowType {
self.fields.iter().position(|f| f.name == field_name)
}
+ pub fn get_field_names(&self) -> Vec<&str> {
+ self.fields.iter().map(|f| f.name.as_str()).collect()
+ }
+
+ pub fn project(&self, project_field_positions: &[usize]) -> Result<RowType> {
+ Ok(RowType::with_nullable(
+ self.nullable,
+ project_field_positions
+ .iter()
+ .map(|pos| {
+ self.fields.get(*pos).cloned().ok_or(IllegalArgument {
Review Comment:
ok_or_else just to build error lazily and avoid format! for successful cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#issuecomment-3744304888
@luoyuxia @fresh-borzoni @Kelvinyu1117
Would appreciate review here. I'm breaking up TableUpsert implementation PR into smaller bits so that it is more easily reviewable.
Will follow up with CompactedRowEncoder PR before UpsertTable PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#issuecomment-3749476562
@fresh-borzoni Thank you for your review! Addressed your comments. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690218129
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
Why do we return Optional if we have default and unwrap?
Should it match get_datalake_format logic or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2692881586
##########
crates/fluss/src/client/write/write_format.rs:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::error::Error::IllegalArgument;
+use crate::error::Result;
+use crate::metadata::KvFormat;
+use std::fmt::Display;
+
+pub enum WriteFormat {
Review Comment:
nit:
only include `ArrowLog`, `CompactedLog`, `CompactedKv`.
Indexedxx is deperated and will be drop in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690461782
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
> Why do we return Option if we have default and unwrap?
Great question. I misread Java side implementation and thought that KvFormat is nullable.
Thank you for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2689743107
##########
crates/fluss/src/metadata/datatype.rs:
##########
@@ -857,6 +857,20 @@ impl RowType {
self.fields.iter().position(|f| f.name == field_name)
}
+ pub fn get_field_names(&self) -> Vec<&str> {
+ self.fields.iter().map(|f| f.name.as_str()).collect()
+ }
+
+ pub fn project(&self, project_field_positions: &[usize]) -> RowType {
+ RowType::with_nullable(
+ self.nullable,
+ project_field_positions
+ .iter()
+ .map(|pos| self.fields.get(*pos).unwrap().clone())
Review Comment:
The use of `unwrap()` will panic if any position in `project_field_positions` is out of bounds for `self.fields`. Consider using a Result return type and proper error handling instead, or add documentation stating that this method assumes valid positions and will panic otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690387139
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
@leekeiabstraction this one as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#issuecomment-3753451588
@luoyuxia Addressed your comment. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task: WriteFormat and various small changes to unblock Upsert implementation [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #158:
URL: https://github.com/apache/fluss-rust/pull/158#discussion_r2690218129
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
Why do we return Option if we have default and unwrap?
Would smth like this work:
```rust
const DEFAULT: &str = "COMPACTED";
let raw = self.properties.get("table.kv.format").map(String::as_str).unwrap_or(DEFAULT);
raw.parse().map_err(Into::into)
```
Then the type would be `Result<KvFormat>`
##########
crates/fluss/src/metadata/table.rs:
##########
@@ -726,6 +728,22 @@ impl TableConfig {
pub fn get_arrow_compression_info(&self) -> Result<ArrowCompressionInfo> {
ArrowCompressionInfo::from_conf(&self.properties)
}
+
+ pub fn get_datalake_format(&self) -> Result<Option<DataLakeFormat>> {
+ self.properties
+ .get("table.datalake.format")
+ .map(|f| f.parse().map_err(Error::from))
+ .transpose()
+ }
+
+ pub fn get_kv_format(&self) -> Result<Option<KvFormat>> {
Review Comment:
Why do we return Option if we have default and unwrap?
Would smth like this work here?
```rust
const DEFAULT: &str = "COMPACTED";
let raw = self.properties.get("table.kv.format").map(String::as_str).unwrap_or(DEFAULT);
raw.parse().map_err(Into::into)
```
Then the type would be `Result<KvFormat>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org