You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "fresh-borzoni (via GitHub)" <gi...@apache.org> on 2026/01/11 18:44:36 UTC
[PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
fresh-borzoni opened a new pull request, #151:
URL: https://github.com/apache/fluss-rust/pull/151
Adds two new methods for column projection:
1. Table.new_log_scanner_with_projection(column_indices: List[int])
- Project columns by index (C++ parity)
- Example: scanner = table.new_log_scanner_with_projection([0, 2, 4])
2. Table.new_log_scanner_with_column_names(column_names: List[str])
- Project columns by name (Python-specific, more idiomatic!)
- Example: scanner = table.new_log_scanner_with_column_names(['id', 'name', 'email'])
Both methods create LogScanner with specified columns only, improving performance by reducing data transfer and processing overhead.
Implementation leverages core scanner.project() and scanner.project_by_name() APIs.
Error handling validates column indices/names before creating scanner.
Closes #149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#issuecomment-3736126508
@leekeiabstraction Thank you for the review.
Addressed the comments. PTAL 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680424455
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
I'm not sure I get what you mean. to_pandas polls internally, but if we are talking about adding separate polling API to python bindings - we can add it. Let's file an issue for it, as it's orthogonal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#issuecomment-3755550975
Changed to unified interface, added silent deduplication and more idiomatic python API with keywords
Updated PR description
@luoyuxia @leekeiabstraction PTAL 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2694732646
##########
bindings/python/src/table.rs:
##########
@@ -70,9 +70,112 @@ impl FlussTable {
let table_scan = fluss_table.new_scan();
let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(format!(
- "Failed to create log scanner: {e:?}"
- ))
+ FlussError::new_err(format!("Failed to create log scanner: {e}"))
+ })?;
+
+ let admin = conn
+ .get_admin()
+ .await
+ .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+ let py_scanner = LogScanner::from_core(rust_scanner, admin, table_info.clone());
+ Python::attach(|py| Py::new(py, py_scanner))
+ })
+ }
+
+ /// Create a new log scanner with column projection (by index).
+ ///
+ /// Args:
+ /// column_indices: List of column indices to include in the scan (0-based)
+ ///
+ /// Returns:
+ /// LogScanner with projection applied
+ ///
+ /// Example:
+ /// >>> scanner = await table.new_log_scanner_with_projection([0, 2, 4])
+ pub fn new_log_scanner_with_projection<'py>(
+ &self,
+ py: Python<'py>,
+ column_indices: Vec<usize>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ // Validate early with Python-friendly error
+ if column_indices.is_empty() {
+ return Err(FlussError::new_err(
+ "column_indices cannot be empty".to_string(),
+ ));
+ }
+
+ let conn = self.connection.clone();
+ let metadata = self.metadata.clone();
+ let table_info = self.table_info.clone();
+
+ future_into_py(py, async move {
+ let fluss_table =
+ fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+ let table_scan = fluss_table.new_scan();
+ let table_scan = table_scan
+ .project(&column_indices)
+ .map_err(|e| FlussError::new_err(format!("Failed to project columns: {e}")))?;
+
+ let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
+ FlussError::new_err(format!("Failed to create log scanner: {e}"))
+ })?;
+
+ let admin = conn
+ .get_admin()
+ .await
+ .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+ let py_scanner = LogScanner::from_core(rust_scanner, admin, table_info.clone());
+ Python::attach(|py| Py::new(py, py_scanner))
+ })
Review Comment:
There is significant code duplication between `new_log_scanner`, `new_log_scanner_with_projection`, and `new_log_scanner_with_column_names`. The three methods share nearly identical boilerplate code for creating the FlussTable, getting the admin, and constructing the LogScanner. Consider extracting a private helper function that accepts an optional projection configuration to reduce duplication and improve maintainability.
```suggestion
Self::create_log_scanner_internal(conn, metadata, table_info, Some(column_indices))
.await
})
}
/// Internal helper to create a log scanner with an optional column projection.
///
/// This centralizes the boilerplate for constructing the FlussTable, creating
/// the scan, applying an optional projection, and wrapping the resulting
/// scanner for Python.
async fn create_log_scanner_internal(
conn: Arc<fcore::client::FlussConnection>,
metadata: Arc<fcore::client::Metadata>,
table_info: fcore::metadata::TableInfo,
column_indices: Option<Vec<usize>>,
) -> PyResult<Py<PyAny>> {
let fluss_table =
fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
let table_scan = fluss_table.new_scan();
let table_scan = if let Some(indices) = column_indices {
table_scan
.project(&indices)
.map_err(|e| FlussError::new_err(format!("Failed to project columns: {e}")))?
} else {
table_scan
};
let rust_scanner = table_scan
.create_log_scanner()
.map_err(|e| FlussError::new_err(format!("Failed to create log scanner: {e}")))?;
let admin = conn
.get_admin()
.await
.map_err(|e| FlussError::new_err(e.to_string()))?;
let py_scanner = LogScanner::from_core(rust_scanner, admin, table_info.clone());
Python::attach(|py| Py::new(py, py_scanner))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680176217
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
Should the polling part also be included (as with C++ example)?
##########
bindings/python/src/table.rs:
##########
@@ -85,6 +85,111 @@ impl FlussTable {
})
}
+ /// Create a new log scanner with column projection (by index).
+ ///
+ /// Args:
+ /// column_indices: List of column indices to include in the scan (0-based)
+ ///
+ /// Returns:
+ /// LogScanner with projection applied
+ ///
+ /// Example:
+ /// >>> scanner = await table.new_log_scanner_with_projection([0, 2, 4])
+ pub fn new_log_scanner_with_projection<'py>(
+ &self,
+ py: Python<'py>,
+ column_indices: Vec<usize>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ // Validate early with Python-friendly error
+ if column_indices.is_empty() {
+ return Err(FlussError::new_err(
+ "column_indices cannot be empty".to_string(),
+ ));
+ }
+
+ let conn = self.connection.clone();
+ let metadata = self.metadata.clone();
+ let table_info = self.table_info.clone();
+
+ future_into_py(py, async move {
+ let fluss_table =
+ fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+ let table_scan = fluss_table.new_scan();
+ let table_scan = table_scan
+ .project(&column_indices)
+ .map_err(|e| FlussError::new_err(format!("Failed to project columns: {e}")))?;
+
+ let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
+ FlussError::new_err(format!("Failed to create log scanner: {e}"))
+ })?;
Review Comment:
We seem to use FlussError and PyErr within this file, for example line 72 to 75 uses PyErr. Can you clarify when each should be used?
```
let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(format!(
"Failed to create log scanner: {e:?}"
))
})?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680424455
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
I'm not sure I get what you mean. to_pandas polls internally, but if we are talking about adding separate polling API to python bindings - we can add it. Let's file an issue for it, as it's orthogonal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #151:
URL: https://github.com/apache/fluss-rust/pull/151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2696602045
##########
bindings/python/src/table.rs:
##########
@@ -126,6 +157,102 @@ impl FlussTable {
has_primary_key,
}
}
+
+ /// Validate and deduplicate column indices (preserving order)
+ fn validate_and_dedupe_indices(&self, indices: &[usize]) -> PyResult<Vec<usize>> {
Review Comment:
@luoyuxia No worries :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2694994437
##########
bindings/python/src/table.rs:
##########
@@ -57,32 +63,64 @@ impl FlussTable {
})
}
- /// Create a new log scanner for the table
- fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
- let conn = self.connection.clone();
- let metadata = self.metadata.clone();
- let table_info = self.table_info.clone();
-
- future_into_py(py, async move {
- let fluss_table =
- fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
-
- let table_scan = fluss_table.new_scan();
-
- let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
- PyErr::new::<pyo3::exceptions::PyRuntimeError, _>(format!(
- "Failed to create log scanner: {e:?}"
- ))
- })?;
-
- let admin = conn
- .get_admin()
- .await
- .map_err(|e| FlussError::new_err(e.to_string()))?;
+ /// Create a new log scanner for the table.
+ ///
+ /// Args:
+ /// project: Optional list of column indices (0-based) to include in the scan.
+ /// Empty list means all columns.
+ /// columns: Optional list of column names to include in the scan.
+ /// Empty list means all columns.
+ ///
+ /// Returns:
+ /// LogScanner, optionally with projection applied
+ ///
+ /// Note:
+ /// Specify only one of 'project' or 'columns'.
+ /// If neither is specified, all columns are included.
+ ///
+ /// Examples:
+ /// >>> # Scan all columns
+ /// >>> scanner = await table.new_log_scanner()
+ ///
+ /// >>> # Scan specific columns by index
+ /// >>> scanner = await table.new_log_scanner(project=[0, 2, 4])
+ ///
+ /// >>> # Scan specific columns by name (more Pythonic)
+ /// >>> scanner = await table.new_log_scanner(columns=["id", "name", "email"])
+ #[pyo3(signature = (project=None, columns=None))]
+ pub fn new_log_scanner<'py>(
+ &self,
+ py: Python<'py>,
+ project: Option<Vec<usize>>,
+ columns: Option<Vec<String>>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ // Validate mutually exclusive parameters and normalize empty lists
+ let projection = match (project, columns) {
+ (Some(_), Some(_)) => {
+ return Err(FlussError::new_err(
+ "Specify only one of 'project' or 'columns'".to_string(),
+ ));
+ }
+ (Some(indices), None) => {
+ if indices.is_empty() {
Review Comment:
can we just rutrn error when indices is emtpy? This behavior align with rust-core and java side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "leekeiabstraction (via GitHub)" <gi...@apache.org>.
leekeiabstraction commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680449465
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
Ah, I didn't realise that to panda polls. Thank you for the clarification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680417877
##########
bindings/python/src/table.rs:
##########
@@ -85,6 +85,111 @@ impl FlussTable {
})
}
+ /// Create a new log scanner with column projection (by index).
+ ///
+ /// Args:
+ /// column_indices: List of column indices to include in the scan (0-based)
+ ///
+ /// Returns:
+ /// LogScanner with projection applied
+ ///
+ /// Example:
+ /// >>> scanner = await table.new_log_scanner_with_projection([0, 2, 4])
+ pub fn new_log_scanner_with_projection<'py>(
+ &self,
+ py: Python<'py>,
+ column_indices: Vec<usize>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ // Validate early with Python-friendly error
+ if column_indices.is_empty() {
+ return Err(FlussError::new_err(
+ "column_indices cannot be empty".to_string(),
+ ));
+ }
+
+ let conn = self.connection.clone();
+ let metadata = self.metadata.clone();
+ let table_info = self.table_info.clone();
+
+ future_into_py(py, async move {
+ let fluss_table =
+ fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+ let table_scan = fluss_table.new_scan();
+ let table_scan = table_scan
+ .project(&column_indices)
+ .map_err(|e| FlussError::new_err(format!("Failed to project columns: {e}")))?;
+
+ let rust_scanner = table_scan.create_log_scanner().map_err(|e| {
+ FlussError::new_err(format!("Failed to create log scanner: {e}"))
+ })?;
Review Comment:
we should use FlussError, this is a leftover.
Thank you for catching this 👍
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
I'm not sure I get what you mean.
`to_pandas()` polls internally, but if we're talking about adding a separate polling API to Python bindings - we can add it.
Let's file an issue for it, as it's orthogonal to the column projection feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680429089
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
Create an issue https://github.com/apache/fluss-rust/issues/152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2680429089
##########
bindings/python/example/example.py:
##########
@@ -178,6 +178,28 @@ async def main():
except Exception as e:
print(f"Error during scanning: {e}")
+ # Demo: Column projection
+ print("\n--- Testing Column Projection ---")
+ try:
+ # Project specific columns by index (C++ parity)
+ print("\n1. Projection by index [0, 1] (id, name):")
+ scanner_index = await table.new_log_scanner_with_projection([0, 1])
+ scanner_index.subscribe(None, None)
+ df_projected = scanner_index.to_pandas()
+ print(df_projected.head())
+ print(f" Projected {df_projected.shape[1]} columns: {list(df_projected.columns)}")
+
+ # Project specific columns by name (Python-specific, more idiomatic!)
+ print("\n2. Projection by name ['name', 'score'] (Pythonic):")
+ scanner_names = await table.new_log_scanner_with_column_names(["name", "score"])
+ scanner_names.subscribe(None, None)
+ df_named = scanner_names.to_pandas()
+ print(df_named.head())
+ print(f" Projected {df_named.shape[1]} columns: {list(df_named.columns)}")
Review Comment:
Created an issue https://github.com/apache/fluss-rust/issues/152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#issuecomment-3755695691
@luoyuxia Thanks for the review!
Fixed, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE-149] Add column projection support to Python LogScanner [fluss-rust]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #151:
URL: https://github.com/apache/fluss-rust/pull/151#discussion_r2696538596
##########
bindings/python/src/table.rs:
##########
@@ -126,6 +157,102 @@ impl FlussTable {
has_primary_key,
}
}
+
+ /// Validate and deduplicate column indices (preserving order)
+ fn validate_and_dedupe_indices(&self, indices: &[usize]) -> PyResult<Vec<usize>> {
Review Comment:
Since the rust code will do the validation, maybe we can remove the validation in here?
Aslo may be we don't need to do deduplicate since java side also allows deduplicate columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org