You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "naivedogger (via GitHub)" <gi...@apache.org> on 2025/08/28 07:52:05 UTC

[PR] create bindings for table writing and reading [fluss-rust]

naivedogger opened a new pull request, #9:
URL: https://github.com/apache/fluss-rust/pull/9

   <!--
   *Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss-rust/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
   
     - Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
   
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
   
   
   **(The sections below can be removed for hotfixes or typos)**
   -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #xxx
   
   <!-- What is the purpose of the change -->
   
   ### Brief change log
   
   <!-- Please describe the changes made in this pull request and explain how they address the issue -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2432355293


##########
crates/fluss/src/client/table/scanner.rs:
##########
@@ -102,6 +105,18 @@ impl LogScanner {
         Ok(())
     }
 
+    pub async fn list_offsets_latest(&self, bucket_ids: Vec<i32>) -> Result<HashMap<i32, i64>> {
+        let cluster = self.metadata.get_cluster();
+        let tablet_server = cluster.get_one_available_server();

Review Comment:
   we can't just get one available_server. we should get the leader server to send list offsets request. Refer to java code for details. The quick fix is to iterate all  bucket_ids one by one and find the leader server, and then send request one by one.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.
+            let start_offset = -2;
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .subscribe(bucket_id, start_offset)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
+        use std::collections::HashMap;
+        use std::time::Duration;
+
+        let mut all_batches = Vec::new();
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let bucket_ids: Vec<i32> = (0..num_buckets).collect();
+
+        // todo: after supporting list_offsets with timestamp, we can use start_timestamp and end_timestamp here
+        let target_offsets: HashMap<i32, i64> = if !bucket_ids.is_empty() {
+            TOKIO_RUNTIME
+                .block_on(async { self.inner.list_offsets_latest(bucket_ids).await })
+                .map_err(|e| FlussError::new_err(e.to_string()))?
+        } else {
+            HashMap::new()
+        };
+
+        let mut current_offsets: HashMap<i32, i64> = HashMap::new();
+
+        if !target_offsets.is_empty() {
+            loop {
+                let batch_result = TOKIO_RUNTIME
+                    .block_on(async { self.inner.poll(Duration::from_millis(1000)).await });

Review Comment:
   ```suggestion
                       .block_on(async { self.inner.poll(Duration::from_millis(500)).await });
   ```
   to make it faster



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.
+            let start_offset = -2;
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .subscribe(bucket_id, start_offset)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
+        use std::collections::HashMap;
+        use std::time::Duration;
+
+        let mut all_batches = Vec::new();
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let bucket_ids: Vec<i32> = (0..num_buckets).collect();
+
+        // todo: after supporting list_offsets with timestamp, we can use start_timestamp and end_timestamp here
+        let target_offsets: HashMap<i32, i64> = if !bucket_ids.is_empty() {
+            TOKIO_RUNTIME
+                .block_on(async { self.inner.list_offsets_latest(bucket_ids).await })
+                .map_err(|e| FlussError::new_err(e.to_string()))?
+        } else {
+            HashMap::new()
+        };
+
+        let mut current_offsets: HashMap<i32, i64> = HashMap::new();
+
+        if !target_offsets.is_empty() {
+            loop {
+                let batch_result = TOKIO_RUNTIME
+                    .block_on(async { self.inner.poll(Duration::from_millis(1000)).await });
+
+                match batch_result {
+                    Ok(scan_records) => {
+                        if !scan_records.is_empty() {
+                            for (bucket, records) in scan_records.buckets().iter() {
+                                if let Some(last_record) = records.last() {
+                                    let max_offset_in_batch = last_record.offset();
+                                    let entry =
+                                        current_offsets.entry(bucket.bucket_id()).or_insert(0);
+                                    *entry = (*entry).max(max_offset_in_batch);
+                                }
+                            }
+
+                            let arrow_batch = Utils::convert_scan_records_to_arrow(scan_records);
+                            all_batches.extend(arrow_batch);
+                        }
+
+                        if Self::check_if_done(&target_offsets, &current_offsets) {
+                            break;
+                        }
+                    }
+                    Err(e) => return Err(FlussError::new_err(e.to_string())),
+                }
+            }
+        }
+
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    fn __repr__(&self) -> String {
+        format!("LogScanner(table={})", self.table_info.table_path)
+    }
+}
+
+impl LogScanner {
+    /// Create LogScanner from core LogScanner
+    pub fn from_core(
+        inner: fcore::client::LogScanner,
+        table_info: fcore::metadata::TableInfo,
+    ) -> Self {
+        Self {
+            inner,
+            table_info,
+            start_timestamp: None,
+            end_timestamp: None,
+        }
+    }
+
+    #[allow(clippy::unnecessary_map_or)]

Review Comment:
   nit:
   ```
   fn check_if_done(
           target_offsets: &HashMap<i32, i64>,
           current_offsets: &HashMap<i32, i64>,
       ) -> bool {
           target_offsets.iter().all(|(bucket_id, &target_offset)| {
               matches!(
               current_offsets.get(bucket_id).copied(),
               Some(current_offset) if current_offset >= target_offset - 1
           )
           })
       }
   ```



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.
+            let start_offset = -2;
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .subscribe(bucket_id, start_offset)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
+        use std::collections::HashMap;
+        use std::time::Duration;
+
+        let mut all_batches = Vec::new();
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let bucket_ids: Vec<i32> = (0..num_buckets).collect();
+
+        // todo: after supporting list_offsets with timestamp, we can use start_timestamp and end_timestamp here
+        let target_offsets: HashMap<i32, i64> = if !bucket_ids.is_empty() {
+            TOKIO_RUNTIME
+                .block_on(async { self.inner.list_offsets_latest(bucket_ids).await })
+                .map_err(|e| FlussError::new_err(e.to_string()))?
+        } else {
+            HashMap::new()
+        };
+
+        let mut current_offsets: HashMap<i32, i64> = HashMap::new();
+
+        if !target_offsets.is_empty() {
+            loop {
+                let batch_result = TOKIO_RUNTIME
+                    .block_on(async { self.inner.poll(Duration::from_millis(1000)).await });
+
+                match batch_result {
+                    Ok(scan_records) => {
+                        if !scan_records.is_empty() {
+                            for (bucket, records) in scan_records.buckets().iter() {
+                                if let Some(last_record) = records.last() {

Review Comment:
   can be 
   ```
   if let Some(last_record) = records.last() {
                                   current_offsets.insert(bucket.bucket_id(), last_record.offset());
                               }
   ```
   The last must be max until to now



##########
crates/fluss/src/rpc/message/list_offsets.rs:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::{impl_read_version_type, impl_write_version_type, proto};
+
+use crate::error::Result as FlussResult;
+use crate::proto::ListOffsetsResponse;
+use crate::rpc::api_key::ApiKey;
+use crate::rpc::api_version::ApiVersion;
+use crate::rpc::frame::{ReadError, WriteError};
+use crate::rpc::message::{ReadVersionedType, RequestBody, WriteVersionedType};
+use futures::future::join_all;
+use std::collections::HashMap;
+use tokio::sync::oneshot;
+
+use bytes::{Buf, BufMut};
+use prost::Message;
+
+/// Offset type constants as per proto comments
+pub const LIST_EARLIEST_OFFSET: i32 = 0;
+pub const LIST_LATEST_OFFSET: i32 = 1;
+pub const LIST_OFFSET_FROM_TIMESTAMP: i32 = 2;
+
+/// Client follower server id constant
+pub const CLIENT_FOLLOWER_SERVER_ID: i32 = -1;
+
+/// Offset specification for list offsets request
+#[derive(Debug, Clone)]
+pub enum OffsetSpec {
+    /// Earliest offset spec
+    Earliest,
+    /// Latest offset spec  
+    Latest,
+    /// Timestamp offset spec
+    Timestamp(i64),
+}
+
+impl OffsetSpec {
+    pub fn offset_type(&self) -> i32 {
+        match self {
+            OffsetSpec::Earliest => LIST_EARLIEST_OFFSET,
+            OffsetSpec::Latest => LIST_LATEST_OFFSET,
+            OffsetSpec::Timestamp(_) => LIST_OFFSET_FROM_TIMESTAMP,
+        }
+    }
+
+    pub fn start_timestamp(&self) -> Option<i64> {
+        match self {
+            OffsetSpec::Timestamp(ts) => Some(*ts),
+            _ => None,
+        }
+    }
+}
+
+/// Result container for list offsets operation

Review Comment:
   I think we can remove `ListOffsetsResult` in this pr.



##########
crates/fluss/src/rpc/message/list_offsets.rs:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::{impl_read_version_type, impl_write_version_type, proto};
+
+use crate::error::Result as FlussResult;
+use crate::proto::ListOffsetsResponse;
+use crate::rpc::api_key::ApiKey;
+use crate::rpc::api_version::ApiVersion;
+use crate::rpc::frame::{ReadError, WriteError};
+use crate::rpc::message::{ReadVersionedType, RequestBody, WriteVersionedType};
+use futures::future::join_all;
+use std::collections::HashMap;
+use tokio::sync::oneshot;
+
+use bytes::{Buf, BufMut};
+use prost::Message;
+
+/// Offset type constants as per proto comments
+pub const LIST_EARLIEST_OFFSET: i32 = 0;
+pub const LIST_LATEST_OFFSET: i32 = 1;
+pub const LIST_OFFSET_FROM_TIMESTAMP: i32 = 2;
+
+/// Client follower server id constant
+pub const CLIENT_FOLLOWER_SERVER_ID: i32 = -1;
+
+/// Offset specification for list offsets request
+#[derive(Debug, Clone)]
+pub enum OffsetSpec {
+    /// Earliest offset spec
+    Earliest,
+    /// Latest offset spec  
+    Latest,
+    /// Timestamp offset spec
+    Timestamp(i64),
+}
+
+impl OffsetSpec {
+    pub fn offset_type(&self) -> i32 {
+        match self {
+            OffsetSpec::Earliest => LIST_EARLIEST_OFFSET,
+            OffsetSpec::Latest => LIST_LATEST_OFFSET,
+            OffsetSpec::Timestamp(_) => LIST_OFFSET_FROM_TIMESTAMP,
+        }
+    }
+
+    pub fn start_timestamp(&self) -> Option<i64> {
+        match self {
+            OffsetSpec::Timestamp(ts) => Some(*ts),
+            _ => None,
+        }
+    }
+}
+
+/// Result container for list offsets operation
+pub struct ListOffsetsResult {
+    futures: HashMap<i32, oneshot::Receiver<FlussResult<i64>>>,
+}
+
+impl ListOffsetsResult {
+    pub fn new(futures: HashMap<i32, oneshot::Receiver<FlussResult<i64>>>) -> Self {
+        Self { futures }
+    }
+
+    /// Get the offset result for a specific bucket
+    pub async fn bucket_result(&mut self, bucket: i32) -> FlussResult<i64> {
+        if let Some(receiver) = self.futures.remove(&bucket) {
+            receiver
+                .await
+                .map_err(|_| crate::error::Error::WriteError("Channel closed".to_string()))?
+        } else {
+            Err(crate::error::Error::IllegalArgument(format!(
+                "Bucket {} not found",
+                bucket
+            )))
+        }
+    }
+
+    /// Wait for all bucket results to complete and return a map
+    pub async fn all(self) -> FlussResult<HashMap<i32, i64>> {
+        let mut results = HashMap::new();
+        let mut tasks = Vec::new();
+
+        // Collect all futures
+        for (bucket_id, receiver) in self.futures {
+            let task = async move {
+                let result = receiver
+                    .await
+                    .map_err(|_| crate::error::Error::WriteError("Channel closed".to_string()))?;
+                Ok::<(i32, i64), crate::error::Error>((bucket_id, result?))
+            };
+            tasks.push(task);
+        }
+
+        // Wait for all futures to complete
+        let task_results = join_all(tasks).await;
+
+        // Collect results
+        for task_result in task_results {
+            let (bucket_id, offset) = task_result?;
+            results.insert(bucket_id, offset);
+        }
+
+        Ok(results)
+    }
+}
+
+#[derive(Debug)]
+pub struct ListOffsetsRequest {
+    pub inner_request: proto::ListOffsetsRequest,
+}
+
+impl ListOffsetsRequest {
+    pub fn new(
+        table_id: i64,
+        partition_id: Option<i64>,
+        bucket_ids: Vec<i32>,
+        offset_spec: OffsetSpec,
+    ) -> FlussResult<Self> {
+        Ok(ListOffsetsRequest {
+            inner_request: proto::ListOffsetsRequest {
+                follower_server_id: CLIENT_FOLLOWER_SERVER_ID,
+                offset_type: offset_spec.offset_type(),
+                table_id,
+                partition_id,
+                bucket_id: bucket_ids,
+                start_timestamp: offset_spec.start_timestamp(),
+            },
+        })
+    }
+}
+
+impl RequestBody for ListOffsetsRequest {
+    type ResponseBody = ListOffsetsResponse;
+
+    const API_KEY: ApiKey = ApiKey::ListOffsets;
+
+    const REQUEST_VERSION: ApiVersion = ApiVersion(0);
+}
+
+impl_write_version_type!(ListOffsetsRequest);
+impl_read_version_type!(ListOffsetsResponse);
+
+impl ListOffsetsResponse {
+    pub fn offsets_map(&self) -> HashMap<i32, i64> {

Review Comment:
   ```suggestion
       pub fn offsets(&self) -> FlussResult<HashMap<i32, i64>> {
           self.buckets_resp
               .iter()
               .map(|resp| {
                   if let Some(_) = &resp.error_code {
                       // todo: consider use another suitable error
                        Err(Error::WriteError(format!("Missing offset, error message: {}", &resp.error_message
                            .as_deref().unwrap_or_else(|| "unknown server exception"))))
                   } else {
                       // if no error msg, offset must exists
                       Ok((resp.bucket_id, resp.offset.unwrap()))
                   }
               })
               .collect()
       }
   ```
   need to handle exception



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {

Review Comment:
   can we remove this method now?



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.
+            let start_offset = -2;
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .subscribe(bucket_id, start_offset)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
+        use std::collections::HashMap;
+        use std::time::Duration;
+
+        let mut all_batches = Vec::new();
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let bucket_ids: Vec<i32> = (0..num_buckets).collect();
+
+        // todo: after supporting list_offsets with timestamp, we can use start_timestamp and end_timestamp here
+        let target_offsets: HashMap<i32, i64> = if !bucket_ids.is_empty() {
+            TOKIO_RUNTIME
+                .block_on(async { self.inner.list_offsets_latest(bucket_ids).await })
+                .map_err(|e| FlussError::new_err(e.to_string()))?
+        } else {
+            HashMap::new()
+        };
+
+        let mut current_offsets: HashMap<i32, i64> = HashMap::new();
+
+        if !target_offsets.is_empty() {
+            loop {
+                let batch_result = TOKIO_RUNTIME
+                    .block_on(async { self.inner.poll(Duration::from_millis(1000)).await });
+
+                match batch_result {
+                    Ok(scan_records) => {

Review Comment:
   The current behavior is not following **only scan to target_offset** .
   If bucket 1 has already reach to target offset 1, but bucket2 is not. 
   The scanner will still scan the record for  bucket 1, and in current implementation, all records including records in bucket 1 will still be included. 
   



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.

Review Comment:
   consider use `const EARLIEST_OFFSET = -2`



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.
+            let start_offset = -2;
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .subscribe(bucket_id, start_offset)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
+        use std::collections::HashMap;
+        use std::time::Duration;
+
+        let mut all_batches = Vec::new();
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let bucket_ids: Vec<i32> = (0..num_buckets).collect();
+
+        // todo: after supporting list_offsets with timestamp, we can use start_timestamp and end_timestamp here
+        let target_offsets: HashMap<i32, i64> = if !bucket_ids.is_empty() {

Review Comment:
   `bucket_ids` should never be empty



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+
+        // If we can't convert, return an error
+        let type_name = value.bind(py).get_type().name()?;
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}",
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    #[allow(dead_code)]
+    start_timestamp: Option<i64>,
+    #[allow(dead_code)]
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        if _start_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying start_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+        if _end_timestamp.is_some() {
+            return Err(FlussError::new_err(
+                "Specifying end_timestamp is not yet supported. Please use None.".to_string(),
+            ));
+        }
+
+        let num_buckets = self.table_info.get_num_buckets();
+        for bucket_id in 0..num_buckets {
+            // -2 for the earliest offset.
+            let start_offset = -2;
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .subscribe(bucket_id, start_offset)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
+        use std::collections::HashMap;
+        use std::time::Duration;
+
+        let mut all_batches = Vec::new();
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let bucket_ids: Vec<i32> = (0..num_buckets).collect();
+
+        // todo: after supporting list_offsets with timestamp, we can use start_timestamp and end_timestamp here
+        let target_offsets: HashMap<i32, i64> = if !bucket_ids.is_empty() {
+            TOKIO_RUNTIME
+                .block_on(async { self.inner.list_offsets_latest(bucket_ids).await })
+                .map_err(|e| FlussError::new_err(e.to_string()))?
+        } else {
+            HashMap::new()
+        };
+
+        let mut current_offsets: HashMap<i32, i64> = HashMap::new();
+
+        if !target_offsets.is_empty() {
+            loop {
+                let batch_result = TOKIO_RUNTIME
+                    .block_on(async { self.inner.poll(Duration::from_millis(1000)).await });
+
+                match batch_result {
+                    Ok(scan_records) => {
+                        if !scan_records.is_empty() {

Review Comment:
   don't need `if`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428009188


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {

Review Comment:
   Thanks for pointing this out. However, IIUC, since the rust client doesn't support ListOffsets currently, I don't know if there's a way to get the offset of a record by its timestamp, right? If that's the case, maybe we leave this as a todo here? Or is there a better way to know if we have reached the end?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2425388625


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+        
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    /// Return an Arrow RecordBatchReader for streaming data
+    // TODO: Support this for streaming reads
+    fn to_arrow_batch_reader(&mut self, py: Python) -> PyResult<()> {
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        format!("LogScanner(table={})", self.table_info.table_path)
+    }
+}
+
+impl LogScanner {
+    /// Create LogScanner from core LogScanner
+    pub fn from_core(
+        inner: fcore::client::LogScanner,
+        table_info: fcore::metadata::TableInfo,
+    ) -> Self {
+        Self {
+            inner,
+            table_info,
+            start_timestamp: None,
+            end_timestamp: None,
+        }
+    }
+
+    /// Filter scan records by timestamp, returns (filtered_records, reached_end)
+    fn filter_records_by_timestamp(
+        scan_records: fcore::record::ScanRecords, 
+        end_timestamp: i64
+    ) -> (fcore::record::ScanRecords, bool) {
+        use std::collections::HashMap;
+        
+        let records_map = scan_records.into_records();
+        let mut filtered_map = HashMap::new();
+        let mut reached_end = false;
+        
+        for (table_bucket, records) in records_map {
+            let mut filtered_records = Vec::new();
+            
+            for record in records {
+                // Check if this record's timestamp exceeds end_timestamp
+                if record.timestamp() > end_timestamp {
+                    reached_end = true;

Review Comment:
   Also, if user uses a bigger timestamp which is more than the latest timestmp in Fluss to subscribe log data, and if no more records comes into fluss to increase the latest timestmp in Fluss. Seems the scan will never stop?
   
   I think the right way is use listOffsets by end timestamp  to get the right log end offset for all buckets, then check by offsets instead of by timestamp. But it needs more works.  We can consider to left it as a todo.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#issuecomment-3410405345

   @naivedogger I append a commit to improve the code. PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428015829


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+        
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    /// Return an Arrow RecordBatchReader for streaming data
+    // TODO: Support this for streaming reads
+    fn to_arrow_batch_reader(&mut self, py: Python) -> PyResult<()> {
+        Ok(())

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428028626


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {

Review Comment:
   fixed



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428029121


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))

Review Comment:
   fixed



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2432216429


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());

Review Comment:
   will refactor in #26, so let's keep what it's in this pr.



##########
crates/fluss/src/record/mod.rs:
##########
@@ -158,6 +158,10 @@ impl ScanRecords {
     pub fn is_empty(&self) -> bool {
         self.records.is_empty()
     }
+
+    pub fn buckets(&self) -> &HashMap<TableBucket, Vec<ScanRecord>> {

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2425297455


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {

Review Comment:
   no need to wrap to `PyResult`



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts

Review Comment:
   Why `LogScanner` is not `Send`? Is any any field cause it not Sendable?



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner

Review Comment:
   if LogScanner does be Send, can this method be removed?



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {

Review Comment:
   if specify _start_timestamp is not supported now, please throw error just like we did when _end_timestamp is specified. Set it to 0 is error-prone.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {

Review Comment:
   can it be:
   ```suggestion
       fn to_pandas(&self, py: Python) -> PyResult<PyObject> {
   ```



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+        
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    /// Return an Arrow RecordBatchReader for streaming data
+    // TODO: Support this for streaming reads
+    fn to_arrow_batch_reader(&mut self, py: Python) -> PyResult<()> {
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        format!("LogScanner(table={})", self.table_info.table_path)
+    }
+}
+
+impl LogScanner {
+    /// Create LogScanner from core LogScanner
+    pub fn from_core(
+        inner: fcore::client::LogScanner,
+        table_info: fcore::metadata::TableInfo,
+    ) -> Self {
+        Self {
+            inner,
+            table_info,
+            start_timestamp: None,
+            end_timestamp: None,
+        }
+    }
+
+    /// Filter scan records by timestamp, returns (filtered_records, reached_end)
+    fn filter_records_by_timestamp(
+        scan_records: fcore::record::ScanRecords, 
+        end_timestamp: i64
+    ) -> (fcore::record::ScanRecords, bool) {
+        use std::collections::HashMap;
+        
+        let records_map = scan_records.into_records();
+        let mut filtered_map = HashMap::new();
+        let mut reached_end = false;
+        
+        for (table_bucket, records) in records_map {
+            let mut filtered_records = Vec::new();
+            
+            for record in records {
+                // Check if this record's timestamp exceeds end_timestamp
+                if record.timestamp() > end_timestamp {
+                    reached_end = true;

Review Comment:
   But one bucket reach end timestamp don't represent all bucket reach the end timestamp. It'll case we miss records in other buckets.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))

Review Comment:
   no need to wrap to `PyResult`



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {

Review Comment:
   we can remove this method



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest

Review Comment:
   we use `-2` to represent it as earliest



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {

Review Comment:
   Can it be:
   ```suggestion
       fn to_arrow(&self, py: Python) -> PyResult<PyObject> {
   ```



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));

Review Comment:
   why return err when batches is empty? I think it should be fine when user scan no records.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+        
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    /// Return an Arrow RecordBatchReader for streaming data
+    // TODO: Support this for streaming reads
+    fn to_arrow_batch_reader(&mut self, py: Python) -> PyResult<()> {
+        Ok(())

Review Comment:
   if it's not supported, don't expose this method. Remove it.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {

Review Comment:
   `total_records == 0` don't means no more data. If it costs more than 1000 millis from server to client, the `total_records` will also be 0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2431986134


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+                        return Ok(Datum::String(leaked_str));
+                    }
+                }
+            }
+        }
+
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());

Review Comment:
   Memory leak: Using Box::leak creates intentional memory leaks for string lifetime management. Consider using Cow<'static, str> or a proper string pool to avoid accumulating leaked memory in long-running processes.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> TableInfo {
+        TableInfo::from_core(self.table_info.clone())
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> TablePath {
+        TablePath::from_core(self.table_path.clone())
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!(
+            "FlussTable(path={}.{})",
+            self.table_path.database(),
+            self.table_path.table()
+        )
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner
+                    .append(generic_row)
+                    .await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into())
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner
+                .flush()
+                .await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a AppendWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self { inner: append }
+    }
+
+    fn convert_python_value_to_datum(
+        &self,
+        py: Python,
+        value: PyObject,
+    ) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Blob, Datum, F32, F64};
+
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+
+        // Try to extract different types
+        if let Ok(type_name) = value.bind(py).get_type().name() {
+            if type_name == "StringScalar" {
+                if let Ok(py_value) = value.call_method0(py, "as_py") {
+                    if let Ok(str_val) = py_value.extract::<String>(py) {
+                        let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());

Review Comment:
   Memory leak: Using Box::leak creates intentional memory leaks for string lifetime management. Consider using Cow<'static, str> or a proper string pool to avoid accumulating leaked memory in long-running processes.



##########
bindings/python/src/metadata.rs:
##########
@@ -509,7 +530,7 @@ impl TableBucket {
 
     /// Convert to core TableBucket (internal use)
     pub fn to_core(&self) -> fcore::metadata::TableBucket {
-        fcore::metadata::TableBucket::new(self.table_id, self.partition_id, self.bucket)
+        fcore::metadata::TableBucket::new(self.table_id, self.bucket)

Review Comment:
   Missing partition_id parameter in TableBucket::new call. The struct has a partition_id field but it's not being passed to the constructor, which may cause data loss or incorrect bucket identification.
   ```suggestion
           fcore::metadata::TableBucket::new(self.table_id, self.partition_id, self.bucket)
   ```



##########
crates/fluss/src/record/mod.rs:
##########
@@ -158,6 +158,10 @@ impl ScanRecords {
     pub fn is_empty(&self) -> bool {
         self.records.is_empty()
     }
+
+    pub fn buckets(&self) -> &HashMap<TableBucket, Vec<ScanRecord>> {

Review Comment:
   [nitpick] Method name 'buckets' is ambiguous. Consider renaming to 'records_by_bucket' or 'bucket_records' to clearly indicate it returns a mapping of buckets to their records.
   ```suggestion
       pub fn records_by_bucket(&self) -> &HashMap<TableBucket, Vec<ScanRecord>> {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #9:
URL: https://github.com/apache/fluss-rust/pull/9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428019497


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {

Review Comment:
   Not to take `total_records == 0` as the end. We can just remove the code. Only take timestamp as the end mark.
   
   Although I  still think "if user uses a bigger timestamp which is more than the latest timestmp in Fluss to subscribe log data, and if no more records comes into fluss to increase the latest timestmp in Fluss. The scan will never stop"
   
   But now, no a better way. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428014862


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));

Review Comment:
   ok



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2425276246


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));

Review Comment:
   Memory is being intentionally leaked for every string conversion. This will cause unbounded memory growth in long-running applications. Consider using an owned string type or implementing proper lifetime management.
   ```suggestion
               // Pass the owned String directly to Datum::String to avoid leaking memory
               return Ok(Datum::String(str_val));
   ```



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name

Review Comment:
   The variable `type_name` is referenced but not defined. This will cause a compilation error. Consider replacing with a string representation of the Python object's type.
   ```suggestion
               "Cannot convert Python value to Datum: {}",
               value.get_type(py).name()?
   ```



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+        
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    /// Return an Arrow RecordBatchReader for streaming data
+    // TODO: Support this for streaming reads
+    fn to_arrow_batch_reader(&mut self, py: Python) -> PyResult<()> {
+        Ok(())
+    }
+

Review Comment:
   This method returns `()` instead of implementing the expected functionality. Consider either implementing the method or removing it to avoid providing a non-functional API surface.
   ```suggestion
   
   ```



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");

Review Comment:
   Debug print statement should be removed from production code. Consider using proper logging instead of println! for better control over output in production environments.



##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0

Review Comment:
   Always using offset 0 for timestamp-based subscriptions ignores the actual start_timestamp parameter. This will cause incorrect behavior when users specify a start timestamp.
   ```suggestion
                   Some(ts) => {
                       // Convert timestamp to offset for this bucket
                       self.table_info.timestamp_into_offset(bucket_id, ts)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428027871


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2426968778


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner

Review Comment:
   sure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2433388390


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::TOKIO_RUNTIME;
+use crate::*;
+use pyo3_async_runtimes::tokio::future_into_py;
+use std::collections::HashMap;
+use std::sync::Arc;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(&conn, metadata, table_info);
+
+            let table_append = fluss_table
+                .new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| Py::new(py, py_writer))
+        })
+    }
+
+    /// Create a new log scanner for the table
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table =
+                fcore::client::FlussTable::new(&conn, metadata.clone(), table_info.clone());
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(rust_scanner, table_info.clone());
+
+            Python::with_gil(|py| Py::new(py, py_scanner))
+        })
+    }
+
+    /// synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {

Review Comment:
   sure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#issuecomment-3410417564

   Thanks @luoyuxia for the code review and improvements. LGTM!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2426968389


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts

Review Comment:
   Maybe I'm mistaken, but I recall encountering an issue related to this... However, it seems that the LogScanner is actually 'Send' now. I tried in my local environment simply removing the 'unsendable' annotation in line #300, and the async method for creating log_scanner works fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428025578


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {
+        let mut all_batches = Vec::new();
+        let end_timestamp = self.end_timestamp;
+        // Poll all data from the scanner
+        loop {
+            let batch_result = TOKIO_RUNTIME.block_on(async {
+                use std::time::Duration;
+                self.inner.poll(Duration::from_millis(1000)).await
+            });
+            
+            match batch_result {
+                Ok(scan_records) => {
+                    let records_map = scan_records.into_records();
+                    
+                    let mut total_records = 0;
+                    for (bucket, records) in &records_map {
+                        total_records += records.len();
+                    }
+
+                    if total_records == 0 {
+                        break; // No more data
+                    }
+
+                    let scan_records = fcore::record::ScanRecords::new(records_map);
+
+                    // Filter records by end_timestamp if specified
+                    let (filtered_records, reached_end) = if let Some(end_ts) = end_timestamp {
+                        Self::filter_records_by_timestamp(scan_records, end_ts)
+                    } else {
+                        (scan_records, false)
+                    };
+                    
+                    let filtered_records_map = filtered_records.into_records();
+                    
+                    let filtered_records = fcore::record::ScanRecords::new(filtered_records_map);
+                    
+                    if !filtered_records.is_empty() {
+                        // Convert ScanRecords to Arrow RecordBatch
+                        let arrow_batch = Utils::convert_scan_records_to_arrow(filtered_records);
+                        all_batches.extend(arrow_batch);
+                    }
+                    
+                    // If we reached the end timestamp, stop polling
+                    if reached_end {
+                        break;
+                    }
+                },
+                Err(e) => return Err(FlussError::new_err(e.to_string())),
+            }
+        }
+        
+        if all_batches.is_empty() {
+            return Err(FlussError::new_err("No data available"));
+        }
+        
+        // Combine all batches into a single Arrow Table
+        Utils::combine_batches_to_table(py, all_batches)
+    }
+
+    /// Convert all data to Pandas DataFrame
+    fn to_pandas(&mut self, py: Python) -> PyResult<PyObject> {
+        let arrow_table = self.to_arrow(py)?;
+        
+        // Convert Arrow Table to Pandas DataFrame using pyarrow
+        let df = arrow_table.call_method0(py, "to_pandas")?;
+        Ok(df)
+    }
+
+    /// Return an Arrow RecordBatchReader for streaming data
+    // TODO: Support this for streaming reads
+    fn to_arrow_batch_reader(&mut self, py: Python) -> PyResult<()> {
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        format!("LogScanner(table={})", self.table_info.table_path)
+    }
+}
+
+impl LogScanner {
+    /// Create LogScanner from core LogScanner
+    pub fn from_core(
+        inner: fcore::client::LogScanner,
+        table_info: fcore::metadata::TableInfo,
+    ) -> Self {
+        Self {
+            inner,
+            table_info,
+            start_timestamp: None,
+            end_timestamp: None,
+        }
+    }
+
+    /// Filter scan records by timestamp, returns (filtered_records, reached_end)
+    fn filter_records_by_timestamp(
+        scan_records: fcore::record::ScanRecords, 
+        end_timestamp: i64
+    ) -> (fcore::record::ScanRecords, bool) {
+        use std::collections::HashMap;
+        
+        let records_map = scan_records.into_records();
+        let mut filtered_map = HashMap::new();
+        let mut reached_end = false;
+        
+        for (table_bucket, records) in records_map {
+            let mut filtered_records = Vec::new();
+            
+            for record in records {
+                // Check if this record's timestamp exceeds end_timestamp
+                if record.timestamp() > end_timestamp {
+                    reached_end = true;

Review Comment:
   I agree with this, as mentioned earlier. We need listOffsets for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat] Create Python bindings for table writing and reading [fluss-rust]

Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #9:
URL: https://github.com/apache/fluss-rust/pull/9#discussion_r2428034852


##########
bindings/python/src/table.rs:
##########
@@ -0,0 +1,474 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use pyo3::prelude::*;
+use crate::*;
+use std::sync::Arc;
+use pyo3_async_runtimes::tokio::future_into_py;
+use crate::TOKIO_RUNTIME;
+
+/// Represents a Fluss table for data operations
+#[pyclass]
+pub struct FlussTable {
+    connection: Arc<fcore::client::FlussConnection>,
+    metadata: Arc<fcore::client::Metadata>,
+    table_info: fcore::metadata::TableInfo,
+    table_path: fcore::metadata::TablePath,
+    has_primary_key: bool,
+}
+
+#[pymethods]
+impl FlussTable {
+    /// Create a new append writer for the table
+    fn new_append_writer<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+        
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata,
+                table_info,
+            );
+
+            let table_append = fluss_table.new_append()
+                .map_err(|e| FlussError::new_err(e.to_string()))?;
+
+            let rust_writer = table_append.create_writer();
+
+            let py_writer = AppendWriter::from_core(rust_writer);
+
+            Python::with_gil(|py| {
+                Py::new(py, py_writer)
+            })
+        })
+    }
+
+    /// Create a new log scanner for the table
+    // Note: LogScanner is not Send, so this may cause issues in async contexts
+    fn new_log_scanner<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        future_into_py(py, async move {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+
+            let rust_scanner = table_scan.create_log_scanner();
+
+            let py_scanner = LogScanner::from_core(
+                rust_scanner,
+                table_info.clone(),
+            );
+
+            Python::with_gil(|py| {
+                Py::new(py, py_scanner)
+            })
+        })
+    }
+
+    /// current workaround - synchronous version of new_log_scanner
+    fn new_log_scanner_sync(&self) -> PyResult<LogScanner> {
+        let conn = self.connection.clone();
+        let metadata = self.metadata.clone();
+        let table_info = self.table_info.clone();
+
+        let rust_scanner = TOKIO_RUNTIME.block_on(async {
+            let fluss_table = fcore::client::FlussTable::new(
+                &conn,
+                metadata.clone(),
+                table_info.clone(),
+            );
+
+            let table_scan = fluss_table.new_scan();
+            table_scan.create_log_scanner()
+        });
+
+        let py_scanner = LogScanner::from_core(
+            rust_scanner,
+            table_info.clone(),
+        );
+
+        Ok(py_scanner)
+    }
+
+    /// Get table information
+    pub fn get_table_info(&self) -> PyResult<TableInfo> {
+        Ok(TableInfo::from_core(self.table_info.clone()))
+    }
+
+    /// Get table path
+    pub fn get_table_path(&self) -> PyResult<TablePath> {
+        Ok(TablePath::from_core(self.table_path.clone()))
+    }
+
+    /// Check if table has primary key
+    pub fn has_primary_key(&self) -> bool {
+        self.has_primary_key
+    }
+
+    fn __repr__(&self) -> String {
+        format!("FlussTable(path={}.{})", 
+                self.table_path.database(), 
+                self.table_path.table())
+    }
+}
+
+impl FlussTable {
+    /// Create a FlussTable
+    pub fn new_table(
+        connection: Arc<fcore::client::FlussConnection>,
+        metadata: Arc<fcore::client::Metadata>,
+        table_info: fcore::metadata::TableInfo,
+        table_path: fcore::metadata::TablePath,
+        has_primary_key: bool,
+    ) -> Self {
+        Self {
+            connection,
+            metadata,
+            table_info,
+            table_path,
+            has_primary_key,
+        }
+    }
+}
+
+/// Writer for appending data to a Fluss table
+#[pyclass]
+pub struct AppendWriter {
+    inner: fcore::client::AppendWriter,
+}
+
+#[pymethods]
+impl AppendWriter {
+    /// Write Arrow table data
+    pub fn write_arrow(&mut self, py: Python, table: PyObject) -> PyResult<()> {
+        // Convert Arrow Table to batches and write each batch
+        let batches = table.call_method0(py, "to_batches")?;
+        let batch_list: Vec<PyObject> = batches.extract(py)?;
+        
+        for batch in batch_list {
+            self.write_arrow_batch(py, batch)?;
+        }
+        Ok(())
+    }
+
+    /// Write Arrow batch data
+    pub fn write_arrow_batch(&mut self, py: Python, batch: PyObject) -> PyResult<()> {
+        // Extract number of rows and columns from the Arrow batch
+        let num_rows: usize = batch.getattr(py, "num_rows")?.extract(py)?;
+        let num_columns: usize = batch.getattr(py, "num_columns")?.extract(py)?;
+        
+        // Process each row in the batch
+        for row_idx in 0..num_rows {
+            let mut generic_row = fcore::row::GenericRow::new();
+            
+            // Extract values for each column in this row
+            for col_idx in 0..num_columns {
+                let column = batch.call_method1(py, "column", (col_idx,))?;
+                let value = column.call_method1(py, "__getitem__", (row_idx,))?;
+                
+                // Convert the Python value to a Datum and add to the row
+                let datum = self.convert_python_value_to_datum(py, value)?;
+                generic_row.set_field(col_idx, datum);
+            }
+            
+            // Append this row using the async append method
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.append(generic_row).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+        
+        Ok(())
+    }
+
+    /// Write Pandas DataFrame data
+    pub fn write_pandas(&mut self, py: Python, df: PyObject) -> PyResult<()> {
+        // Import pyarrow module
+        let pyarrow = py.import("pyarrow")?;
+        
+        // Get the Table class from pyarrow module
+        let table_class = pyarrow.getattr("Table")?;
+        
+        // Call Table.from_pandas(df) - from_pandas is a class method
+        let pa_table = table_class.call_method1("from_pandas", (df,))?;
+        
+        // Then call write_arrow with the converted table
+        self.write_arrow(py, pa_table.into_py(py))
+    }
+
+    /// Flush any pending data
+    pub fn flush(&mut self) -> PyResult<()> {
+        TOKIO_RUNTIME.block_on(async {
+            self.inner.flush().await
+                .map_err(|e| FlussError::new_err(e.to_string()))
+        })
+    }
+
+    /// Close the writer and flush any pending data
+    pub fn close(&mut self) -> PyResult<()> {
+        self.flush()?;
+        println!("AppendWriter closed");
+        Ok(())
+    }
+
+    fn __repr__(&self) -> String {
+        "AppendWriter()".to_string()
+    }
+}
+
+impl AppendWriter {
+    /// Create a TableWriter from a core append writer
+    pub fn from_core(append: fcore::client::AppendWriter) -> Self {
+        Self {
+            inner: append,
+        }
+    }
+
+    fn convert_python_value_to_datum(&self, py: Python, value: PyObject) -> PyResult<fcore::row::Datum<'static>> {
+        use fcore::row::{Datum, F32, F64, Blob};
+        
+        // Check for None (null)
+        if value.is_none(py) {
+            return Ok(Datum::Null);
+        }
+        
+        // Try to extract different types
+        if let Ok(bool_val) = value.extract::<bool>(py) {
+            return Ok(Datum::Bool(bool_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i32>(py) {
+            return Ok(Datum::Int32(int_val));
+        }
+        
+        if let Ok(int_val) = value.extract::<i64>(py) {
+            return Ok(Datum::Int64(int_val));
+        }
+        
+        if let Ok(float_val) = value.extract::<f32>(py) {
+            return Ok(Datum::Float32(F32::from(float_val)));
+        }
+        
+        if let Ok(float_val) = value.extract::<f64>(py) {
+            return Ok(Datum::Float64(F64::from(float_val)));
+        }
+        
+        if let Ok(str_val) = value.extract::<String>(py) {
+            // Convert String to &'static str by leaking memory
+            // This is a simplified approach - in production, you might want better lifetime management
+            let leaked_str: &'static str = Box::leak(str_val.into_boxed_str());
+            return Ok(Datum::String(leaked_str));
+        }
+
+        if let Ok(bytes_val) = value.extract::<Vec<u8>>(py) {
+            let blob = Blob::from(bytes_val);
+            return Ok(Datum::Blob(blob));
+        }
+        
+        // If we can't convert, return an error
+        Err(FlussError::new_err(format!(
+            "Cannot convert Python value to Datum: {:?}", 
+            type_name
+        )))
+    }
+}
+
+/// Scanner for reading log data from a Fluss table
+#[pyclass(unsendable)]
+pub struct LogScanner {
+    inner: fcore::client::LogScanner,
+    table_info: fcore::metadata::TableInfo,
+    start_timestamp: Option<i64>,
+    end_timestamp: Option<i64>,
+}
+
+#[pymethods]
+impl LogScanner {
+    /// Subscribe to log data with timestamp range
+    fn subscribe(
+        &mut self,
+        _start_timestamp: Option<i64>,
+        _end_timestamp: Option<i64>,
+    ) -> PyResult<()> {
+        let end_timestamp = match _end_timestamp {
+            Some(ts) => ts,
+            None => {
+                return Err(FlussError::new_err(
+                    "end_timestamp must be specified for LogScanner".to_string()));
+            }
+        };
+    
+        self.start_timestamp = _start_timestamp;
+        self.end_timestamp = Some(end_timestamp);
+
+        let num_buckets = self.table_info.get_num_buckets();
+        let start_timestamp = self.start_timestamp; 
+
+        for bucket_id in 0..num_buckets {
+            let start_offset = match start_timestamp {
+                Some(_ts) => {
+                    // TODO: implement timestampIntoOffset in Rust client.
+                    0
+                },
+                None => 0, // earliest
+            };
+
+            TOKIO_RUNTIME.block_on(async {
+                self.inner.subscribe(bucket_id, start_offset).await
+                    .map_err(|e| FlussError::new_err(e.to_string()))
+            })?;
+        }
+
+        Ok(())
+    }
+
+    /// Convert all data to Arrow Table
+    fn to_arrow(&mut self, py: Python) -> PyResult<PyObject> {

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org