You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jesspav (via GitHub)" <gi...@apache.org> on 2025/10/27 22:52:37 UTC
[PR] Prototype raster [sedona-db]
jesspav opened a new pull request, #249:
URL: https://github.com/apache/sedona-db/pull/249
Simple prototype code to demonstrate viability of a StructArray data type for rasters (issue: #247 ).
If we are happy with the basic ideas here, we will break this up into many small PRs so we can get down to the nitty gritty. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prototype raster [sedona-db]
Posted by "jesspav (via GitHub)" <gi...@apache.org>.
jesspav commented on PR #249:
URL: https://github.com/apache/sedona-db/pull/249#issuecomment-3577377385
Closing - was just a prototype.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prototype raster [sedona-db]
Posted by "jesspav (via GitHub)" <gi...@apache.org>.
jesspav closed pull request #249: Prototype raster
URL: https://github.com/apache/sedona-db/pull/249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prototype raster [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on code in PR #249:
URL: https://github.com/apache/sedona-db/pull/249#discussion_r2468141365
##########
rust/sedona-raster/src/display_functions.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::datatype_functions::{bytes_per_pixel, read_pixel_value};
+use arrow::error::ArrowError;
+use sedona_schema::datatypes::{RasterRef, RasterRefImpl, StorageType};
+use std::fmt::Write;
+
+/// Write raster band matrix directly to a StringBuilder with specified precision
+pub fn write_band_to_builder(
+ raster: &RasterRefImpl,
+ band_number: usize,
+ precision: usize,
+ out: &mut impl Write,
+) -> Result<(), ArrowError> {
+ let band = raster.bands().band(band_number).unwrap();
+ let metadata = raster.metadata();
+ let height = metadata.height() as usize;
+ let width = metadata.width() as usize;
+
+ let slice = band.data() as &[u8];
+ let data_type = band.metadata().data_type();
+ if band.metadata().storage_type() != StorageType::InDb {
+ return Err(ArrowError::InvalidArgumentError(
+ "Pretty print indb not supported for non-InDb storage".to_string(),
Review Comment:
`Pretty print not supported for non-InDb storage`
##########
rust/sedona-raster/src/datatype_functions.rs:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow_schema::ArrowError;
+use sedona_schema::datatypes::BandDataType;
+
+pub fn bytes_per_pixel(data_type: BandDataType) -> Result<usize, ArrowError> {
+ match data_type {
+ BandDataType::UInt8 => Ok(1),
+ BandDataType::Int16 => Ok(2),
+ BandDataType::UInt16 => Ok(2),
+ BandDataType::Int32 => Ok(4),
+ BandDataType::UInt32 => Ok(4),
+ BandDataType::Float32 => Ok(4),
+ BandDataType::Float64 => Ok(8),
+ }
+}
Review Comment:
We don't need to wrap the return type with `Result` if the match is always exhaustive.
##########
rust/sedona-raster/src/display_functions.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::datatype_functions::{bytes_per_pixel, read_pixel_value};
+use arrow::error::ArrowError;
+use sedona_schema::datatypes::{RasterRef, RasterRefImpl, StorageType};
+use std::fmt::Write;
+
+/// Write raster band matrix directly to a StringBuilder with specified precision
+pub fn write_band_to_builder(
+ raster: &RasterRefImpl,
+ band_number: usize,
+ precision: usize,
+ out: &mut impl Write,
+) -> Result<(), ArrowError> {
+ let band = raster.bands().band(band_number).unwrap();
+ let metadata = raster.metadata();
+ let height = metadata.height() as usize;
+ let width = metadata.width() as usize;
+
+ let slice = band.data() as &[u8];
+ let data_type = band.metadata().data_type();
+ if band.metadata().storage_type() != StorageType::InDb {
+ return Err(ArrowError::InvalidArgumentError(
+ "Pretty print indb not supported for non-InDb storage".to_string(),
+ ));
+ }
+ let bytes_per_pixel = bytes_per_pixel(data_type.clone()).unwrap_or(1);
+
+ for row in 0..height {
+ for col in 0..width {
+ let start = (row * width + col) * bytes_per_pixel;
+ let end = start + bytes_per_pixel;
+ let pixel_bytes = &slice[start..end];
+
+ match read_pixel_value(pixel_bytes, data_type.clone()) {
+ Ok(value) => {
+ out.write_fmt(format_args!("{:8.*} ", precision, value))
+ .unwrap();
Review Comment:
The implementation of `Write` trait may fail writing bytes due to lots of reasons, and the caller may want to handle some of them. We may want to propagate this error using `?` instead of `.unwrap()`.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
Review Comment:
This method has incorrect lifetime notation: the lifetime of the returned &mut [u8] is actually related to the lifetime of the returned `MutableBuffer` object, and not related to `self` (RasterBuilder).
We can simply remove this method since we already have `create_band_buffer`. The holder of the `MutableBuffer` can simply call `as_slice_mut` if they want a `&mut [u8]` from it.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
Review Comment:
We already have another method `commit_band_buffer`, so we don't have to return a committer with the buffer.
##########
rust/sedona-gdal/src/readers.rs:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::dataset::{geotransform_components, tile_size};
+use arrow_array::StructArray;
+use arrow_schema::ArrowError;
+use gdal::raster::{GdalDataType, RasterBand};
+use gdal::Dataset;
+use sedona_raster::datatype_functions::{bytes_per_pixel, f64_to_bandtype_bytes};
+use sedona_schema::datatypes::{
+ BandDataType, BandMetadata, RasterBuilder, RasterMetadata, StorageType,
+};
+use std::sync::Arc;
+
+pub fn read_raster(filepath: &str) -> Result<Arc<StructArray>, ArrowError> {
+ let dataset = Dataset::open(filepath.to_string())
+ .map_err(|err| ArrowError::ParseError(err.to_string()))?;
+
+ // Extract geotransform components
+ let (origin_x, origin_y, pixel_width, pixel_height, rotation_x, rotation_y) =
+ geotransform_components(&dataset)?;
+
+ let (raster_width, raster_height) = dataset.raster_size();
+
+ let (tile_width, tile_height) = tile_size(&dataset)?;
+
+ let x_tile_count = (raster_width + tile_width - 1) / tile_width;
+ let y_tile_count = (raster_height + tile_height - 1) / tile_height;
+
+ let mut raster_builder = RasterBuilder::new(x_tile_count * y_tile_count);
+ let band_count = dataset.raster_count();
+
+ for tile_y in 0..y_tile_count {
+ for tile_x in 0..x_tile_count {
+ let x_offset = tile_x * tile_width;
+ let y_offset = tile_y * tile_height;
+
+ // Calculate geographic coordinates for this tile
+ // using the geotransform from the original raster
+ let tile_origin_x =
+ origin_x + (x_offset as f64) * pixel_width + (y_offset as f64) * rotation_x;
+ let tile_origin_y =
+ origin_y + (x_offset as f64) * rotation_y + (y_offset as f64) * pixel_height;
+
+ // Create raster metadata for this tile with actual geotransform values
+ let tile_metadata = RasterMetadata {
+ width: tile_width as u64,
+ height: tile_height as u64,
+ upperleft_x: tile_origin_x,
+ upperleft_y: tile_origin_y,
+ scale_x: pixel_width,
+ scale_y: pixel_height,
+ skew_x: rotation_x,
+ skew_y: rotation_y,
+ bounding_box: None, // TODO: should we calculate bounding box here?
+ };
+
+ raster_builder.start_raster(&tile_metadata, None, None)?;
+
+ for band_number in 1..=band_count {
+ let band: RasterBand = dataset.rasterband(band_number).unwrap();
+ // This should be the same as tile width/height, except for edge tiles
+ // but we would need to update the width/height in the metadata above then.
+ // For now, fail if sizes don't match.
+ let (x_size, y_size) = band.size();
+ if x_size != tile_width || y_size != tile_height {
+ return Err(ArrowError::ParseError(format!(
+ "Band size ({}, {}) does not match expected tile size ({}, {})",
+ x_size, y_size, tile_width, tile_height
+ )));
+ }
+
+ let data_type = gdaldatatype_to_banddatatype(band.band_type())?;
+ let data_type_bytes = bytes_per_pixel(data_type.clone())?;
+ let buffer_size_bytes = x_size * y_size * data_type_bytes.clone();
+
+ // Get a mutable buffer slice for GDAL to write directly into
+ let (buffer, slice) = raster_builder.get_band_buffer_slice(buffer_size_bytes);
+
+ // TODO: Do we need resampling? If so set buffer_size to different from window_size
+ // and have a ResampleAlgorithm.
+ band.read_into_slice(
Review Comment:
I suggest starting with no resampling
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
Review Comment:
I believe that `BinaryView` is a better option. Growing the binary array when appending in-db band data could be less painful.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
+
+ /// Commit a MutableBuffer to the band data
+ pub fn commit_band_buffer(&mut self, buffer: MutableBuffer) {
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ }
+
+ /// Finish writing the current band with its metadata
+ /// TODO: The band_metadata is in the finish in the band call, but in the
+ /// start in the raster call. Make it consistent.
Review Comment:
I don't think it is a problem. `start_raster` takes raster metadata as pararameter, while `finish_band` takes band metadata as parameter.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
+
+ /// Commit a MutableBuffer to the band data
+ pub fn commit_band_buffer(&mut self, buffer: MutableBuffer) {
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ }
+
+ /// Finish writing the current band with its metadata
+ /// TODO: The band_metadata is in the finish in the band call, but in the
+ /// start in the raster call. Make it consistent.
+ pub fn finish_band(&mut self, band_metadata: BandMetadata) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+
+ let metadata_builder = band_builder.field_builder::<StructBuilder>(0).unwrap();
+
+ if let Some(nodata) = band_metadata.nodata_value {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_value(&nodata);
+ } else {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::STORAGE_TYPE)
+ .unwrap()
+ .append_value(band_metadata.storage_type as u32);
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::DATATYPE)
+ .unwrap()
+ .append_value(band_metadata.datatype as u32);
+
+ // Handle OutDb URL
+ if let Some(url) = band_metadata.outdb_url {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_value(&url);
+ } else {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_null();
+ }
+
+ // Handle OutDb band ID
+ if let Some(band_id) = band_metadata.outdb_band_id {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_value(band_id);
+ } else {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder.append(true);
+
+ // Finish the band
+ band_builder.append(true);
+ Ok(())
+ }
+
+ /// Finish all bands for the current raster
+ pub fn finish_raster(&mut self) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(true);
+ // Mark this raster as valid (not null) in the main struct
+ self.main_builder.append(true);
+ Ok(())
+ }
+
+ /// Append raster metadata from a MetadataRef trait object
+ fn append_metadata_from_ref(&mut self, metadata: &dyn MetadataRef) -> Result<(), ArrowError> {
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ // Width
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(metadata.width());
+
+ // Height
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(metadata.height());
+
+ // Geotransform parameters
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(metadata.upper_left_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(metadata.upper_left_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(metadata.scale_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(metadata.scale_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(metadata.skew_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(metadata.skew_y());
+
+ metadata_builder.append(true);
+
+ Ok(())
+ }
+
+ /// Set the CRS for the current raster
+ pub fn set_crs(&mut self, crs: Option<&str>) -> Result<(), ArrowError> {
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ match crs {
+ Some(crs_data) => crs_builder.append_value(crs_data),
+ None => crs_builder.append_null(),
+ }
+ Ok(())
+ }
+
+ /// Append a bounding box to the current raster
+ pub fn append_bounding_box(&mut self, bbox: Option<&BoundingBox>) -> Result<(), ArrowError> {
+ let bbox_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::BBOX)
+ .unwrap();
+
+ if let Some(bbox) = bbox {
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_value(bbox.min_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_value(bbox.min_y);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_value(bbox.max_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_value(bbox.max_y);
+
+ bbox_builder.append(true);
+ } else {
+ // Append null bounding box - need to fill in null values for all fields
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder.append(false);
+ }
+ Ok(())
+ }
+
+ /// Append a null raster
+ pub fn append_null(&mut self) -> Result<(), ArrowError> {
+ // Since metadata fields are non-nullable, provide default values
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ // Mark the metadata struct as valid since it has valid values
+ metadata_builder.append(true);
+
+ // Append null CRS (now using StringBuilder instead of StringViewBuilder)
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ crs_builder.append_null();
+
+ // Append null bounding box
+ self.append_bounding_box(None)?;
+
+ // Append null bands
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(false);
+
+ // Mark this raster as null in the main struct
+ self.main_builder.append(false);
+
+ Ok(())
+ }
+
+ /// Finish building and return the constructed StructArray
+ pub fn finish(mut self) -> Result<StructArray, ArrowError> {
+ Ok(self.main_builder.finish())
+ }
+}
+
+/// Convenience wrapper for the zero-copy band writing approach
+impl RasterBuilder {
+ /// High-level method that allows for zero-copy with a callback approach
+ pub fn append_raster_with_callback<F>(
+ &mut self,
+ metadata: RasterMetadata,
+ band_count: usize,
+ mut write_bands: F,
+ ) -> Result<(), ArrowError>
+ where
+ F: FnMut(usize, &mut BinaryBuilder) -> Result<BandMetadata, ArrowError>,
+ {
+ self.start_raster(&metadata, None, metadata.bounding_box.as_ref())?;
+
+ for band_index in 0..band_count {
+ let band_metadata = {
+ let binary_builder = self.band_data_writer();
+ write_bands(band_index, binary_builder)?
+ };
+ self.finish_band(band_metadata)?;
+ }
+
+ self.finish_raster()?;
+ Ok(())
+ }
+}
+
+/// Iterator and accessor traits for reading raster data from Arrow arrays.
+///
+/// These traits provide a zero-copy interface for accessing raster metadata and band data
+/// from the Arrow-based storage format. The implementation handles both InDb and OutDbRef
+/// storage types seamlessly.
+
+/// Trait for accessing raster metadata (dimensions, geotransform, bounding box, etc.)
+pub trait MetadataRef {
+ /// Width of the raster in pixels (using u64 to match schema)
+ fn width(&self) -> u64;
+ /// Height of the raster in pixels (using u64 to match schema)
+ fn height(&self) -> u64;
+ /// X coordinate of the upper-left corner
+ fn upper_left_x(&self) -> f64;
+ /// Y coordinate of the upper-left corner
+ fn upper_left_y(&self) -> f64;
+ /// X-direction pixel size (scale)
+ fn scale_x(&self) -> f64;
+ /// Y-direction pixel size (scale)
+ fn scale_y(&self) -> f64;
+ /// X-direction skew/rotation
+ fn skew_x(&self) -> f64;
+ /// Y-direction skew/rotation
+ fn skew_y(&self) -> f64;
+}
+
+/// Implement MetadataRef for RasterMetadata to allow direct use with builder
+impl MetadataRef for RasterMetadata {
+ fn width(&self) -> u64 {
+ self.width
+ }
+ fn height(&self) -> u64 {
+ self.height
+ }
+ fn upper_left_x(&self) -> f64 {
+ self.upperleft_x
+ }
+ fn upper_left_y(&self) -> f64 {
+ self.upperleft_y
+ }
+ fn scale_x(&self) -> f64 {
+ self.scale_x
+ }
+ fn scale_y(&self) -> f64 {
+ self.scale_y
+ }
+ fn skew_x(&self) -> f64 {
+ self.skew_x
+ }
+ fn skew_y(&self) -> f64 {
+ self.skew_y
+ }
+}
+
+/// Trait for accessing individual band metadata
+pub trait BandMetadataRef {
+ /// No-data value as raw bytes (None if null)
+ fn nodata_value(&self) -> Option<&[u8]>;
+ /// Storage type (InDb, OutDbRef, etc)
+ fn storage_type(&self) -> StorageType;
+ /// Band data type (UInt8, Float32, etc.)
+ fn data_type(&self) -> BandDataType;
+ /// OutDb URL (only used when storage_type == OutDbRef)
+ fn outdb_url(&self) -> Option<&str>;
+ /// OutDb band ID (only used when storage_type == OutDbRef)
+ fn outdb_band_id(&self) -> Option<u32>;
+}
+
+/// Trait for accessing individual band data
+pub trait BandRef {
+ /// Band metadata accessor
+ fn metadata(&self) -> &dyn BandMetadataRef;
+ /// Raw band data as bytes (zero-copy access)
+ fn data(&self) -> &[u8];
+}
+
+/// Trait for accessing all bands in a raster
+pub trait BandsRef {
+ /// Number of bands in the raster
+ fn len(&self) -> usize;
+ /// Check if no bands are present
+ fn is_empty(&self) -> bool {
+ self.len() == 0
+ }
+ /// Get a specific band by number (returns None if out of bounds)
+ /// By convention, band numbers are 1-based
+ fn band(&self, number: usize) -> Option<Box<dyn BandRef + '_>>;
+ /// Iterator over all bands
+ fn iter(&self) -> BandIterator<'_>;
+}
+
+/// Trait for accessing complete raster data
+pub trait RasterRef {
+ /// Raster metadata accessor
+ fn metadata(&self) -> &dyn MetadataRef;
+ /// CRS accessor
+ fn crs(&self) -> Option<&str>;
+ /// Bands accessor
+ fn bands(&self) -> &dyn BandsRef;
+}
+
+/// Implementation of MetadataRef for Arrow StructArray
+struct MetadataRefImpl<'a> {
+ metadata_struct: &'a StructArray,
+ index: usize,
+}
+
+impl<'a> MetadataRef for MetadataRefImpl<'a> {
+ fn width(&self) -> u64 {
+ self.metadata_struct
+ .column(metadata_indices::WIDTH)
+ .as_any()
+ .downcast_ref::<UInt64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn height(&self) -> u64 {
+ self.metadata_struct
+ .column(metadata_indices::HEIGHT)
+ .as_any()
+ .downcast_ref::<UInt64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn upper_left_x(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::UPPERLEFT_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn upper_left_y(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::UPPERLEFT_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn scale_x(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SCALE_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn scale_y(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SCALE_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn skew_x(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SKEW_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn skew_y(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SKEW_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+}
+
+/// Implementation of BandMetadataRef for Arrow StructArray
+struct BandMetadataRefImpl<'a> {
+ metadata_struct: &'a StructArray,
+ band_index: usize,
+}
+
+impl<'a> BandMetadataRef for BandMetadataRefImpl<'a> {
+ fn nodata_value(&self) -> Option<&[u8]> {
+ let nodata_array = self
+ .metadata_struct
+ .column(band_metadata_indices::NODATAVALUE)
+ .as_any()
+ .downcast_ref::<BinaryArray>()
+ .expect("Expected BinaryArray for nodata");
+
+ if nodata_array.is_null(self.band_index) {
+ None
+ } else {
+ Some(nodata_array.value(self.band_index))
+ }
+ }
+
+ fn storage_type(&self) -> StorageType {
+ let storage_type_array = self
+ .metadata_struct
+ .column(band_metadata_indices::STORAGE_TYPE)
+ .as_any()
+ .downcast_ref::<UInt32Array>()
+ .expect("Expected UInt32Array for storage_type");
+
+ match storage_type_array.value(self.band_index) {
+ 0 => StorageType::InDb,
+ 1 => StorageType::OutDbRef,
+ _ => panic!(
+ "Unknown storage type: {}",
+ storage_type_array.value(self.band_index)
+ ),
+ }
+ }
+
+ fn data_type(&self) -> BandDataType {
+ let datatype_array = self
+ .metadata_struct
+ .column(band_metadata_indices::DATATYPE)
+ .as_any()
+ .downcast_ref::<UInt32Array>()
+ .expect("Expected UInt32Array for datatype");
+
+ match datatype_array.value(self.band_index) {
+ 0 => BandDataType::UInt8,
+ 1 => BandDataType::UInt16,
+ 2 => BandDataType::Int16,
+ 3 => BandDataType::UInt32,
+ 4 => BandDataType::Int32,
+ 5 => BandDataType::Float32,
+ 6 => BandDataType::Float64,
+ _ => panic!(
+ "Unknown band data type: {}",
+ datatype_array.value(self.band_index)
+ ),
+ }
+ }
+
+ fn outdb_url(&self) -> Option<&str> {
+ let url_array = self
+ .metadata_struct
+ .column(band_metadata_indices::OUTDB_URL)
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("Expected StringArray for outdb_url");
+
+ if url_array.is_null(self.band_index) {
+ None
+ } else {
+ Some(url_array.value(self.band_index))
+ }
+ }
+
+ fn outdb_band_id(&self) -> Option<u32> {
+ let band_id_array = self
+ .metadata_struct
+ .column(band_metadata_indices::OUTDB_BAND_ID)
+ .as_any()
+ .downcast_ref::<UInt32Array>()
+ .expect("Expected UInt32Array for outdb_band_id");
+
+ if band_id_array.is_null(self.band_index) {
+ None
+ } else {
+ Some(band_id_array.value(self.band_index))
+ }
+ }
+}
+
+/// Implementation of BandRef for accessing individual band data
+struct BandRefImpl<'a> {
+ band_metadata: BandMetadataRefImpl<'a>,
+ band_data: &'a [u8],
+}
+
+impl<'a> BandRef for BandRefImpl<'a> {
+ fn metadata(&self) -> &dyn BandMetadataRef {
+ &self.band_metadata
+ }
+
+ fn data(&self) -> &[u8] {
+ self.band_data
+ }
+}
+
+/// Implementation of BandsRef for accessing all bands in a raster
+struct BandsRefImpl<'a> {
+ bands_list: &'a ListArray,
+ raster_index: usize,
+}
+
+impl<'a> BandsRef for BandsRefImpl<'a> {
+ fn len(&self) -> usize {
+ let start = self.bands_list.value_offsets()[self.raster_index] as usize;
+ let end = self.bands_list.value_offsets()[self.raster_index + 1] as usize;
+ end - start
+ }
+
+ /// Get a specific band by number (1-based index)
+ fn band(&self, number: usize) -> Option<Box<dyn BandRef + '_>> {
+ // TODO: Error instead of returning None for out-of-bounds
+ if number == 0 {
+ return None; // Band numbers are 1-based
+ }
+ // By convention, band numbers are 1-based.
+ // Convert to zero-based index.
+ let index = number - 1;
+ if index >= self.len() {
+ return None;
+ }
+
+ let start = self.bands_list.value_offsets()[self.raster_index] as usize;
+ let band_row = start + index;
+
+ let bands_struct = self
+ .bands_list
+ .values()
+ .as_any()
+ .downcast_ref::<StructArray>()?;
+
+ // Get the metadata substructure from the band struct
+ let band_metadata_struct = bands_struct
+ .column(band_indices::METADATA)
+ .as_any()
+ .downcast_ref::<StructArray>()?;
+
+ let band_metadata = BandMetadataRefImpl {
+ metadata_struct: band_metadata_struct,
+ band_index: band_row,
+ };
+
+ // Get band data from the Binary column within the band struct
+ let band_data_array = bands_struct
+ .column(band_indices::DATA)
+ .as_any()
+ .downcast_ref::<BinaryArray>()?;
+
+ let band_data = band_data_array.value(band_row);
+
+ Some(Box::new(BandRefImpl {
+ band_metadata,
+ band_data,
+ }))
+ }
+
+ fn iter(&self) -> BandIterator<'_> {
+ BandIterator {
+ bands: self,
+ current: 1, // Start at 1 for 1-based band numbering
+ }
+ }
+}
+
+/// Iterator for bands within a raster
+pub struct BandIterator<'a> {
+ bands: &'a dyn BandsRef,
+ current: usize,
+}
+
+impl<'a> Iterator for BandIterator<'a> {
+ type Item = Box<dyn BandRef + 'a>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ // current is 1-based, compare against len() + 1
+ if self.current <= self.bands.len() {
+ let band = self.bands.band(self.current);
+ self.current += 1;
+ band
+ } else {
+ None
+ }
+ }
+
+ fn size_hint(&self) -> (usize, Option<usize>) {
+ // current is 1-based, so remaining calculation needs adjustment
+ let remaining = self.bands.len().saturating_sub(self.current - 1);
+ (remaining, Some(remaining))
+ }
+}
+
+impl ExactSizeIterator for BandIterator<'_> {}
+
+/// Implementation of RasterRef for complete raster access
+pub struct RasterRefImpl<'a> {
+ metadata: MetadataRefImpl<'a>,
+ crs: &'a StringArray,
+ bbox: &'a StructArray,
+ bands: BandsRefImpl<'a>,
+}
+
+impl<'a> RasterRefImpl<'a> {
+ /// Create a new RasterRefImpl from a struct array and index using hard-coded indices
+ pub fn new(raster_struct: &'a StructArray, raster_index: usize) -> Self {
+ let metadata_struct = raster_struct
+ .column(raster_indices::METADATA)
+ .as_any()
+ .downcast_ref::<StructArray>()
+ .unwrap();
+
+ let crs = raster_struct
+ .column(raster_indices::CRS)
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ let bbox = raster_struct
+ .column(raster_indices::BBOX)
+ .as_any()
+ .downcast_ref::<StructArray>()
+ .unwrap();
+
+ let bands_list = raster_struct
+ .column(raster_indices::BANDS)
+ .as_any()
+ .downcast_ref::<ListArray>()
+ .unwrap();
+
+ let metadata = MetadataRefImpl {
+ metadata_struct,
+ index: raster_index,
+ };
+
+ let bands = BandsRefImpl {
+ bands_list,
+ raster_index,
+ };
+
+ Self {
+ metadata,
+ crs,
+ bbox,
+ bands,
+ }
+ }
+
+ /// Access the bounding box for this raster
+ pub fn bounding_box(&self, raster_index: usize) -> Option<BoundingBox> {
+ if self.bbox.is_null(raster_index) {
+ None
+ } else {
+ Some(BoundingBox {
+ min_x: self
+ .bbox
+ .column(bounding_box_indices::MIN_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(raster_index),
+ min_y: self
+ .bbox
+ .column(bounding_box_indices::MIN_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(raster_index),
+ max_x: self
+ .bbox
+ .column(bounding_box_indices::MAX_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(raster_index),
+ max_y: self
+ .bbox
+ .column(bounding_box_indices::MAX_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(raster_index),
+ })
+ }
+ }
+}
+
+impl<'a> RasterRef for RasterRefImpl<'a> {
+ fn metadata(&self) -> &dyn MetadataRef {
+ &self.metadata
+ }
+
+ fn crs(&self) -> Option<&str> {
+ if self.crs.is_null(self.bands.raster_index) {
+ None
+ } else {
+ Some(&self.crs.value(self.bands.raster_index))
+ }
+ }
+
+ fn bands(&self) -> &dyn BandsRef {
+ &self.bands
+ }
+}
+
+/// Iterator over raster structs in an Arrow StructArray
+///
+/// This provides efficient, zero-copy access to raster data stored in Arrow format.
+/// Each iteration yields a `RasterRefImpl` that provides access to both metadata and band data.
+pub struct RasterStructIterator<'a> {
+ raster_array: &'a StructArray,
+ current_row: usize,
+}
+
+impl<'a> RasterStructIterator<'a> {
+ /// Create a new iterator over the raster struct array
+ pub fn new(raster_array: &'a StructArray) -> Self {
+ Self {
+ raster_array,
+ current_row: 0,
+ }
+ }
+
+ /// Get the total number of rasters in the array
+ pub fn len(&self) -> usize {
+ self.raster_array.len()
+ }
+
+ /// Check if the array is empty
+ pub fn is_empty(&self) -> bool {
+ self.raster_array.is_empty()
+ }
+
+ /// Get a specific raster by index without consuming the iterator
+ pub fn get(&self, index: usize) -> Option<RasterRefImpl<'a>> {
+ if index >= self.raster_array.len() {
+ return None;
+ }
+
+ Some(RasterRefImpl::new(self.raster_array, index))
+ }
+}
+
+impl<'a> Iterator for RasterStructIterator<'a> {
+ type Item = RasterRefImpl<'a>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ if self.current_row < self.raster_array.len() {
+ let result = self.get(self.current_row)?;
+ self.current_row += 1;
+ Some(result)
+ } else {
+ None
+ }
+ }
+
+ fn size_hint(&self) -> (usize, Option<usize>) {
+ let remaining = self.raster_array.len().saturating_sub(self.current_row);
+ (remaining, Some(remaining))
+ }
+}
Review Comment:
I'm uncertain should we implement `Iterator` trait for `RasterStructIterator`, since the only actual usage of this type is random access (`get(&self, index: usize)`).
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
+
+ /// Commit a MutableBuffer to the band data
+ pub fn commit_band_buffer(&mut self, buffer: MutableBuffer) {
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ }
+
+ /// Finish writing the current band with its metadata
+ /// TODO: The band_metadata is in the finish in the band call, but in the
+ /// start in the raster call. Make it consistent.
+ pub fn finish_band(&mut self, band_metadata: BandMetadata) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+
+ let metadata_builder = band_builder.field_builder::<StructBuilder>(0).unwrap();
+
+ if let Some(nodata) = band_metadata.nodata_value {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_value(&nodata);
+ } else {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::STORAGE_TYPE)
+ .unwrap()
+ .append_value(band_metadata.storage_type as u32);
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::DATATYPE)
+ .unwrap()
+ .append_value(band_metadata.datatype as u32);
+
+ // Handle OutDb URL
+ if let Some(url) = band_metadata.outdb_url {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_value(&url);
+ } else {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_null();
+ }
+
+ // Handle OutDb band ID
+ if let Some(band_id) = band_metadata.outdb_band_id {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_value(band_id);
+ } else {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder.append(true);
+
+ // Finish the band
+ band_builder.append(true);
+ Ok(())
+ }
+
+ /// Finish all bands for the current raster
+ pub fn finish_raster(&mut self) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(true);
+ // Mark this raster as valid (not null) in the main struct
+ self.main_builder.append(true);
+ Ok(())
+ }
+
+ /// Append raster metadata from a MetadataRef trait object
+ fn append_metadata_from_ref(&mut self, metadata: &dyn MetadataRef) -> Result<(), ArrowError> {
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ // Width
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(metadata.width());
+
+ // Height
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(metadata.height());
+
+ // Geotransform parameters
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(metadata.upper_left_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(metadata.upper_left_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(metadata.scale_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(metadata.scale_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(metadata.skew_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(metadata.skew_y());
Review Comment:
Calling `field_builder` for every column when inserting a struct value has huge performance overhead, we hit this problem in the initial vectorized engine POC for geometry types. However, I think we can keep it this way, since raster operations could be heavy weight enough to make it negligible.
##########
rust/sedona-gdal/src/readers.rs:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::dataset::{geotransform_components, tile_size};
+use arrow_array::StructArray;
+use arrow_schema::ArrowError;
+use gdal::raster::{GdalDataType, RasterBand};
+use gdal::Dataset;
+use sedona_raster::datatype_functions::{bytes_per_pixel, f64_to_bandtype_bytes};
+use sedona_schema::datatypes::{
+ BandDataType, BandMetadata, RasterBuilder, RasterMetadata, StorageType,
+};
+use std::sync::Arc;
+
+pub fn read_raster(filepath: &str) -> Result<Arc<StructArray>, ArrowError> {
+ let dataset = Dataset::open(filepath.to_string())
+ .map_err(|err| ArrowError::ParseError(err.to_string()))?;
+
+ // Extract geotransform components
+ let (origin_x, origin_y, pixel_width, pixel_height, rotation_x, rotation_y) =
+ geotransform_components(&dataset)?;
+
+ let (raster_width, raster_height) = dataset.raster_size();
+
+ let (tile_width, tile_height) = tile_size(&dataset)?;
+
+ let x_tile_count = (raster_width + tile_width - 1) / tile_width;
+ let y_tile_count = (raster_height + tile_height - 1) / tile_height;
+
+ let mut raster_builder = RasterBuilder::new(x_tile_count * y_tile_count);
+ let band_count = dataset.raster_count();
+
+ for tile_y in 0..y_tile_count {
+ for tile_x in 0..x_tile_count {
+ let x_offset = tile_x * tile_width;
+ let y_offset = tile_y * tile_height;
+
+ // Calculate geographic coordinates for this tile
+ // using the geotransform from the original raster
+ let tile_origin_x =
+ origin_x + (x_offset as f64) * pixel_width + (y_offset as f64) * rotation_x;
+ let tile_origin_y =
+ origin_y + (x_offset as f64) * rotation_y + (y_offset as f64) * pixel_height;
+
+ // Create raster metadata for this tile with actual geotransform values
+ let tile_metadata = RasterMetadata {
+ width: tile_width as u64,
+ height: tile_height as u64,
+ upperleft_x: tile_origin_x,
+ upperleft_y: tile_origin_y,
+ scale_x: pixel_width,
+ scale_y: pixel_height,
+ skew_x: rotation_x,
+ skew_y: rotation_y,
+ bounding_box: None, // TODO: should we calculate bounding box here?
Review Comment:
How about computing bounding_box lazily in `RasterMetadata`? We can define a constructor that takes an affine transformation as parameter and initialize bounding_box as None.
##########
rust/sedona-gdal/src/readers.rs:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::dataset::{geotransform_components, tile_size};
+use arrow_array::StructArray;
+use arrow_schema::ArrowError;
+use gdal::raster::{GdalDataType, RasterBand};
+use gdal::Dataset;
+use sedona_raster::datatype_functions::{bytes_per_pixel, f64_to_bandtype_bytes};
+use sedona_schema::datatypes::{
+ BandDataType, BandMetadata, RasterBuilder, RasterMetadata, StorageType,
+};
+use std::sync::Arc;
+
+pub fn read_raster(filepath: &str) -> Result<Arc<StructArray>, ArrowError> {
+ let dataset = Dataset::open(filepath.to_string())
+ .map_err(|err| ArrowError::ParseError(err.to_string()))?;
+
+ // Extract geotransform components
+ let (origin_x, origin_y, pixel_width, pixel_height, rotation_x, rotation_y) =
+ geotransform_components(&dataset)?;
+
+ let (raster_width, raster_height) = dataset.raster_size();
+
+ let (tile_width, tile_height) = tile_size(&dataset)?;
+
+ let x_tile_count = (raster_width + tile_width - 1) / tile_width;
+ let y_tile_count = (raster_height + tile_height - 1) / tile_height;
+
+ let mut raster_builder = RasterBuilder::new(x_tile_count * y_tile_count);
+ let band_count = dataset.raster_count();
+
+ for tile_y in 0..y_tile_count {
+ for tile_x in 0..x_tile_count {
+ let x_offset = tile_x * tile_width;
+ let y_offset = tile_y * tile_height;
+
+ // Calculate geographic coordinates for this tile
+ // using the geotransform from the original raster
+ let tile_origin_x =
+ origin_x + (x_offset as f64) * pixel_width + (y_offset as f64) * rotation_x;
+ let tile_origin_y =
+ origin_y + (x_offset as f64) * rotation_y + (y_offset as f64) * pixel_height;
+
+ // Create raster metadata for this tile with actual geotransform values
+ let tile_metadata = RasterMetadata {
+ width: tile_width as u64,
+ height: tile_height as u64,
+ upperleft_x: tile_origin_x,
+ upperleft_y: tile_origin_y,
+ scale_x: pixel_width,
+ scale_y: pixel_height,
+ skew_x: rotation_x,
+ skew_y: rotation_y,
+ bounding_box: None, // TODO: should we calculate bounding box here?
+ };
+
+ raster_builder.start_raster(&tile_metadata, None, None)?;
+
+ for band_number in 1..=band_count {
+ let band: RasterBand = dataset.rasterband(band_number).unwrap();
Review Comment:
We should propagate the error instead of calling `.unwrap()`.
##########
rust/sedona-gdal/src/readers.rs:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::dataset::{geotransform_components, tile_size};
+use arrow_array::StructArray;
+use arrow_schema::ArrowError;
+use gdal::raster::{GdalDataType, RasterBand};
+use gdal::Dataset;
+use sedona_raster::datatype_functions::{bytes_per_pixel, f64_to_bandtype_bytes};
+use sedona_schema::datatypes::{
+ BandDataType, BandMetadata, RasterBuilder, RasterMetadata, StorageType,
+};
+use std::sync::Arc;
+
+pub fn read_raster(filepath: &str) -> Result<Arc<StructArray>, ArrowError> {
+ let dataset = Dataset::open(filepath.to_string())
+ .map_err(|err| ArrowError::ParseError(err.to_string()))?;
+
+ // Extract geotransform components
+ let (origin_x, origin_y, pixel_width, pixel_height, rotation_x, rotation_y) =
+ geotransform_components(&dataset)?;
+
+ let (raster_width, raster_height) = dataset.raster_size();
+
+ let (tile_width, tile_height) = tile_size(&dataset)?;
+
+ let x_tile_count = (raster_width + tile_width - 1) / tile_width;
+ let y_tile_count = (raster_height + tile_height - 1) / tile_height;
+
+ let mut raster_builder = RasterBuilder::new(x_tile_count * y_tile_count);
+ let band_count = dataset.raster_count();
+
+ for tile_y in 0..y_tile_count {
+ for tile_x in 0..x_tile_count {
+ let x_offset = tile_x * tile_width;
+ let y_offset = tile_y * tile_height;
+
+ // Calculate geographic coordinates for this tile
+ // using the geotransform from the original raster
+ let tile_origin_x =
+ origin_x + (x_offset as f64) * pixel_width + (y_offset as f64) * rotation_x;
+ let tile_origin_y =
+ origin_y + (x_offset as f64) * rotation_y + (y_offset as f64) * pixel_height;
Review Comment:
We can define a struct for affine transformation parameters and some utility functions for performing affine transformations.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
+
+ /// Commit a MutableBuffer to the band data
+ pub fn commit_band_buffer(&mut self, buffer: MutableBuffer) {
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ }
+
+ /// Finish writing the current band with its metadata
+ /// TODO: The band_metadata is in the finish in the band call, but in the
+ /// start in the raster call. Make it consistent.
+ pub fn finish_band(&mut self, band_metadata: BandMetadata) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+
+ let metadata_builder = band_builder.field_builder::<StructBuilder>(0).unwrap();
+
+ if let Some(nodata) = band_metadata.nodata_value {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_value(&nodata);
+ } else {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::STORAGE_TYPE)
+ .unwrap()
+ .append_value(band_metadata.storage_type as u32);
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::DATATYPE)
+ .unwrap()
+ .append_value(band_metadata.datatype as u32);
+
+ // Handle OutDb URL
+ if let Some(url) = band_metadata.outdb_url {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_value(&url);
+ } else {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_null();
+ }
+
+ // Handle OutDb band ID
+ if let Some(band_id) = band_metadata.outdb_band_id {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_value(band_id);
+ } else {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder.append(true);
+
+ // Finish the band
+ band_builder.append(true);
+ Ok(())
+ }
+
+ /// Finish all bands for the current raster
+ pub fn finish_raster(&mut self) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(true);
+ // Mark this raster as valid (not null) in the main struct
+ self.main_builder.append(true);
+ Ok(())
+ }
+
+ /// Append raster metadata from a MetadataRef trait object
+ fn append_metadata_from_ref(&mut self, metadata: &dyn MetadataRef) -> Result<(), ArrowError> {
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ // Width
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(metadata.width());
+
+ // Height
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(metadata.height());
+
+ // Geotransform parameters
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(metadata.upper_left_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(metadata.upper_left_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(metadata.scale_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(metadata.scale_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(metadata.skew_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(metadata.skew_y());
+
+ metadata_builder.append(true);
+
+ Ok(())
+ }
+
+ /// Set the CRS for the current raster
+ pub fn set_crs(&mut self, crs: Option<&str>) -> Result<(), ArrowError> {
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ match crs {
+ Some(crs_data) => crs_builder.append_value(crs_data),
+ None => crs_builder.append_null(),
+ }
+ Ok(())
+ }
+
+ /// Append a bounding box to the current raster
+ pub fn append_bounding_box(&mut self, bbox: Option<&BoundingBox>) -> Result<(), ArrowError> {
+ let bbox_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::BBOX)
+ .unwrap();
+
+ if let Some(bbox) = bbox {
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_value(bbox.min_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_value(bbox.min_y);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_value(bbox.max_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_value(bbox.max_y);
+
+ bbox_builder.append(true);
+ } else {
+ // Append null bounding box - need to fill in null values for all fields
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder.append(false);
+ }
+ Ok(())
+ }
+
+ /// Append a null raster
+ pub fn append_null(&mut self) -> Result<(), ArrowError> {
+ // Since metadata fields are non-nullable, provide default values
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ // Mark the metadata struct as valid since it has valid values
+ metadata_builder.append(true);
+
+ // Append null CRS (now using StringBuilder instead of StringViewBuilder)
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ crs_builder.append_null();
+
+ // Append null bounding box
+ self.append_bounding_box(None)?;
+
+ // Append null bands
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(false);
+
+ // Mark this raster as null in the main struct
+ self.main_builder.append(false);
+
+ Ok(())
+ }
+
+ /// Finish building and return the constructed StructArray
+ pub fn finish(mut self) -> Result<StructArray, ArrowError> {
+ Ok(self.main_builder.finish())
+ }
+}
+
+/// Convenience wrapper for the zero-copy band writing approach
+impl RasterBuilder {
+ /// High-level method that allows for zero-copy with a callback approach
+ pub fn append_raster_with_callback<F>(
+ &mut self,
+ metadata: RasterMetadata,
+ band_count: usize,
+ mut write_bands: F,
+ ) -> Result<(), ArrowError>
+ where
+ F: FnMut(usize, &mut BinaryBuilder) -> Result<BandMetadata, ArrowError>,
+ {
+ self.start_raster(&metadata, None, metadata.bounding_box.as_ref())?;
+
+ for band_index in 0..band_count {
+ let band_metadata = {
+ let binary_builder = self.band_data_writer();
+ write_bands(band_index, binary_builder)?
+ };
+ self.finish_band(band_metadata)?;
+ }
+
+ self.finish_raster()?;
+ Ok(())
+ }
+}
+
+/// Iterator and accessor traits for reading raster data from Arrow arrays.
+///
+/// These traits provide a zero-copy interface for accessing raster metadata and band data
+/// from the Arrow-based storage format. The implementation handles both InDb and OutDbRef
+/// storage types seamlessly.
+
+/// Trait for accessing raster metadata (dimensions, geotransform, bounding box, etc.)
+pub trait MetadataRef {
+ /// Width of the raster in pixels (using u64 to match schema)
+ fn width(&self) -> u64;
+ /// Height of the raster in pixels (using u64 to match schema)
+ fn height(&self) -> u64;
+ /// X coordinate of the upper-left corner
+ fn upper_left_x(&self) -> f64;
+ /// Y coordinate of the upper-left corner
+ fn upper_left_y(&self) -> f64;
+ /// X-direction pixel size (scale)
+ fn scale_x(&self) -> f64;
+ /// Y-direction pixel size (scale)
+ fn scale_y(&self) -> f64;
+ /// X-direction skew/rotation
+ fn skew_x(&self) -> f64;
+ /// Y-direction skew/rotation
+ fn skew_y(&self) -> f64;
+}
+
+/// Implement MetadataRef for RasterMetadata to allow direct use with builder
+impl MetadataRef for RasterMetadata {
+ fn width(&self) -> u64 {
+ self.width
+ }
+ fn height(&self) -> u64 {
+ self.height
+ }
+ fn upper_left_x(&self) -> f64 {
+ self.upperleft_x
+ }
+ fn upper_left_y(&self) -> f64 {
+ self.upperleft_y
+ }
+ fn scale_x(&self) -> f64 {
+ self.scale_x
+ }
+ fn scale_y(&self) -> f64 {
+ self.scale_y
+ }
+ fn skew_x(&self) -> f64 {
+ self.skew_x
+ }
+ fn skew_y(&self) -> f64 {
+ self.skew_y
+ }
+}
+
+/// Trait for accessing individual band metadata
+pub trait BandMetadataRef {
+ /// No-data value as raw bytes (None if null)
+ fn nodata_value(&self) -> Option<&[u8]>;
+ /// Storage type (InDb, OutDbRef, etc)
+ fn storage_type(&self) -> StorageType;
+ /// Band data type (UInt8, Float32, etc.)
+ fn data_type(&self) -> BandDataType;
+ /// OutDb URL (only used when storage_type == OutDbRef)
+ fn outdb_url(&self) -> Option<&str>;
+ /// OutDb band ID (only used when storage_type == OutDbRef)
+ fn outdb_band_id(&self) -> Option<u32>;
+}
+
+/// Trait for accessing individual band data
+pub trait BandRef {
+ /// Band metadata accessor
+ fn metadata(&self) -> &dyn BandMetadataRef;
+ /// Raw band data as bytes (zero-copy access)
+ fn data(&self) -> &[u8];
+}
+
+/// Trait for accessing all bands in a raster
+pub trait BandsRef {
+ /// Number of bands in the raster
+ fn len(&self) -> usize;
+ /// Check if no bands are present
+ fn is_empty(&self) -> bool {
+ self.len() == 0
+ }
+ /// Get a specific band by number (returns None if out of bounds)
+ /// By convention, band numbers are 1-based
+ fn band(&self, number: usize) -> Option<Box<dyn BandRef + '_>>;
+ /// Iterator over all bands
+ fn iter(&self) -> BandIterator<'_>;
+}
+
+/// Trait for accessing complete raster data
+pub trait RasterRef {
+ /// Raster metadata accessor
+ fn metadata(&self) -> &dyn MetadataRef;
+ /// CRS accessor
+ fn crs(&self) -> Option<&str>;
+ /// Bands accessor
+ fn bands(&self) -> &dyn BandsRef;
+}
+
+/// Implementation of MetadataRef for Arrow StructArray
+struct MetadataRefImpl<'a> {
+ metadata_struct: &'a StructArray,
+ index: usize,
+}
+
+impl<'a> MetadataRef for MetadataRefImpl<'a> {
+ fn width(&self) -> u64 {
+ self.metadata_struct
+ .column(metadata_indices::WIDTH)
+ .as_any()
+ .downcast_ref::<UInt64Array>()
+ .unwrap()
+ .value(self.index)
Review Comment:
Calling `.column`, `.as_any` and `.downcast_ref` each time we access the metadata fields could also be a huge performance overhead. We may observe this when benchmarking `rs_width`.
##########
rust/sedona-gdal/src/readers.rs:
##########
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::dataset::{geotransform_components, tile_size};
+use arrow_array::StructArray;
+use arrow_schema::ArrowError;
+use gdal::raster::{GdalDataType, RasterBand};
+use gdal::Dataset;
+use sedona_raster::datatype_functions::{bytes_per_pixel, f64_to_bandtype_bytes};
+use sedona_schema::datatypes::{
+ BandDataType, BandMetadata, RasterBuilder, RasterMetadata, StorageType,
+};
+use std::sync::Arc;
+
+pub fn read_raster(filepath: &str) -> Result<Arc<StructArray>, ArrowError> {
+ let dataset = Dataset::open(filepath.to_string())
+ .map_err(|err| ArrowError::ParseError(err.to_string()))?;
+
+ // Extract geotransform components
+ let (origin_x, origin_y, pixel_width, pixel_height, rotation_x, rotation_y) =
+ geotransform_components(&dataset)?;
+
+ let (raster_width, raster_height) = dataset.raster_size();
+
+ let (tile_width, tile_height) = tile_size(&dataset)?;
+
+ let x_tile_count = (raster_width + tile_width - 1) / tile_width;
+ let y_tile_count = (raster_height + tile_height - 1) / tile_height;
+
+ let mut raster_builder = RasterBuilder::new(x_tile_count * y_tile_count);
+ let band_count = dataset.raster_count();
+
+ for tile_y in 0..y_tile_count {
+ for tile_x in 0..x_tile_count {
+ let x_offset = tile_x * tile_width;
+ let y_offset = tile_y * tile_height;
+
+ // Calculate geographic coordinates for this tile
+ // using the geotransform from the original raster
+ let tile_origin_x =
+ origin_x + (x_offset as f64) * pixel_width + (y_offset as f64) * rotation_x;
+ let tile_origin_y =
+ origin_y + (x_offset as f64) * rotation_y + (y_offset as f64) * pixel_height;
+
+ // Create raster metadata for this tile with actual geotransform values
+ let tile_metadata = RasterMetadata {
+ width: tile_width as u64,
+ height: tile_height as u64,
+ upperleft_x: tile_origin_x,
+ upperleft_y: tile_origin_y,
+ scale_x: pixel_width,
+ scale_y: pixel_height,
+ skew_x: rotation_x,
+ skew_y: rotation_y,
+ bounding_box: None, // TODO: should we calculate bounding box here?
+ };
+
+ raster_builder.start_raster(&tile_metadata, None, None)?;
+
+ for band_number in 1..=band_count {
+ let band: RasterBand = dataset.rasterband(band_number).unwrap();
+ // This should be the same as tile width/height, except for edge tiles
+ // but we would need to update the width/height in the metadata above then.
+ // For now, fail if sizes don't match.
+ let (x_size, y_size) = band.size();
+ if x_size != tile_width || y_size != tile_height {
+ return Err(ArrowError::ParseError(format!(
+ "Band size ({}, {}) does not match expected tile size ({}, {})",
+ x_size, y_size, tile_width, tile_height
+ )));
+ }
+
+ let data_type = gdaldatatype_to_banddatatype(band.band_type())?;
+ let data_type_bytes = bytes_per_pixel(data_type.clone())?;
+ let buffer_size_bytes = x_size * y_size * data_type_bytes.clone();
+
+ // Get a mutable buffer slice for GDAL to write directly into
+ let (buffer, slice) = raster_builder.get_band_buffer_slice(buffer_size_bytes);
+
+ // TODO: Do we need resampling? If so set buffer_size to different from window_size
+ // and have a ResampleAlgorithm.
+ band.read_into_slice(
+ (x_offset as isize, y_offset as isize), // window_origin
+ (x_size, y_size), // window_size
+ (x_size, y_size), // buffer_size (no resampling)
+ slice, // buffer
+ None, // resampling algorithms
+ )
+ .map_err(|e| {
+ ArrowError::ParseError(format!("Failed to read band {band_number} {e}"))
+ })?;
+
+ raster_builder.commit_band_buffer(buffer);
+
+ let nodata_value = match band.no_data_value() {
+ Some(val) => Some(f64_to_bandtype_bytes(val, data_type.clone())?),
+ None => None,
+ };
+
+ let band_metadata = BandMetadata {
+ nodata_value: nodata_value,
+ storage_type: StorageType::InDb,
+ datatype: data_type,
+ outdb_url: None,
+ outdb_band_id: None,
+ };
Review Comment:
We can add a parameter to control if the loaded tiles are in-db or out-db, so that we can test basic functions of out-db rasters.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
+
+ /// Commit a MutableBuffer to the band data
+ pub fn commit_band_buffer(&mut self, buffer: MutableBuffer) {
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ }
+
+ /// Finish writing the current band with its metadata
+ /// TODO: The band_metadata is in the finish in the band call, but in the
+ /// start in the raster call. Make it consistent.
+ pub fn finish_band(&mut self, band_metadata: BandMetadata) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+
+ let metadata_builder = band_builder.field_builder::<StructBuilder>(0).unwrap();
+
+ if let Some(nodata) = band_metadata.nodata_value {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_value(&nodata);
+ } else {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::STORAGE_TYPE)
+ .unwrap()
+ .append_value(band_metadata.storage_type as u32);
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::DATATYPE)
+ .unwrap()
+ .append_value(band_metadata.datatype as u32);
+
+ // Handle OutDb URL
+ if let Some(url) = band_metadata.outdb_url {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_value(&url);
+ } else {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_null();
+ }
+
+ // Handle OutDb band ID
+ if let Some(band_id) = band_metadata.outdb_band_id {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_value(band_id);
+ } else {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder.append(true);
+
+ // Finish the band
+ band_builder.append(true);
+ Ok(())
+ }
+
+ /// Finish all bands for the current raster
+ pub fn finish_raster(&mut self) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(true);
+ // Mark this raster as valid (not null) in the main struct
+ self.main_builder.append(true);
+ Ok(())
+ }
+
+ /// Append raster metadata from a MetadataRef trait object
+ fn append_metadata_from_ref(&mut self, metadata: &dyn MetadataRef) -> Result<(), ArrowError> {
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ // Width
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(metadata.width());
+
+ // Height
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(metadata.height());
+
+ // Geotransform parameters
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(metadata.upper_left_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(metadata.upper_left_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(metadata.scale_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(metadata.scale_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(metadata.skew_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(metadata.skew_y());
+
+ metadata_builder.append(true);
+
+ Ok(())
+ }
+
+ /// Set the CRS for the current raster
+ pub fn set_crs(&mut self, crs: Option<&str>) -> Result<(), ArrowError> {
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ match crs {
+ Some(crs_data) => crs_builder.append_value(crs_data),
+ None => crs_builder.append_null(),
+ }
+ Ok(())
+ }
+
+ /// Append a bounding box to the current raster
+ pub fn append_bounding_box(&mut self, bbox: Option<&BoundingBox>) -> Result<(), ArrowError> {
+ let bbox_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::BBOX)
+ .unwrap();
+
+ if let Some(bbox) = bbox {
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_value(bbox.min_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_value(bbox.min_y);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_value(bbox.max_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_value(bbox.max_y);
+
+ bbox_builder.append(true);
+ } else {
+ // Append null bounding box - need to fill in null values for all fields
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder.append(false);
+ }
+ Ok(())
+ }
+
+ /// Append a null raster
+ pub fn append_null(&mut self) -> Result<(), ArrowError> {
+ // Since metadata fields are non-nullable, provide default values
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ // Mark the metadata struct as valid since it has valid values
+ metadata_builder.append(true);
+
+ // Append null CRS (now using StringBuilder instead of StringViewBuilder)
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ crs_builder.append_null();
+
+ // Append null bounding box
+ self.append_bounding_box(None)?;
+
+ // Append null bands
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(false);
+
+ // Mark this raster as null in the main struct
+ self.main_builder.append(false);
+
+ Ok(())
+ }
+
+ /// Finish building and return the constructed StructArray
+ pub fn finish(mut self) -> Result<StructArray, ArrowError> {
+ Ok(self.main_builder.finish())
+ }
+}
+
+/// Convenience wrapper for the zero-copy band writing approach
+impl RasterBuilder {
+ /// High-level method that allows for zero-copy with a callback approach
+ pub fn append_raster_with_callback<F>(
+ &mut self,
+ metadata: RasterMetadata,
+ band_count: usize,
+ mut write_bands: F,
+ ) -> Result<(), ArrowError>
+ where
+ F: FnMut(usize, &mut BinaryBuilder) -> Result<BandMetadata, ArrowError>,
+ {
+ self.start_raster(&metadata, None, metadata.bounding_box.as_ref())?;
+
+ for band_index in 0..band_count {
+ let band_metadata = {
+ let binary_builder = self.band_data_writer();
+ write_bands(band_index, binary_builder)?
+ };
+ self.finish_band(band_metadata)?;
+ }
+
+ self.finish_raster()?;
+ Ok(())
+ }
+}
+
+/// Iterator and accessor traits for reading raster data from Arrow arrays.
+///
+/// These traits provide a zero-copy interface for accessing raster metadata and band data
+/// from the Arrow-based storage format. The implementation handles both InDb and OutDbRef
+/// storage types seamlessly.
+
+/// Trait for accessing raster metadata (dimensions, geotransform, bounding box, etc.)
+pub trait MetadataRef {
+ /// Width of the raster in pixels (using u64 to match schema)
+ fn width(&self) -> u64;
+ /// Height of the raster in pixels (using u64 to match schema)
+ fn height(&self) -> u64;
+ /// X coordinate of the upper-left corner
+ fn upper_left_x(&self) -> f64;
+ /// Y coordinate of the upper-left corner
+ fn upper_left_y(&self) -> f64;
+ /// X-direction pixel size (scale)
+ fn scale_x(&self) -> f64;
+ /// Y-direction pixel size (scale)
+ fn scale_y(&self) -> f64;
+ /// X-direction skew/rotation
+ fn skew_x(&self) -> f64;
+ /// Y-direction skew/rotation
+ fn skew_y(&self) -> f64;
+}
+
+/// Implement MetadataRef for RasterMetadata to allow direct use with builder
+impl MetadataRef for RasterMetadata {
+ fn width(&self) -> u64 {
+ self.width
+ }
+ fn height(&self) -> u64 {
+ self.height
+ }
+ fn upper_left_x(&self) -> f64 {
+ self.upperleft_x
+ }
+ fn upper_left_y(&self) -> f64 {
+ self.upperleft_y
+ }
+ fn scale_x(&self) -> f64 {
+ self.scale_x
+ }
+ fn scale_y(&self) -> f64 {
+ self.scale_y
+ }
+ fn skew_x(&self) -> f64 {
+ self.skew_x
+ }
+ fn skew_y(&self) -> f64 {
+ self.skew_y
+ }
+}
+
+/// Trait for accessing individual band metadata
+pub trait BandMetadataRef {
+ /// No-data value as raw bytes (None if null)
+ fn nodata_value(&self) -> Option<&[u8]>;
+ /// Storage type (InDb, OutDbRef, etc)
+ fn storage_type(&self) -> StorageType;
+ /// Band data type (UInt8, Float32, etc.)
+ fn data_type(&self) -> BandDataType;
+ /// OutDb URL (only used when storage_type == OutDbRef)
+ fn outdb_url(&self) -> Option<&str>;
+ /// OutDb band ID (only used when storage_type == OutDbRef)
+ fn outdb_band_id(&self) -> Option<u32>;
+}
+
+/// Trait for accessing individual band data
+pub trait BandRef {
+ /// Band metadata accessor
+ fn metadata(&self) -> &dyn BandMetadataRef;
+ /// Raw band data as bytes (zero-copy access)
+ fn data(&self) -> &[u8];
+}
+
+/// Trait for accessing all bands in a raster
+pub trait BandsRef {
+ /// Number of bands in the raster
+ fn len(&self) -> usize;
+ /// Check if no bands are present
+ fn is_empty(&self) -> bool {
+ self.len() == 0
+ }
+ /// Get a specific band by number (returns None if out of bounds)
+ /// By convention, band numbers are 1-based
+ fn band(&self, number: usize) -> Option<Box<dyn BandRef + '_>>;
+ /// Iterator over all bands
+ fn iter(&self) -> BandIterator<'_>;
+}
+
+/// Trait for accessing complete raster data
+pub trait RasterRef {
+ /// Raster metadata accessor
+ fn metadata(&self) -> &dyn MetadataRef;
+ /// CRS accessor
+ fn crs(&self) -> Option<&str>;
+ /// Bands accessor
+ fn bands(&self) -> &dyn BandsRef;
+}
+
+/// Implementation of MetadataRef for Arrow StructArray
+struct MetadataRefImpl<'a> {
+ metadata_struct: &'a StructArray,
+ index: usize,
+}
+
+impl<'a> MetadataRef for MetadataRefImpl<'a> {
+ fn width(&self) -> u64 {
+ self.metadata_struct
+ .column(metadata_indices::WIDTH)
+ .as_any()
+ .downcast_ref::<UInt64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn height(&self) -> u64 {
+ self.metadata_struct
+ .column(metadata_indices::HEIGHT)
+ .as_any()
+ .downcast_ref::<UInt64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn upper_left_x(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::UPPERLEFT_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn upper_left_y(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::UPPERLEFT_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn scale_x(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SCALE_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn scale_y(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SCALE_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn skew_x(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SKEW_X)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+
+ fn skew_y(&self) -> f64 {
+ self.metadata_struct
+ .column(metadata_indices::SKEW_Y)
+ .as_any()
+ .downcast_ref::<Float64Array>()
+ .unwrap()
+ .value(self.index)
+ }
+}
+
+/// Implementation of BandMetadataRef for Arrow StructArray
+struct BandMetadataRefImpl<'a> {
+ metadata_struct: &'a StructArray,
+ band_index: usize,
+}
+
+impl<'a> BandMetadataRef for BandMetadataRefImpl<'a> {
+ fn nodata_value(&self) -> Option<&[u8]> {
+ let nodata_array = self
+ .metadata_struct
+ .column(band_metadata_indices::NODATAVALUE)
+ .as_any()
+ .downcast_ref::<BinaryArray>()
+ .expect("Expected BinaryArray for nodata");
+
+ if nodata_array.is_null(self.band_index) {
+ None
+ } else {
+ Some(nodata_array.value(self.band_index))
+ }
+ }
+
+ fn storage_type(&self) -> StorageType {
+ let storage_type_array = self
+ .metadata_struct
+ .column(band_metadata_indices::STORAGE_TYPE)
+ .as_any()
+ .downcast_ref::<UInt32Array>()
+ .expect("Expected UInt32Array for storage_type");
+
+ match storage_type_array.value(self.band_index) {
+ 0 => StorageType::InDb,
+ 1 => StorageType::OutDbRef,
+ _ => panic!(
+ "Unknown storage type: {}",
+ storage_type_array.value(self.band_index)
+ ),
+ }
+ }
+
+ fn data_type(&self) -> BandDataType {
+ let datatype_array = self
+ .metadata_struct
+ .column(band_metadata_indices::DATATYPE)
+ .as_any()
+ .downcast_ref::<UInt32Array>()
+ .expect("Expected UInt32Array for datatype");
+
+ match datatype_array.value(self.band_index) {
+ 0 => BandDataType::UInt8,
+ 1 => BandDataType::UInt16,
+ 2 => BandDataType::Int16,
+ 3 => BandDataType::UInt32,
+ 4 => BandDataType::Int32,
+ 5 => BandDataType::Float32,
+ 6 => BandDataType::Float64,
+ _ => panic!(
+ "Unknown band data type: {}",
+ datatype_array.value(self.band_index)
+ ),
+ }
+ }
+
+ fn outdb_url(&self) -> Option<&str> {
+ let url_array = self
+ .metadata_struct
+ .column(band_metadata_indices::OUTDB_URL)
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("Expected StringArray for outdb_url");
+
+ if url_array.is_null(self.band_index) {
+ None
+ } else {
+ Some(url_array.value(self.band_index))
+ }
+ }
+
+ fn outdb_band_id(&self) -> Option<u32> {
+ let band_id_array = self
+ .metadata_struct
+ .column(band_metadata_indices::OUTDB_BAND_ID)
+ .as_any()
+ .downcast_ref::<UInt32Array>()
+ .expect("Expected UInt32Array for outdb_band_id");
+
+ if band_id_array.is_null(self.band_index) {
+ None
+ } else {
+ Some(band_id_array.value(self.band_index))
+ }
+ }
+}
+
+/// Implementation of BandRef for accessing individual band data
+struct BandRefImpl<'a> {
+ band_metadata: BandMetadataRefImpl<'a>,
+ band_data: &'a [u8],
+}
+
+impl<'a> BandRef for BandRefImpl<'a> {
+ fn metadata(&self) -> &dyn BandMetadataRef {
+ &self.band_metadata
+ }
+
+ fn data(&self) -> &[u8] {
+ self.band_data
+ }
+}
+
+/// Implementation of BandsRef for accessing all bands in a raster
+struct BandsRefImpl<'a> {
+ bands_list: &'a ListArray,
+ raster_index: usize,
+}
+
+impl<'a> BandsRef for BandsRefImpl<'a> {
+ fn len(&self) -> usize {
+ let start = self.bands_list.value_offsets()[self.raster_index] as usize;
+ let end = self.bands_list.value_offsets()[self.raster_index + 1] as usize;
+ end - start
+ }
+
+ /// Get a specific band by number (1-based index)
+ fn band(&self, number: usize) -> Option<Box<dyn BandRef + '_>> {
+ // TODO: Error instead of returning None for out-of-bounds
+ if number == 0 {
+ return None; // Band numbers are 1-based
+ }
+ // By convention, band numbers are 1-based.
+ // Convert to zero-based index.
+ let index = number - 1;
+ if index >= self.len() {
+ return None;
+ }
+
+ let start = self.bands_list.value_offsets()[self.raster_index] as usize;
+ let band_row = start + index;
+
+ let bands_struct = self
+ .bands_list
+ .values()
+ .as_any()
+ .downcast_ref::<StructArray>()?;
+
+ // Get the metadata substructure from the band struct
+ let band_metadata_struct = bands_struct
+ .column(band_indices::METADATA)
+ .as_any()
+ .downcast_ref::<StructArray>()?;
+
+ let band_metadata = BandMetadataRefImpl {
+ metadata_struct: band_metadata_struct,
+ band_index: band_row,
+ };
+
+ // Get band data from the Binary column within the band struct
+ let band_data_array = bands_struct
+ .column(band_indices::DATA)
+ .as_any()
+ .downcast_ref::<BinaryArray>()?;
+
+ let band_data = band_data_array.value(band_row);
+
+ Some(Box::new(BandRefImpl {
+ band_metadata,
+ band_data,
+ }))
+ }
+
+ fn iter(&self) -> BandIterator<'_> {
+ BandIterator {
+ bands: self,
+ current: 1, // Start at 1 for 1-based band numbering
+ }
+ }
+}
+
+/// Iterator for bands within a raster
+pub struct BandIterator<'a> {
+ bands: &'a dyn BandsRef,
+ current: usize,
+}
+
+impl<'a> Iterator for BandIterator<'a> {
+ type Item = Box<dyn BandRef + 'a>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ // current is 1-based, compare against len() + 1
+ if self.current <= self.bands.len() {
+ let band = self.bands.band(self.current);
+ self.current += 1;
+ band
+ } else {
+ None
+ }
+ }
+
+ fn size_hint(&self) -> (usize, Option<usize>) {
+ // current is 1-based, so remaining calculation needs adjustment
+ let remaining = self.bands.len().saturating_sub(self.current - 1);
+ (remaining, Some(remaining))
+ }
+}
+
+impl ExactSizeIterator for BandIterator<'_> {}
+
+/// Implementation of RasterRef for complete raster access
+pub struct RasterRefImpl<'a> {
+ metadata: MetadataRefImpl<'a>,
+ crs: &'a StringArray,
+ bbox: &'a StructArray,
+ bands: BandsRefImpl<'a>,
+}
+
+impl<'a> RasterRefImpl<'a> {
+ /// Create a new RasterRefImpl from a struct array and index using hard-coded indices
+ pub fn new(raster_struct: &'a StructArray, raster_index: usize) -> Self {
+ let metadata_struct = raster_struct
+ .column(raster_indices::METADATA)
+ .as_any()
+ .downcast_ref::<StructArray>()
+ .unwrap();
+
+ let crs = raster_struct
+ .column(raster_indices::CRS)
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+
+ let bbox = raster_struct
+ .column(raster_indices::BBOX)
+ .as_any()
+ .downcast_ref::<StructArray>()
+ .unwrap();
+
+ let bands_list = raster_struct
+ .column(raster_indices::BANDS)
+ .as_any()
+ .downcast_ref::<ListArray>()
+ .unwrap();
Review Comment:
Performing all these `.column`, `.as_any` and `downcast_ref` for each raster could be a huge performance overhead, especially for queries that only fetches the metadata.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
+
+impl RasterBuilder {
+ /// Create a new raster builder with the specified capacity
+ pub fn new(capacity: usize) -> Self {
+ // Create individual builders that we know work
+ let metadata_builder = StructBuilder::from_fields(
+ match RasterSchema::metadata_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for metadata"),
+ },
+ capacity,
+ );
+
+ let crs_builder = StringBuilder::new();
+
+ let bbox_builder = StructBuilder::from_fields(
+ match RasterSchema::bounding_box_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for bounding box"),
+ },
+ capacity,
+ );
+
+ let band_struct_builder = StructBuilder::from_fields(
+ match RasterSchema::band_type() {
+ DataType::Struct(fields) => fields,
+ _ => panic!("Expected struct type for band"),
+ },
+ 0,
+ );
+
+ let bands_builder = ListBuilder::new(band_struct_builder).with_field(Field::new(
+ column::BAND,
+ RasterSchema::band_type(),
+ false,
+ ));
+
+ // Now create the main builder with pre-built components
+ let main_builder = StructBuilder::new(
+ RasterSchema::fields(),
+ vec![
+ Box::new(metadata_builder),
+ Box::new(crs_builder),
+ Box::new(bbox_builder),
+ Box::new(bands_builder),
+ ],
+ );
+
+ Self { main_builder }
+ }
+
+ /// Start a new raster with metadata, optional CRS, and optional bounding box
+ ///
+ /// This is the unified method for starting a raster with all optional parameters.
+ ///
+ /// # Arguments
+ /// * `metadata` - Raster metadata (dimensions, geotransform parameters)
+ /// * `crs` - Optional coordinate reference system as string
+ /// * `bbox` - Optional bounding box coordinates
+ ///
+ /// # Examples
+ /// // From iterator - copy all fields from existing raster
+ /// builder.start_raster(raster.metadata(), raster.crs(), raster.bounding_box(0).as_ref())?;
+ ///
+ /// // From RasterMetadata struct with all fields
+ /// builder.start_raster(&metadata, Some("EPSG:4326"), metadata.bounding_box.as_ref())?;
+ ///
+ /// // Minimal - just metadata
+ /// builder.start_raster(&metadata, None, None)?;
+ /// ```
+ pub fn start_raster(
+ &mut self,
+ metadata: &dyn MetadataRef,
+ crs: Option<&str>,
+ bbox: Option<&BoundingBox>,
+ ) -> Result<(), ArrowError> {
+ self.append_metadata_from_ref(metadata)?;
+ self.set_crs(crs)?;
+ self.append_bounding_box(bbox)?;
+ Ok(())
+ }
+
+ /// Get direct access to the BinaryBuilder for writing the current band's data
+ pub fn band_data_writer(&mut self) -> &mut BinaryBuilder {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+ // Ensure we have at least one field (band metadata and data)
+ // Field 0 = metadata (StructBuilder), Field 1 = data (BinaryBuilder)
+ band_builder.field_builder::<BinaryBuilder>(1).unwrap()
+ }
+
+ /// Create a MutableBuffer that can be written to directly
+ pub fn create_band_buffer(
+ &mut self,
+ capacity: usize,
+ ) -> (MutableBuffer, impl FnOnce(MutableBuffer) + '_) {
+ let mut buffer = MutableBuffer::with_capacity(capacity);
+
+ // Pre-allocate the buffer to the exact size
+ buffer.resize(capacity, 0);
+
+ let commit = move |buffer: MutableBuffer| {
+ // Convert MutableBuffer to &[u8] and append to BinaryBuilder
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ };
+
+ (buffer, commit)
+ }
+
+ /// Alternative: Get a mutable slice from a MutableBuffer for GDAL
+ /// This provides the most direct access for zero-copy operations
+ /// TODO: have this 3 different way.... pick one!!
+ pub fn get_band_buffer_slice(&mut self, size: usize) -> (MutableBuffer, &mut [u8]) {
+ let mut buffer = MutableBuffer::with_capacity(size);
+ buffer.resize(size, 0);
+
+ // Get mutable slice that GDAL can write to
+ let slice = unsafe {
+ // This is safe because we just allocated the buffer with the exact size
+ std::slice::from_raw_parts_mut(buffer.as_mut_ptr(), size)
+ };
+
+ (buffer, slice)
+ }
+
+ /// Commit a MutableBuffer to the band data
+ pub fn commit_band_buffer(&mut self, buffer: MutableBuffer) {
+ let data = buffer.as_slice();
+ self.band_data_writer().append_value(data);
+ }
+
+ /// Finish writing the current band with its metadata
+ /// TODO: The band_metadata is in the finish in the band call, but in the
+ /// start in the raster call. Make it consistent.
+ pub fn finish_band(&mut self, band_metadata: BandMetadata) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ let band_builder = bands_builder.values();
+
+ let metadata_builder = band_builder.field_builder::<StructBuilder>(0).unwrap();
+
+ if let Some(nodata) = band_metadata.nodata_value {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_value(&nodata);
+ } else {
+ metadata_builder
+ .field_builder::<BinaryBuilder>(band_metadata_indices::NODATAVALUE)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::STORAGE_TYPE)
+ .unwrap()
+ .append_value(band_metadata.storage_type as u32);
+
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::DATATYPE)
+ .unwrap()
+ .append_value(band_metadata.datatype as u32);
+
+ // Handle OutDb URL
+ if let Some(url) = band_metadata.outdb_url {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_value(&url);
+ } else {
+ metadata_builder
+ .field_builder::<StringBuilder>(band_metadata_indices::OUTDB_URL)
+ .unwrap()
+ .append_null();
+ }
+
+ // Handle OutDb band ID
+ if let Some(band_id) = band_metadata.outdb_band_id {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_value(band_id);
+ } else {
+ metadata_builder
+ .field_builder::<UInt32Builder>(band_metadata_indices::OUTDB_BAND_ID)
+ .unwrap()
+ .append_null();
+ }
+
+ metadata_builder.append(true);
+
+ // Finish the band
+ band_builder.append(true);
+ Ok(())
+ }
+
+ /// Finish all bands for the current raster
+ pub fn finish_raster(&mut self) -> Result<(), ArrowError> {
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(true);
+ // Mark this raster as valid (not null) in the main struct
+ self.main_builder.append(true);
+ Ok(())
+ }
+
+ /// Append raster metadata from a MetadataRef trait object
+ fn append_metadata_from_ref(&mut self, metadata: &dyn MetadataRef) -> Result<(), ArrowError> {
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ // Width
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(metadata.width());
+
+ // Height
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(metadata.height());
+
+ // Geotransform parameters
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(metadata.upper_left_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(metadata.upper_left_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(metadata.scale_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(metadata.scale_y());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(metadata.skew_x());
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(metadata.skew_y());
+
+ metadata_builder.append(true);
+
+ Ok(())
+ }
+
+ /// Set the CRS for the current raster
+ pub fn set_crs(&mut self, crs: Option<&str>) -> Result<(), ArrowError> {
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ match crs {
+ Some(crs_data) => crs_builder.append_value(crs_data),
+ None => crs_builder.append_null(),
+ }
+ Ok(())
+ }
+
+ /// Append a bounding box to the current raster
+ pub fn append_bounding_box(&mut self, bbox: Option<&BoundingBox>) -> Result<(), ArrowError> {
+ let bbox_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::BBOX)
+ .unwrap();
+
+ if let Some(bbox) = bbox {
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_value(bbox.min_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_value(bbox.min_y);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_value(bbox.max_x);
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_value(bbox.max_y);
+
+ bbox_builder.append(true);
+ } else {
+ // Append null bounding box - need to fill in null values for all fields
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MIN_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_X)
+ .unwrap()
+ .append_null();
+
+ bbox_builder
+ .field_builder::<Float64Builder>(bounding_box_indices::MAX_Y)
+ .unwrap()
+ .append_null();
+
+ bbox_builder.append(false);
+ }
+ Ok(())
+ }
+
+ /// Append a null raster
+ pub fn append_null(&mut self) -> Result<(), ArrowError> {
+ // Since metadata fields are non-nullable, provide default values
+ let metadata_builder = self
+ .main_builder
+ .field_builder::<StructBuilder>(raster_indices::METADATA)
+ .unwrap();
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::WIDTH)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<UInt64Builder>(metadata_indices::HEIGHT)
+ .unwrap()
+ .append_value(0u64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::UPPERLEFT_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SCALE_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_X)
+ .unwrap()
+ .append_value(0.0f64);
+
+ metadata_builder
+ .field_builder::<Float64Builder>(metadata_indices::SKEW_Y)
+ .unwrap()
+ .append_value(0.0f64);
+
+ // Mark the metadata struct as valid since it has valid values
+ metadata_builder.append(true);
+
+ // Append null CRS (now using StringBuilder instead of StringViewBuilder)
+ let crs_builder = self
+ .main_builder
+ .field_builder::<StringBuilder>(raster_indices::CRS)
+ .unwrap();
+ crs_builder.append_null();
+
+ // Append null bounding box
+ self.append_bounding_box(None)?;
+
+ // Append null bands
+ let bands_builder = self
+ .main_builder
+ .field_builder::<ListBuilder<StructBuilder>>(raster_indices::BANDS)
+ .unwrap();
+ bands_builder.append(false);
+
+ // Mark this raster as null in the main struct
+ self.main_builder.append(false);
+
+ Ok(())
+ }
+
+ /// Finish building and return the constructed StructArray
+ pub fn finish(mut self) -> Result<StructArray, ArrowError> {
+ Ok(self.main_builder.finish())
+ }
+}
+
+/// Convenience wrapper for the zero-copy band writing approach
+impl RasterBuilder {
+ /// High-level method that allows for zero-copy with a callback approach
+ pub fn append_raster_with_callback<F>(
+ &mut self,
+ metadata: RasterMetadata,
+ band_count: usize,
+ mut write_bands: F,
+ ) -> Result<(), ArrowError>
+ where
+ F: FnMut(usize, &mut BinaryBuilder) -> Result<BandMetadata, ArrowError>,
+ {
+ self.start_raster(&metadata, None, metadata.bounding_box.as_ref())?;
+
+ for band_index in 0..band_count {
+ let band_metadata = {
+ let binary_builder = self.band_data_writer();
+ write_bands(band_index, binary_builder)?
+ };
+ self.finish_band(band_metadata)?;
+ }
+
+ self.finish_raster()?;
+ Ok(())
+ }
+}
Review Comment:
The method signature looks cubersome and it is currently unused in the codebase. How it will be used by other raster functions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prototype raster [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #249:
URL: https://github.com/apache/sedona-db/pull/249#discussion_r2469976727
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
Review Comment:
I think a Utf8View here is a good choice since this value will be frequently repeated, less than 12 bytes, or both!
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -29,6 +39,7 @@ pub enum SedonaType {
Arrow(DataType),
Wkb(Edges, Crs),
WkbView(Edges, Crs),
+ Raster(RasterSchema),
Review Comment:
I think this can probably be unparameterized (i.e., `SedonaType::Raster`). My reading of this PR is that the underlying Arrow schema is a constant (so it can be cloned from RASTER_DATATYPE when requested by `storage_type()`).
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
Review Comment:
A potential future consideration, that it might be just as fast to do something like:
```rust
#[derive(Serialize, Deserialize)
struct RasterMetadata {
shape: [u64; 2],
upper_left: [f64; 2],
geo_transform: [f64; 6],
}
```
...and use `serde_json` to read and write from JSON. (i.e., the `metadata` field would just be `Utf8`).
You've done a great job abstracting this so that nobody ever has to manually create or read this and so it should be easy to experiment with changing it in the future 🙂
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
Review Comment:
With JSON storage this could also collapse to:
```rust
#[derive(Serialize, Deserialize)]
struct BandMetadata {
nodata_value: Vec<u8>,
storage_type: StorageType,
data_type: BandDataType,
#[serde(skip_serializing_if = None)]
outdb_url: Option<String>,
#[serde(skip_serializing_if = None)]
outdb_band_id: Option<String>
}
```
(Again, this is abstracted nicely so we don't have to consider this now!)
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
Review Comment:
I think it might be nice to have this live in a dedicated `sedona-schema/src/raster.rs` (arguably we should move the Wkb types into their own file as well in some future 😬 )
##########
rust/sedona-functions/src/executor.rs:
##########
@@ -75,6 +75,116 @@ pub struct GenericExecutor<'a, 'b, Factory0, Factory1> {
/// Alias for an executor that iterates over geometries as [Wkb]
pub type WkbExecutor<'a, 'b> = GenericExecutor<'a, 'b, WkbGeometryFactory, WkbGeometryFactory>;
+/// Helper for writing raster kernel implementations
+///
+/// The [RasterExecutor] provides a simplified interface for executing functions
+/// on raster arrays, handling the common pattern of downcasting to StructArray,
+/// creating raster iterators, and handling null values.
+pub struct RasterExecutor<'a, 'b> {
+ pub arg_types: &'a [SedonaType],
+ pub args: &'b [ColumnarValue],
+ num_iterations: usize,
+}
Review Comment:
I think having a `sedona-raster-functions` crate would make sense! (sedona-functions is already a little crowded and might benefit from being split even in its current form)
##########
rust/sedona-gdal/src/rs_value.rs:
##########
@@ -0,0 +1,443 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use std::sync::Arc;
+
+use crate::dataset::outdb_dataset;
+use arrow_array::builder::Float64Builder;
+use arrow_schema::{ArrowError, DataType};
+use datafusion_common::{error::Result, scalar::ScalarValue};
+use datafusion_expr::ColumnarValue;
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_functions::executor::RasterExecutor;
+use sedona_raster::datatype_functions::{bytes_per_pixel, read_pixel_value};
+use sedona_schema::datatypes::{BandMetadataRef, BandRef, RasterRef, SedonaType, StorageType};
+
+/// RS_Value() implementation
+pub fn rs_value_impl() -> ScalarKernelRef {
+ Arc::new(RSValue {})
+}
Review Comment:
This is probably something we want to be an async scalar function (which we'd need to implement a Sedona wrapper for to get the kernel dispatch thing working): https://datafusion.apache.org/library-user-guide/functions/adding-udfs.html#adding-an-async-scalar-udf . That would provide a mechanism for doing the IO required in parallel and a way to tell DataFusion that operating in small batches is a good idea.
##########
rust/sedona-schema/src/datatypes.rs:
##########
@@ -333,6 +365,1241 @@ fn deserialize_edges(edges: &Value) -> Result<Edges> {
}
}
+/// Schema for storing raster data in Apache Arrow format.
+/// Utilizing nested structs and lists to represent raster metadata and bands.
+#[derive(Debug, PartialEq, Clone)]
+pub struct RasterSchema;
+impl RasterSchema {
+ // Raster schema:
+ pub fn fields() -> Fields {
+ Fields::from(vec![
+ Field::new(column::METADATA, Self::metadata_type(), false),
+ Field::new(column::CRS, Self::crs_type(), true),
+ Field::new(column::BBOX, Self::bounding_box_type(), true),
+ Field::new(column::BANDS, Self::bands_type(), true),
+ ])
+ }
+
+ /// Raster metadata schema
+ pub fn metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ // Raster dimensions
+ Field::new(column::WIDTH, DataType::UInt64, false),
+ Field::new(column::HEIGHT, DataType::UInt64, false),
+ // Geospatial transformation parameters
+ Field::new(column::UPPERLEFT_X, DataType::Float64, false),
+ Field::new(column::UPPERLEFT_Y, DataType::Float64, false),
+ Field::new(column::SCALE_X, DataType::Float64, false),
+ Field::new(column::SCALE_Y, DataType::Float64, false),
+ Field::new(column::SKEW_X, DataType::Float64, false),
+ Field::new(column::SKEW_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bounding box schema
+ pub fn bounding_box_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::MIN_X, DataType::Float64, false),
+ Field::new(column::MIN_Y, DataType::Float64, false),
+ Field::new(column::MAX_X, DataType::Float64, false),
+ Field::new(column::MAX_Y, DataType::Float64, false),
+ ]))
+ }
+
+ /// Bands list schema
+ pub fn bands_type() -> DataType {
+ DataType::List(FieldRef::new(Field::new(
+ column::BAND,
+ Self::band_type(),
+ false,
+ )))
+ }
+
+ /// Individual band schema
+ pub fn band_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::METADATA, Self::band_metadata_type(), false),
+ Field::new(column::DATA, Self::band_data_type(), false),
+ ]))
+ }
+
+ /// Band metadata schema
+ pub fn band_metadata_type() -> DataType {
+ DataType::Struct(Fields::from(vec![
+ Field::new(column::NODATAVALUE, DataType::Binary, true), // Allow null nodata values
+ Field::new(column::STORAGE_TYPE, DataType::UInt32, false),
+ Field::new(column::DATATYPE, DataType::UInt32, false),
+ // OutDb reference fields - only used when storage_type == OutDbRef
+ Field::new(column::OUTDB_URL, DataType::Utf8, true),
+ Field::new(column::OUTDB_BAND_ID, DataType::UInt32, true),
+ ]))
+ }
+
+ /// Band data schema (single binary blob)
+ pub fn band_data_type() -> DataType {
+ DataType::Binary // consider switching to BinaryView
+ }
+
+ /// CRS schema to store json representation
+ pub fn crs_type() -> DataType {
+ DataType::Utf8 // TODO: Consider Utf8View
+ }
+}
+
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum BandDataType {
+ UInt8 = 0,
+ UInt16 = 1,
+ Int16 = 2,
+ UInt32 = 3,
+ Int32 = 4,
+ Float32 = 5,
+ Float64 = 6,
+ // Consider support for complex types for scientific data
+}
+
+/// Storage strategy for raster band data within Apache Arrow arrays.
+///
+/// This enum defines how raster data is physically stored and accessed:
+///
+/// **InDb**: Raster data is embedded directly in the Arrow array as binary blobs.
+/// - Self-contained, no external dependencies, fast access for small-medium rasters
+/// - Increases Arrow array size, memory usage grows and copy times increase with raster size
+/// - Best for: Tiles, thumbnails, processed results, small rasters (<10MB per band)
+///
+/// **OutDbRef**: Raster data is stored externally with references in the Arrow array.
+/// - Keeps Arrow arrays lightweight, supports massive rasters, enables lazy loading
+/// - Requires external storage management, potential for broken references
+/// - Best for: Large satellite imagery, time series data, cloud-native workflows
+/// - Supported backends: S3, GCS, Azure Blob, local filesystem, HTTP endpoints
+#[repr(u16)]
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StorageType {
+ InDb = 0,
+ OutDbRef = 1,
+}
+
+/// Builder for constructing raster arrays with zero-copy band data writing
+pub struct RasterBuilder {
+ main_builder: StructBuilder,
+}
Review Comment:
Nice! I think your sedona-raster crate would be a good home for this one. I know our existing sedona-geometry doesn't depend on Arrow but I think that's fine for sedona-raster (we can move these things around if there's good reason to do so later on).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org