You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2025/05/28 08:09:45 UTC

[PR] feat: implement avro file reader [iceberg-cpp]

wgtmac opened a new pull request, #113:
URL: https://github.com/apache/iceberg-cpp/pull/113

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#issuecomment-2935107822

   @mapleFU Sounds good. I think the current implementation aims to provide a minimal Avro reader implementation. We can revisit this once we have complete Avro and Parquet reader impls and make them robust.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115790873


##########
src/iceberg/expected.h:
##########
@@ -122,7 +122,7 @@ inline constexpr bool is_error_type_valid_v = is_error_type_valid<T>::value;
 }  // namespace expected_detail
 
 template <class E>
-class ICEBERG_EXPORT [[nodiscard]] unexpected {

Review Comment:
   Hmm, if it works it works. I really don't get how the Windows model works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#issuecomment-2935121277

   We can just move forward fast and leave this as a minor todo


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115359723


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {
+    if (reader_ != nullptr) {
+      reader_->close();
+      reader_.reset();
+    }
+    context_.reset();
+    return {};
+  }
+
+ private:
+  Status InitReadContext() {
+    context_ = std::make_unique<ReadContext>();
+    context_->datum_ = std::make_unique<::avro::GenericDatum>(reader_->readerSchema());
+
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    auto import_result = ::arrow::ImportSchema(&arrow_schema);
+    if (!import_result.ok()) {
+      return InvalidSchema("Failed to import the arrow schema: {}",
+                           import_result.status().message());
+    }
+    context_->arrow_schema_ = import_result.MoveValueUnsafe();
+
+    auto arrow_struct_type =
+        std::make_shared<::arrow::StructType>(context_->arrow_schema_->fields());
+    auto builder_result = ::arrow::MakeBuilder(arrow_struct_type);
+    if (!builder_result.ok()) {

Review Comment:
   Not required for this PR but maybe we should consider a way to convert between Arrow and Iceberg Result?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2125286726


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping

Review Comment:
   Yes, I think we just need to pass `NameMapping` object (if available) via ReaderOptions to read data files. For manifest and manifest list files, it should error out when field ids are missing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2125288754


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;

Review Comment:
   Sounds good to me. I just blindly chose a default value for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2124737995


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;

Review Comment:
   Looking at the TODO, we can also defer this to a later PR. Implementations aim to create manifest files of ~8MB, and performance-wise wise it is best to read them all the way directly. The manifest list can be unbounded (theoretically).



##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping

Review Comment:
   Keep in mind that name-mapping only applies to data-files



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#issuecomment-2934673162

   Personally I prefer set a internal `status_` when error happens, and when `Next` is called, internal `status_` is checked before doing operations. Since this is generally a batch interface, this would not harm the performance and making using reader more robustness. However, current impl also lgtm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115383164


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {
+    if (reader_ != nullptr) {
+      reader_->close();
+      reader_.reset();
+    }
+    context_.reset();
+    return {};
+  }
+
+ private:
+  Status InitReadContext() {
+    context_ = std::make_unique<ReadContext>();
+    context_->datum_ = std::make_unique<::avro::GenericDatum>(reader_->readerSchema());
+
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    auto import_result = ::arrow::ImportSchema(&arrow_schema);
+    if (!import_result.ok()) {
+      return InvalidSchema("Failed to import the arrow schema: {}",
+                           import_result.status().message());
+    }
+    context_->arrow_schema_ = import_result.MoveValueUnsafe();
+
+    auto arrow_struct_type =
+        std::make_shared<::arrow::StructType>(context_->arrow_schema_->fields());
+    auto builder_result = ::arrow::MakeBuilder(arrow_struct_type);
+    if (!builder_result.ok()) {
+      return InvalidSchema("Failed to make the arrow builder: {}",
+                           builder_result.status().message());
+    }
+    context_->builder_ = builder_result.MoveValueUnsafe();
+
+    return {};
+  }
+
+  Result<Data> ConvertBuilderToArrowArray() {
+    if (context_->builder_->length() == 0) {
+      return {};
+    }
+
+    auto builder_result = context_->builder_->Finish();
+    if (!builder_result.ok()) {
+      return InvalidArrowData("Failed to finish the arrow array builder: {}",
+                              builder_result.status().message());
+    }
+
+    auto array = builder_result.MoveValueUnsafe();

Review Comment:
   ```suggestion
       auto array = std::move(builder_result).MoveValueUnsafe();
   ```
   
   ?



##########
src/iceberg/avro/avro_data_util.cc:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_data_util_internal.h"
+
+namespace iceberg::avro {
+
+Status AppendDatumToBuilder(const ::avro::NodePtr& avro_node,
+                            const ::avro::GenericDatum& avro_datum,
+                            const SchemaProjection& projection,
+                            const Schema& arrow_schema,
+                            ::arrow::ArrayBuilder* array_builder) {
+  return NotImplemented("AppendDatumToBuilder is not yet implemented");

Review Comment:
   Would this implement in another patch?



##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {

Review Comment:
   should we mark close and checkClosed here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115398710


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {
+    if (reader_ != nullptr) {
+      reader_->close();
+      reader_.reset();
+    }
+    context_.reset();
+    return {};
+  }
+
+ private:
+  Status InitReadContext() {
+    context_ = std::make_unique<ReadContext>();
+    context_->datum_ = std::make_unique<::avro::GenericDatum>(reader_->readerSchema());
+
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    auto import_result = ::arrow::ImportSchema(&arrow_schema);
+    if (!import_result.ok()) {
+      return InvalidSchema("Failed to import the arrow schema: {}",
+                           import_result.status().message());
+    }
+    context_->arrow_schema_ = import_result.MoveValueUnsafe();
+
+    auto arrow_struct_type =
+        std::make_shared<::arrow::StructType>(context_->arrow_schema_->fields());
+    auto builder_result = ::arrow::MakeBuilder(arrow_struct_type);
+    if (!builder_result.ok()) {
+      return InvalidSchema("Failed to make the arrow builder: {}",
+                           builder_result.status().message());
+    }
+    context_->builder_ = builder_result.MoveValueUnsafe();
+
+    return {};
+  }
+
+  Result<Data> ConvertBuilderToArrowArray() {
+    if (context_->builder_->length() == 0) {
+      return {};
+    }
+
+    auto builder_result = context_->builder_->Finish();
+    if (!builder_result.ok()) {
+      return InvalidArrowData("Failed to finish the arrow array builder: {}",
+                              builder_result.status().message());
+    }
+
+    auto array = builder_result.MoveValueUnsafe();

Review Comment:
   Oh my, just MoveValueUnsafe is ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#issuecomment-2934289267

   cc @mapleFU would you like to take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115396425


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {
+    if (reader_ != nullptr) {
+      reader_->close();
+      reader_.reset();
+    }
+    context_.reset();
+    return {};
+  }
+
+ private:
+  Status InitReadContext() {
+    context_ = std::make_unique<ReadContext>();
+    context_->datum_ = std::make_unique<::avro::GenericDatum>(reader_->readerSchema());
+
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    auto import_result = ::arrow::ImportSchema(&arrow_schema);
+    if (!import_result.ok()) {
+      return InvalidSchema("Failed to import the arrow schema: {}",
+                           import_result.status().message());
+    }
+    context_->arrow_schema_ = import_result.MoveValueUnsafe();
+
+    auto arrow_struct_type =
+        std::make_shared<::arrow::StructType>(context_->arrow_schema_->fields());
+    auto builder_result = ::arrow::MakeBuilder(arrow_struct_type);
+    if (!builder_result.ok()) {
+      return InvalidSchema("Failed to make the arrow builder: {}",
+                           builder_result.status().message());
+    }
+    context_->builder_ = builder_result.MoveValueUnsafe();
+
+    return {};
+  }
+
+  Result<Data> ConvertBuilderToArrowArray() {
+    if (context_->builder_->length() == 0) {
+      return {};
+    }
+
+    auto builder_result = context_->builder_->Finish();
+    if (!builder_result.ok()) {
+      return InvalidArrowData("Failed to finish the arrow array builder: {}",
+                              builder_result.status().message());
+    }
+
+    auto array = builder_result.MoveValueUnsafe();

Review Comment:
   Why need this move?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115395582


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {

Review Comment:
   I think we don't need this because the state is cleared.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2125320434


##########
src/iceberg/arrow/arrow_fs_file_io.h:
##########
@@ -46,6 +46,9 @@ class ICEBERG_BUNDLE_EXPORT ArrowFileSystemFileIO : public FileIO {
   /// \brief Delete a file at the given location.
   Status DeleteFile(const std::string& file_location) override;
 
+  /// \brief Get the Arrow file system.
+  const std::shared_ptr<::arrow::fs::FileSystem>& fs() const { return arrow_fs_; }

Review Comment:
   @Fokko We have two libraries: `libiceberg` and `libiceberg-bundle`:
   - libiceberg: it only uses interfaces of `FileIO` and `FileReader`, downstream projects should provide their own implementations.
   - libiceberg-bundle: it uses `ArrowFileSystemFileIO` as the implementation and both Avro and Parquet reader implementations assume that arrow::FileSystem is available.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115392858


##########
src/iceberg/avro/avro_reader.cc:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_reader.h"
+
+#include <memory>
+
+#include <arrow/array/builder_base.h>
+#include <arrow/c/bridge.h>
+#include <arrow/filesystem/filesystem.h>
+#include <arrow/record_batch.h>
+#include <arrow/result.h>
+#include <arrow/type.h>
+#include <avro/DataFile.hh>
+#include <avro/Generic.hh>
+#include <avro/GenericDatum.hh>
+
+#include "iceberg/arrow/arrow_fs_file_io.h"
+#include "iceberg/avro/avro_data_util_internal.h"
+#include "iceberg/avro/avro_schema_util_internal.h"
+#include "iceberg/avro/avro_stream_internal.h"
+#include "iceberg/schema_internal.h"
+#include "iceberg/util/checked_cast.h"
+#include "iceberg/util/macros.h"
+
+namespace iceberg::avro {
+
+namespace {
+
+Result<std::unique_ptr<AvroInputStream>> CreateInputStream(const ReaderOptions& options,
+                                                           int64_t buffer_size) {
+  ::arrow::fs::FileInfo file_info(options.path, ::arrow::fs::FileType::File);
+  if (options.length) {
+    file_info.set_size(options.length.value());
+  }
+
+  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
+  auto result = io->fs()->OpenInputFile(file_info);
+  if (!result.ok()) {
+    return IOError("Failed to open file {} for {}", options.path,
+                   result.status().message());
+  }
+
+  return std::make_unique<AvroInputStream>(result.MoveValueUnsafe(), buffer_size);
+}
+
+}  // namespace
+
+// A stateful context to keep track of the reading progress.
+struct ReadContext {
+  // The datum to reuse for reading the data.
+  std::unique_ptr<::avro::GenericDatum> datum_;
+  // The arrow schema to build the record batch.
+  std::shared_ptr<::arrow::Schema> arrow_schema_;
+  // The builder to build the record batch.
+  std::shared_ptr<::arrow::ArrayBuilder> builder_;
+};
+
+// TODO(gang.wu): there are a lot to do to make this reader work.
+// 1. prune the reader schema based on the projection
+// 2. read key-value metadata from the avro file
+// 3. collect basic reader metrics
+class AvroBatchReader::Impl {
+ public:
+  Status Open(const ReaderOptions& options) {
+    batch_size_ = options.batch_size;
+    read_schema_ = options.projection;
+
+    // Open the input stream and adapt to the avro interface.
+    // TODO(gangwu): make this configurable
+    constexpr int64_t kDefaultBufferSize = 1024 * 1024;
+    ICEBERG_ASSIGN_OR_RAISE(auto input_stream,
+                            CreateInputStream(options, kDefaultBufferSize));
+
+    // Create a base reader without setting reader schema to enable projection.
+    auto base_reader =
+        std::make_unique<::avro::DataFileReaderBase>(std::move(input_stream));
+    const ::avro::ValidSchema& file_schema = base_reader->dataSchema();
+
+    // Validate field ids in the file schema.
+    HasIdVisitor has_id_visitor;
+    ICEBERG_RETURN_UNEXPECTED(has_id_visitor.Visit(file_schema));
+    if (has_id_visitor.HasNoIds()) {
+      // TODO(gangwu): support applying field-ids based on name mapping
+      return NotImplemented("Avro file schema has no field IDs");
+    }
+    if (!has_id_visitor.AllHaveIds()) {
+      return InvalidSchema("Not all fields in the Avro file schema have field IDs");
+    }
+
+    // Project the read schema on top of the file schema.
+    // TODO(gangwu): support pruning source fields
+    ICEBERG_ASSIGN_OR_RAISE(projection_, Project(*options.projection, file_schema.root(),
+                                                 /*prune_source=*/false));
+    base_reader->init(file_schema);
+    reader_ = std::make_unique<::avro::DataFileReader<::avro::GenericDatum>>(
+        std::move(base_reader));
+
+    if (options.split) {
+      reader_->sync(options.split->offset);
+      split_end_ = options.split->offset + options.split->length;
+    }
+    return {};
+  }
+
+  Result<Data> Next() {
+    if (!context_) {
+      ICEBERG_RETURN_UNEXPECTED(InitReadContext());
+    }
+
+    while (context_->builder_->length() < batch_size_) {
+      if (split_end_ && reader_->pastSync(split_end_.value())) {
+        break;
+      }
+      if (!reader_->read(*context_->datum_)) {
+        break;
+      }
+      ICEBERG_RETURN_UNEXPECTED(
+          AppendDatumToBuilder(reader_->readerSchema().root(), *context_->datum_,
+                               projection_, *read_schema_, context_->builder_.get()));
+    }
+
+    return ConvertBuilderToArrowArray();
+  }
+
+  Status Close() {
+    if (reader_ != nullptr) {
+      reader_->close();
+      reader_.reset();
+    }
+    context_.reset();
+    return {};
+  }
+
+ private:
+  Status InitReadContext() {
+    context_ = std::make_unique<ReadContext>();
+    context_->datum_ = std::make_unique<::avro::GenericDatum>(reader_->readerSchema());
+
+    ArrowSchema arrow_schema;
+    ICEBERG_RETURN_UNEXPECTED(ToArrowSchema(*read_schema_, &arrow_schema));
+    auto import_result = ::arrow::ImportSchema(&arrow_schema);
+    if (!import_result.ok()) {
+      return InvalidSchema("Failed to import the arrow schema: {}",
+                           import_result.status().message());
+    }
+    context_->arrow_schema_ = import_result.MoveValueUnsafe();
+
+    auto arrow_struct_type =
+        std::make_shared<::arrow::StructType>(context_->arrow_schema_->fields());
+    auto builder_result = ::arrow::MakeBuilder(arrow_struct_type);
+    if (!builder_result.ok()) {

Review Comment:
   Yes, I've thought about this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115393660


##########
src/iceberg/avro/avro_data_util.cc:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "iceberg/avro/avro_data_util_internal.h"
+
+namespace iceberg::avro {
+
+Status AppendDatumToBuilder(const ::avro::NodePtr& avro_node,
+                            const ::avro::GenericDatum& avro_datum,
+                            const SchemaProjection& projection,
+                            const Schema& arrow_schema,
+                            ::arrow::ArrayBuilder* array_builder) {
+  return NotImplemented("AppendDatumToBuilder is not yet implemented");

Review Comment:
   Yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat: implement avro file reader [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #113:
URL: https://github.com/apache/iceberg-cpp/pull/113#discussion_r2115528845


##########
src/iceberg/expected.h:
##########
@@ -122,7 +122,7 @@ inline constexpr bool is_error_type_valid_v = is_error_type_valid<T>::value;
 }  // namespace expected_detail
 
 template <class E>
-class ICEBERG_EXPORT [[nodiscard]] unexpected {

Review Comment:
   I have to remove these to fix https://github.com/apache/iceberg-cpp/actions/runs/15342734208/job/43172206905. I think adding `ICEBERG_EXPORT` here makes all `Result<E>` defined in the `libiceberg-bundle` as  import symbols and the linker cannot find them anywhere. @zhjwpku @lidavidm 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org