You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kamcheungting-db (via GitHub)" <gi...@apache.org> on 2026/09/30 08:16:39 UTC

[PR] refactor(serde): move DataFile JSON serde into core [iceberg-cpp]

kamcheungting-db opened a new pull request, #974:
URL: https://github.com/apache/iceberg-cpp/pull/974

   ## Summary
   
   Move `DataFile` JSON serialization and deserialization from the REST catalog into core serde.
   
   This is a mechanical move. It does not intentionally change the JSON format or validation behavior. REST keeps forwarding overloads, so existing REST callers continue to use the same API.
   
   ## Why
   
   Core `FileScanTask` serde needs to read and write data files without depending on the REST catalog.
   
   Before:
   
   ```cpp
   iceberg::rest::DataFileFromJson(json, specs, schema);
   ```
   
   After:
   
   ```cpp
   iceberg::DataFileFromJson(json, specs, schema);
   ```
   
   ## Stack
   
   This is **1 of 4**:
   
   1. **This PR:** move `DataFile` serde into core
   2. Make `DataFile` JSON Java-compatible
   3. Add core `FileScanTask` serde (#939)
   4. Validate schema and partition IDs strictly (#940)
   
   Each PR targets `main`; later PRs contain the commits from earlier PRs.
   
   ## Testing
   
   - Built `json_serde_test` and `rest_catalog_test` on the complete stack.
   - Full stack: 142 JSON serde tests passed.
   - Full stack: 332 REST tests passed; 1 existing test skipped.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor(serde): move DataFile JSON handling from REST to core [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #974:
URL: https://github.com/apache/iceberg-cpp/pull/974


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor(serde): move DataFile JSON handling from REST to core [iceberg-cpp]

Posted by "kamcheungting-db (via GitHub)" <gi...@apache.org>.
kamcheungting-db commented on code in PR #974:
URL: https://github.com/apache/iceberg-cpp/pull/974#discussion_r4175927877


##########
src/iceberg/test/json_serde_test.cc:
##########
@@ -1044,4 +1048,37 @@ TEST(TableRequirementJsonTest, TableRequirementUnknownType) {
   EXPECT_THAT(result, HasErrorMessage("Unknown table requirement type"));
 }
 
+std::unordered_map<int32_t, std::shared_ptr<PartitionSpec>> UnpartitionedSpecs() {

Review Comment:
   Done



##########
src/iceberg/catalog/rest/json_serde.cc:
##########
@@ -357,97 +192,12 @@ Result<std::vector<std::shared_ptr<FileScanTask>>> FileScanTasksFromJson(
   return file_scan_tasks;
 }
 
-Result<nlohmann::json> DataFileToJsonUnchecked(const DataFile& data_file) {
-  nlohmann::json json;
-  switch (data_file.content) {
-    case DataFile::Content::kData:
-      json[kContent] = kContentData;
-      break;
-    case DataFile::Content::kPositionDeletes:
-      json[kContent] = kContentPositionDeletes;
-      break;
-    case DataFile::Content::kEqualityDeletes:
-      json[kContent] = kContentEqualityDeletes;
-      break;
-  }
-  json[kFilePath] = data_file.file_path;
-  json[kFileFormat] = ToString(data_file.file_format);
-
-  if (!data_file.partition_spec_id.has_value()) {
-    return ValidationFailed("Cannot serialize REST content file without 'spec-id'");
-  }
-  json[kSpecId] = data_file.partition_spec_id.value();
-
-  nlohmann::json partition_json = nlohmann::json::array();
-  for (const auto& literal : data_file.partition.values()) {
-    ICEBERG_ASSIGN_OR_RAISE(auto lit_json, iceberg::ToJson(literal));
-    partition_json.push_back(std::move(lit_json));
-  }
-  json[kPartition] = std::move(partition_json);
-
-  json[kRecordCount] = data_file.record_count;
-  json[kFileSizeInBytes] = data_file.file_size_in_bytes;
-
-  SetKeyValueMap(json, kColumnSizes, data_file.column_sizes);
-  SetKeyValueMap(json, kValueCounts, data_file.value_counts);
-  SetKeyValueMap(json, kNullValueCounts, data_file.null_value_counts);
-  SetKeyValueMap(json, kNanValueCounts, data_file.nan_value_counts);
-  SetKeyValueMap(json, kLowerBounds, data_file.lower_bounds);
-  SetKeyValueMap(json, kUpperBounds, data_file.upper_bounds);
-
-  if (!data_file.key_metadata.empty()) {
-    json[kKeyMetadata] = data_file.key_metadata;
-  }
-  if (!data_file.split_offsets.empty()) {
-    json[kSplitOffsets] = data_file.split_offsets;
-  }
-  if (!data_file.equality_ids.empty()) {
-    json[kEqualityIds] = data_file.equality_ids;
-  }
-  if (data_file.sort_order_id.has_value()) {
-    json[kSortOrderId] = data_file.sort_order_id.value();
-  }
-  if (data_file.first_row_id.has_value()) {
-    json[kFirstRowId] = data_file.first_row_id.value();
-  }
-  if (data_file.referenced_data_file.has_value()) {
-    json[kReferencedDataFile] = data_file.referenced_data_file.value();
-  }
-  if (data_file.content_offset.has_value()) {
-    json[kContentOffset] = data_file.content_offset.value();
-  }
-  if (data_file.content_size_in_bytes.has_value()) {
-    json[kContentSizeInBytes] = data_file.content_size_in_bytes.value();
-  }
-
-  return json;
-}
-
 Result<nlohmann::json> ToJson(

Review Comment:
   done



##########
src/iceberg/catalog/rest/json_serde.cc:
##########
@@ -152,154 +126,14 @@ Result<StorageCredential> StorageCredentialFromJson(const nlohmann::json& json)
   return credential;
 }
 
-template <typename Value>
-Result<std::map<int32_t, Value>> KeyValueMapFromJson(const nlohmann::json& json,
-                                                     std::string_view key) {
-  std::map<int32_t, Value> result;
-  if (!json.contains(key) || json.at(key).is_null()) {
-    return result;
-  }
-
-  ICEBERG_ASSIGN_OR_RAISE(auto map_json, GetJsonValue<nlohmann::json>(json, key));
-  ICEBERG_ASSIGN_OR_RAISE(auto keys,
-                          GetJsonValue<std::vector<int32_t>>(map_json, kMapKeys));
-  ICEBERG_ASSIGN_OR_RAISE(auto values,
-                          GetJsonValue<std::vector<Value>>(map_json, kMapValues));
-  if (keys.size() != values.size()) {
-    return JsonParseError("'{}' map keys and values have different lengths", key);
-  }
-
-  for (size_t i = 0; i < keys.size(); ++i) {
-    result[keys[i]] = std::move(values[i]);
-  }
-  return result;
-}
-
-template <typename Value>
-void SetKeyValueMap(nlohmann::json& json, std::string_view key,
-                    const std::map<int32_t, Value>& map) {
-  if (map.empty()) {
-    return;
-  }
-
-  std::vector<int32_t> keys;
-  std::vector<Value> values;
-  keys.reserve(map.size());
-  values.reserve(map.size());
-  for (const auto& [field_id, value] : map) {
-    keys.push_back(field_id);
-    values.push_back(value);
-  }
-  json[key] = {{kMapKeys, std::move(keys)}, {kMapValues, std::move(values)}};
-}
-
 }  // namespace
 
 Result<DataFile> DataFileFromJson(

Review Comment:
   Good point



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor(serde): move DataFile JSON handling from REST to core [iceberg-cpp]

Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on PR #974:
URL: https://github.com/apache/iceberg-cpp/pull/974#issuecomment-6104587230

   @kamcheungting-db gentle ping, could you please rebase this PR when you have a chance?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor(serde): move DataFile JSON handling from REST to core [iceberg-cpp]

Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on code in PR #974:
URL: https://github.com/apache/iceberg-cpp/pull/974#discussion_r4164147826


##########
src/iceberg/catalog/rest/json_serde.cc:
##########
@@ -152,154 +126,14 @@ Result<StorageCredential> StorageCredentialFromJson(const nlohmann::json& json)
   return credential;
 }
 
-template <typename Value>
-Result<std::map<int32_t, Value>> KeyValueMapFromJson(const nlohmann::json& json,
-                                                     std::string_view key) {
-  std::map<int32_t, Value> result;
-  if (!json.contains(key) || json.at(key).is_null()) {
-    return result;
-  }
-
-  ICEBERG_ASSIGN_OR_RAISE(auto map_json, GetJsonValue<nlohmann::json>(json, key));
-  ICEBERG_ASSIGN_OR_RAISE(auto keys,
-                          GetJsonValue<std::vector<int32_t>>(map_json, kMapKeys));
-  ICEBERG_ASSIGN_OR_RAISE(auto values,
-                          GetJsonValue<std::vector<Value>>(map_json, kMapValues));
-  if (keys.size() != values.size()) {
-    return JsonParseError("'{}' map keys and values have different lengths", key);
-  }
-
-  for (size_t i = 0; i < keys.size(); ++i) {
-    result[keys[i]] = std::move(values[i]);
-  }
-  return result;
-}
-
-template <typename Value>
-void SetKeyValueMap(nlohmann::json& json, std::string_view key,
-                    const std::map<int32_t, Value>& map) {
-  if (map.empty()) {
-    return;
-  }
-
-  std::vector<int32_t> keys;
-  std::vector<Value> values;
-  keys.reserve(map.size());
-  values.reserve(map.size());
-  for (const auto& [field_id, value] : map) {
-    keys.push_back(field_id);
-    values.push_back(value);
-  }
-  json[key] = {{kMapKeys, std::move(keys)}, {kMapValues, std::move(values)}};
-}
-
 }  // namespace
 
 Result<DataFile> DataFileFromJson(

Review Comment:
   I'm wondering if we can delete this wrapper? It is only called in this file and rest_json_serde_test.cc, I think it's ok to delete this API for long term maintainance.



##########
src/iceberg/test/json_serde_test.cc:
##########
@@ -1044,4 +1048,37 @@ TEST(TableRequirementJsonTest, TableRequirementUnknownType) {
   EXPECT_THAT(result, HasErrorMessage("Unknown table requirement type"));
 }
 
+std::unordered_map<int32_t, std::shared_ptr<PartitionSpec>> UnpartitionedSpecs() {

Review Comment:
   Move `DataFileFromJsonTest` from rest_json_serde_test.cc to here?



##########
src/iceberg/catalog/rest/json_serde.cc:
##########
@@ -357,97 +192,12 @@ Result<std::vector<std::shared_ptr<FileScanTask>>> FileScanTasksFromJson(
   return file_scan_tasks;
 }
 
-Result<nlohmann::json> DataFileToJsonUnchecked(const DataFile& data_file) {
-  nlohmann::json json;
-  switch (data_file.content) {
-    case DataFile::Content::kData:
-      json[kContent] = kContentData;
-      break;
-    case DataFile::Content::kPositionDeletes:
-      json[kContent] = kContentPositionDeletes;
-      break;
-    case DataFile::Content::kEqualityDeletes:
-      json[kContent] = kContentEqualityDeletes;
-      break;
-  }
-  json[kFilePath] = data_file.file_path;
-  json[kFileFormat] = ToString(data_file.file_format);
-
-  if (!data_file.partition_spec_id.has_value()) {
-    return ValidationFailed("Cannot serialize REST content file without 'spec-id'");
-  }
-  json[kSpecId] = data_file.partition_spec_id.value();
-
-  nlohmann::json partition_json = nlohmann::json::array();
-  for (const auto& literal : data_file.partition.values()) {
-    ICEBERG_ASSIGN_OR_RAISE(auto lit_json, iceberg::ToJson(literal));
-    partition_json.push_back(std::move(lit_json));
-  }
-  json[kPartition] = std::move(partition_json);
-
-  json[kRecordCount] = data_file.record_count;
-  json[kFileSizeInBytes] = data_file.file_size_in_bytes;
-
-  SetKeyValueMap(json, kColumnSizes, data_file.column_sizes);
-  SetKeyValueMap(json, kValueCounts, data_file.value_counts);
-  SetKeyValueMap(json, kNullValueCounts, data_file.null_value_counts);
-  SetKeyValueMap(json, kNanValueCounts, data_file.nan_value_counts);
-  SetKeyValueMap(json, kLowerBounds, data_file.lower_bounds);
-  SetKeyValueMap(json, kUpperBounds, data_file.upper_bounds);
-
-  if (!data_file.key_metadata.empty()) {
-    json[kKeyMetadata] = data_file.key_metadata;
-  }
-  if (!data_file.split_offsets.empty()) {
-    json[kSplitOffsets] = data_file.split_offsets;
-  }
-  if (!data_file.equality_ids.empty()) {
-    json[kEqualityIds] = data_file.equality_ids;
-  }
-  if (data_file.sort_order_id.has_value()) {
-    json[kSortOrderId] = data_file.sort_order_id.value();
-  }
-  if (data_file.first_row_id.has_value()) {
-    json[kFirstRowId] = data_file.first_row_id.value();
-  }
-  if (data_file.referenced_data_file.has_value()) {
-    json[kReferencedDataFile] = data_file.referenced_data_file.value();
-  }
-  if (data_file.content_offset.has_value()) {
-    json[kContentOffset] = data_file.content_offset.value();
-  }
-  if (data_file.content_size_in_bytes.has_value()) {
-    json[kContentSizeInBytes] = data_file.content_size_in_bytes.value();
-  }
-
-  return json;
-}
-
 Result<nlohmann::json> ToJson(

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] refactor(serde): move DataFile JSON handling from REST to core [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #974:
URL: https://github.com/apache/iceberg-cpp/pull/974#issuecomment-6109055622

   Thanks @kamcheungting-db for the work and @zhjwpku for the review! I've rebased it and will merge once ci is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org