You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "willtemperley (via GitHub)" <gi...@apache.org> on 2026/09/01 05:16:54 UTC
[PR] Add Arrow JSON integration testing executable. Fix ArrowReader bug. [arrow-swift]
willtemperley opened a new pull request, #191:
URL: https://github.com/apache/arrow-swift/pull/191
## What's Changed
Added an Arrow-JSON integration testing executable as discussed in #96. This includes an ArrowJSON struct for decoding and encoding the Arrow JSON test files.
Added a rule to .swiftlint.yml to ignore case statements in cyclomatic complexity checks.
Made some Arrow Type declarations public to support encoding and decoding.
Fix: ArrowReader now creates a null buffer of the length specified by the flatbuffers definition, instead of calculating it from the array length. This fixes a bug where a null buffer of full length would be created and allocated as zeros, making a non-null array become all-null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Arrow JSON integration testing executable. Fix ArrowReader bug. [arrow-swift]
Posted by "maltzsama (via GitHub)" <gi...@apache.org>.
maltzsama commented on PR #191:
URL: https://github.com/apache/arrow-swift/pull/191#issuecomment-5603410907
I tested the integration test executable on Debian 13 against generated_primitive from apache/arrow-testing (cpp-21.0.0 gold files).
What works:
- ARROW_TO_JSON
- VALIDATE (on the .arrow_file format)
- Build compiles cleanly (only deprecation warnings from flatbuffers and ArrowCExporter)
What doesn't work yet (not a blocker for this PR):
- JSON_TO_ARROW and VALIDATE on .arrow_file fail with:
```shell
$ swift run arrow-json-integration-test \ base 15:40:04
--arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
--json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
--mode=ARROW_TO_JSON
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.28s)
$ swift run arrow-json-integration-test \ base 15:40:09
--arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
--json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
--mode=JSON_TO_ARROW
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.28s)
Error: DecodingError.typeMismatch: expected value of type DataValue. Path: batches[0].columns[0].DATA[0]. Debug description: Cannot decode DataValue
$ swift run arrow-json-integration-test \ base 15:40:16
--arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
--json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
--mode=VALIDATE
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.29s)
Error: DecodingError.typeMismatch: expected value of type DataValue. Path: batches[0].columns[0].DATA[0]. Debug description: Cannot decode DataValue
```
But I believe this `DataValue` in `ArrowJSON.swift` decode can be addressed in another issue.
@kou LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Arrow JSON integration testing executable. Fix ArrowReader bug. [arrow-swift]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #191:
URL: https://github.com/apache/arrow-swift/pull/191#discussion_r4003723083
##########
Sources/ArrowJSONIntegrationTest/ArrowJSONIntegrationTest.swift:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import ArgumentParser
+import Arrow
+import Foundation
+
+enum IntegrationMode: String, ExpressibleByArgument, CaseIterable {
+ case arrowToJson = "ARROW_TO_JSON"
+ case jsonToArrow = "JSON_TO_ARROW"
+ case validate = "VALIDATE"
+}
+
+@main
+struct ArrowJSONIntegrationTest: ParsableCommand {
+ static let configuration = CommandConfiguration(
+ commandName: "arrow-json-integration-test",
+ abstract: "Converts between and validates Arrow and JSON integration test files."
+ )
+
+ @Option(name: .long, help: "Path to the Arrow file.")
+ var arrow: String
+
+ @Option(name: .long, help: "Path to the JSON file.")
+ var json: String
+
+ @Option(name: .long, help: "Mode: ARROW_TO_JSON, JSON_TO_ARROW, or VALIDATE.")
+ var mode: IntegrationMode
+
+ func validate() throws {
+ // Runs automatically before `run()` — good place for input checks.
+ if mode != .jsonToArrow {
+ guard FileManager.default.fileExists(atPath: arrow) else {
+ throw ValidationError("Arrow file not found at path: \(arrow)")
+ }
+ }
+ if mode != .arrowToJson {
+ guard FileManager.default.fileExists(atPath: json) else {
+ throw ValidationError("JSON file not found at path: \(json)")
+ }
+ }
+ }
+
+ func run() throws {
+ let arrowURL = URL(fileURLWithPath: arrow)
+ let jsonURL = URL(fileURLWithPath: json)
+
+ switch mode {
+ case .arrowToJson:
+ try convertArrowToJSON(arrowURL: arrowURL, jsonURL: jsonURL)
+ case .jsonToArrow:
+ try convertJSONToArrow(jsonURL: jsonURL, arrowURL: arrowURL)
+ case .validate:
+ try validateFiles(arrowURL: arrowURL, jsonURL: jsonURL)
+ }
+ }
+
+ private func convertJSONToArrow(jsonURL: URL, arrowURL: URL) throws {
+ let arrowJSON = try JSONDecoder().decode(ArrowJSON.self, from: try Data(contentsOf: jsonURL))
+ let jsonFields = arrowJSON.schema.fields // [ArrowJSON.Field]
+ let arrowFields = try jsonFields.map(decodeField) // [ArrowField]
+
+ var batches: [RecordBatch] = []
+ for batch in arrowJSON.batches {
+ let builder = RecordBatch.Builder()
+ for (jsonField, arrowField) in zip(jsonFields, arrowFields) {
+ guard let column = batch.columns.first(where: { $0.name == jsonField.name }) else {
+ throw ArrowError.invalid("Missing column \(jsonField.name)")
+ }
+ _ = builder.addColumn(
+ arrowField.name,
+ arrowArray: try buildHolder(column: column, type: arrowField.type))
+ }
+ switch builder.finish() {
+ case .success(let recordBatch): batches.append(recordBatch)
+ case .failure(let error): throw error
+ }
+ }
+
+ let schemaBuilder = ArrowSchema.Builder()
+ for arrowField in arrowFields {
+ schemaBuilder.addField(arrowField)
+ }
+ let schema = schemaBuilder.finish()
+
+ switch ArrowWriter().writeFile(ArrowWriter.Info(.recordbatch, schema: schema, batches: batches)) {
+ case .success(let data): try data.write(to: arrowURL)
+ case .failure(let error): throw error
+ }
+ }
+
+ /// Converts an Arrow JSON field into the real arrow-swift `ArrowField`.
+ func decodeField(_ field: ArrowJSON.Field) throws -> ArrowField {
+ ArrowField(field.name, type: try decodeArrowType(field), isNullable: field.nullable)
+ }
+
+ private func validateFiles(arrowURL: URL, jsonURL: URL) throws {
+ let expected = try JSONDecoder().decode(
+ ArrowJSON.self, from: try Data(contentsOf: jsonURL))
+
+ let actual = try decodeArrowJSON(from: arrowURL)
+
+ guard expected.normalized() == actual.normalized() else {
+ throw ValidationError("Arrow file does not match JSON file")
+ }
Review Comment:
How about comparing in Apache Arrow data instead of JSON data?
Archery doesn't use the Apache Arrow -> JSON conversion. If we compare data in Apache Arrow, we don't need to implement Apache Arrow -> JSON conversion. The Apache Arrow data comparison will be also useful in other use cases.
##########
Sources/ArrowJSONIntegrationTest/ArrowJSONIntegrationTest.swift:
##########
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import ArgumentParser
+import Arrow
+import Foundation
+
+enum IntegrationMode: String, ExpressibleByArgument, CaseIterable {
+ case arrowToJson = "ARROW_TO_JSON"
+ case jsonToArrow = "JSON_TO_ARROW"
+ case validate = "VALIDATE"
+}
+
+@main
+struct ArrowJSONIntegrationTest: ParsableCommand {
+ static let configuration = CommandConfiguration(
+ commandName: "arrow-json-integration-test",
+ abstract: "Converts between and validates Arrow and JSON integration test files."
+ )
+
+ @Option(name: .long, help: "Path to the Arrow file.")
+ var arrow: String
+
+ @Option(name: .long, help: "Path to the JSON file.")
+ var json: String
+
+ @Option(name: .long, help: "Mode: ARROW_TO_JSON, JSON_TO_ARROW, or VALIDATE.")
+ var mode: IntegrationMode
+
+ func validate() throws {
+ // Runs automatically before `run()` — good place for input checks.
+ if mode != .jsonToArrow {
+ guard FileManager.default.fileExists(atPath: arrow) else {
+ throw ValidationError("Arrow file not found at path: \(arrow)")
+ }
+ }
+ if mode != .arrowToJson {
+ guard FileManager.default.fileExists(atPath: json) else {
+ throw ValidationError("JSON file not found at path: \(json)")
+ }
+ }
+ }
+
+ func run() throws {
+ let arrowURL = URL(fileURLWithPath: arrow)
+ let jsonURL = URL(fileURLWithPath: json)
+
+ switch mode {
+ case .arrowToJson:
+ try convertArrowToJSON(arrowURL: arrowURL, jsonURL: jsonURL)
+ case .jsonToArrow:
+ try convertJSONToArrow(jsonURL: jsonURL, arrowURL: arrowURL)
+ case .validate:
+ try validateFiles(arrowURL: arrowURL, jsonURL: jsonURL)
+ }
+ }
+
+ private func convertJSONToArrow(jsonURL: URL, arrowURL: URL) throws {
+ let arrowJSON = try JSONDecoder().decode(ArrowJSON.self, from: try Data(contentsOf: jsonURL))
+ let jsonFields = arrowJSON.schema.fields // [ArrowJSON.Field]
+ let arrowFields = try jsonFields.map(decodeField) // [ArrowField]
+
+ var batches: [RecordBatch] = []
+ for batch in arrowJSON.batches {
+ let builder = RecordBatch.Builder()
+ for (jsonField, arrowField) in zip(jsonFields, arrowFields) {
+ guard let column = batch.columns.first(where: { $0.name == jsonField.name }) else {
+ throw ArrowError.invalid("Missing column \(jsonField.name)")
+ }
+ _ = builder.addColumn(
+ arrowField.name,
+ arrowArray: try buildHolder(column: column, type: arrowField.type))
+ }
+ switch builder.finish() {
+ case .success(let recordBatch): batches.append(recordBatch)
+ case .failure(let error): throw error
+ }
+ }
+
+ let schemaBuilder = ArrowSchema.Builder()
+ for arrowField in arrowFields {
+ schemaBuilder.addField(arrowField)
+ }
+ let schema = schemaBuilder.finish()
+
+ switch ArrowWriter().writeFile(ArrowWriter.Info(.recordbatch, schema: schema, batches: batches)) {
Review Comment:
Should we use `toFile()` instead of `writeFile()` to write the `ARROW1` marker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org