You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Reranko05 (via GitHub)" <gi...@apache.org> on 2026/07/21 04:46:19 UTC
[PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Reranko05 opened a new pull request, #50568:
URL: https://github.com/apache/arrow/pull/50568
### What changes are included?
This PR introduces a reusable `JsonWriter` wrapper around simdjson's JSON builder API and migrates the integration JSON writer to use it instead of RapidJSON.
Specifically, this PR:
- Adds a reusable `JsonWriter` abstraction in `arrow/json`.
- Migrates the integration JSON writer implementation to `JsonWriter`.
- Updates the integration tests to use `JsonWriter`.
- Adds unit tests for `JsonWriter`.
This is part of the incremental migration from RapidJSON to simdjson.
### Are these changes tested?
Yes.
I added unit tests for `JsonWriter` and verified that both JSON and integration tests pass locally:
- `arrow-json-test`
- `arrow-json-integration-test`
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5055190147
I used `-DCMAKE_BUILD_TYPE=Release` via `--preset=ninja-release-maximal`.
I think that we need to use a release build to measure. I expect that simdjson uses SIMD for writing. It'll require a release build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654451528
##########
dev/tasks/tasks.yml:
##########
@@ -645,7 +645,7 @@ tasks:
ci: github
template: docker-tests/github.linux.yml
params:
- flags: '-e ARROW_DEPENDENCY_SOURCE=SYSTEM -e ARROW_GCS=OFF -e xsimd_SOURCE=BUNDLED'
+ flags: '-e ARROW_DEPENDENCY_SOURCE=SYSTEM -e ARROW_GCS=OFF -e xsimd_SOURCE=BUNDLED -e simdjson_SOURCE=BUNDLED'
Review Comment:
Addressed.
##########
cpp/src/arrow/meson.build:
##########
@@ -517,6 +517,7 @@ if needs_json
'json/from_string.cc',
'json/object_parser.cc',
'json/object_writer.cc',
+ 'json/json_writer_internal.cc',
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5105631277
Except the comment above, I think rest of the CI failures are unrelated to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3626750364
##########
cpp/src/arrow/json/json_writer_internal.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
We can require simdjson 4.0 or later. If system simdjson is old, we can fallback to bundled simdjson.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622190595
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -493,7 +496,7 @@ class ArrayWriter {
void WriteRawNumber(std::string_view v) {
// Avoid RawNumber() as it misleadingly adds quotes
// (see https://github.com/Tencent/rapidjson/pull/1155)
Review Comment:
Yep, good catch. Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622419823
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
I will look into reworking the implementation to remain compatible with 3.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5036503969
> > @pitrou did we ever discuss JSONL writer?
>
> There's probably an issue open for it :)
There is an [issue](https://github.com/apache/arrow/issues/47173) and a [PR](https://github.com/apache/arrow/pull/48812).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5104663293
@pitrou @rok @kou I ran into a Windows Unity build failure after this PR. The errors were all from `reader_test.cc`, where unqualified `internal:: `references (`checked_cast`, `Executor`, `UnwrapOrRaise`, `ThreadPool`) were resolved as `arrow::json::internal`.
As a debugging experiment, I qualified them as `arrow::internal::...`, and the Windows Unity build passed.
Since my PR doesn't modify `reader_test.cc`, I'm not sure whether this indicates a latent issue in the test or a Unity/MSVC-specific lookup interaction. Would you prefer that I include those qualifications in this PR, or is there another approach you'd recommend?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3623023501
##########
cpp/src/arrow/json/json_writer_internal.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
Review Comment:
I don't think so. `JsonWriter` is only used internally by the integration JSON writer, and after renaming the header to `json_writer_internal.h` it's no longer intended as part of the public API. I left `ARROW_EXPORT` unchanged because there are other`*_internal.h headers` that still export their classes, but I could to remove it as well if you think that's more appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3619962952
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -493,7 +496,7 @@ class ArrayWriter {
void WriteRawNumber(std::string_view v) {
// Avoid RawNumber() as it misleadingly adds quotes
// (see https://github.com/Tencent/rapidjson/pull/1155)
Review Comment:
Should we update this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622428810
##########
cpp/src/arrow/integration/json_internal.h:
##########
@@ -80,17 +85,17 @@ namespace arrow::internal::integration::json {
/// \brief Append integration test Schema format to rapidjson writer
Review Comment:
The comment still says this appends to a RapidJSON writer, but the API now takes arrow::json::JsonWriter (simdjson-based). Update the comment to avoid misleading future readers.
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -469,7 +472,7 @@ Status SchemaWriter::VisitType(const DataType& type) {
class ArrayWriter {
public:
- ArrayWriter(const std::string& name, const Array& array, RjWriter* writer)
+ ArrayWriter(const std::string& name, const Array& array, JsonWriter* writer)
: name_(name), array_(array), writer_(writer) {}
Review Comment:
ArrayWriter has been migrated to JsonWriter, but later in this class there are still writer_->String(...) calls casting lengths to rj::SizeType (e.g. writer_->String(repr.data(), static_cast<rj::SizeType>(repr.size())) / view.size() / v.size()). With JsonWriter::String taking size_t, these casts are unnecessary and can truncate very large strings; they also keep RapidJSON types in the writer path. Update those calls to pass .size() directly (size_t).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5087279280
@kou Sure
`json_writer_internal.h`
```cpp
#pragma once
#include <cstdint>
#include <sstream>
#include <string>
#include <string_view>
#include "arrow/util/visibility.h"
namespace arrow::json {
class ARROW_EXPORT JsonWriter {
public:
JsonWriter() = default;
void StartObject();
void EndObject();
void StartArray();
void EndArray();
void Key(std::string_view key);
void String(std::string_view value);
void RawValue(std::string_view value);
void Bool(bool value);
void Int(int32_t value);
void Int64(int64_t value);
void Uint(uint32_t value);
void Uint64(uint64_t value);
void Double(double value);
void Null();
std::string_view GetString() const;
void Clear();
private:
void MaybeComma();
void WriteQuotedString(std::string_view value);
std::ostringstream output_;
mutable std::string output_cache_;
bool needs_comma_ = false;
};
} // namespace arrow::json
```
`json_writer_internal.cc`
```cpp
#include "arrow/json/json_writer_internal.h"
namespace arrow::json {
void JsonWriter::StartObject() {
MaybeComma();
output_ << '{';
needs_comma_ = false;
}
void JsonWriter::EndObject() {
output_ << '}';
needs_comma_ = true;
}
void JsonWriter::StartArray() {
MaybeComma();
output_ << '[';
needs_comma_ = false;
}
void JsonWriter::EndArray() {
output_ << ']';
needs_comma_ = true;
}
void JsonWriter::Key(std::string_view key) {
MaybeComma();
WriteQuotedString(key);
output_ << ':';
needs_comma_ = false;
}
void JsonWriter::String(std::string_view value) {
MaybeComma();
WriteQuotedString(value);
needs_comma_ = true;
}
void JsonWriter::RawValue(std::string_view value) {
MaybeComma();
output_ << value;
needs_comma_ = true;
}
void JsonWriter::Bool(bool value) {
MaybeComma();
output_ << (value ? "true" : "false");
needs_comma_ = true;
}
void JsonWriter::Int(int32_t value) {
MaybeComma();
output_ << value;
needs_comma_ = true;
}
void JsonWriter::Int64(int64_t value) {
MaybeComma();
output_ << value;
needs_comma_ = true;
}
void JsonWriter::Uint(uint32_t value) {
MaybeComma();
output_ << value;
needs_comma_ = true;
}
void JsonWriter::Uint64(uint64_t value) {
MaybeComma();
output_ << value;
needs_comma_ = true;
}
void JsonWriter::Double(double value) {
MaybeComma();
output_ << value;
needs_comma_ = true;
}
void JsonWriter::Null() {
MaybeComma();
output_ << "null";
needs_comma_ = true;
}
std::string_view JsonWriter::GetString() const {
output_cache_ = output_.str();
return output_cache_;
}
void JsonWriter::Clear() {
output_.str("");
output_.clear();
output_cache_.clear();
needs_comma_ = false;
}
void JsonWriter::MaybeComma() {
if (needs_comma_) {
output_ << ',';
}
}
void JsonWriter::WriteQuotedString(std::string_view value) {
output_ << '"';
for (char c : value) {
switch (c) {
case '"':
output_ << "\\\"";
break;
case '\\':
output_ << "\\\\";
break;
case '\b':
output_ << "\\b";
break;
case '\f':
output_ << "\\f";
break;
case '\n':
output_ << "\\n";
break;
case '\r':
output_ << "\\r";
break;
case '\t':
output_ << "\\t";
break;
default:
output_ << c;
break;
}
}
output_ << '"';
}
} // namespace arrow::json
```
Add temporary benchmark in `json_writer_internal_test.cc`
```cpp
#include <chrono>
#include <iostream>
```
```cpp
TEST(JsonWriter, BenchmarkSerialization) {
JsonWriter writer;
constexpr int kIterations = 1000000;
auto start = std::chrono::steady_clock::now();
for (int i = 0; i < kIterations; ++i) {
writer.Clear();
writer.StartObject();
writer.Key("id");
writer.Int(i);
writer.Key("name");
writer.String("Apache Arrow");
writer.Key("value");
writer.Double(3.1415926535);
writer.Key("valid");
writer.Bool(true);
writer.EndObject();
auto s = writer.GetString();
(void)s;
}
auto end = std::chrono::steady_clock::now();
auto ms =
std::chrono::duration_cast<std::chrono::milliseconds>(end - start)
.count();
std::cout << "Time: " << ms << " ms\n";
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5049458022
> > We can require simdjson 4.0 or later. If system simdjson is old, we can fallback to bundled simdjson.
>
> @rok Does this approach work for you as well?
Yes, this sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5034227225
Another thought - do we have conbench cpp micro benchmarks to monitor improvement of json parsing and writing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5040925006
@kou It was just a temporary benchmark. I used the following to measure `IntegrationJsonWriter::WriteRecordBatch() + Finish()` on a 1M-row `RecordBatch`:
```
auto schema = ::arrow::schema({
field("f1", boolean()),
field("f2", int8()),
field("f3", int32()),
field("f4", uint64()),
field("f5", utf8())
});
std::vector<std::shared_ptr<Array>> arrays;
const int num_rows = 1000000;
MakeBatchArrays(schema, num_rows, &arrays);
auto batch = RecordBatch::Make(schema, num_rows, arrays);
ASSERT_OK_AND_ASSIGN(auto writer, IntegrationJsonWriter::Open(schema));
auto start = std::chrono::steady_clock::now();
ASSERT_OK(writer->WriteRecordBatch(*batch));
ASSERT_OK_AND_ASSIGN(auto json_data, writer->Finish());
auto end = std::chrono::steady_clock::now();
std::cout << "Serialization time: "
<< std::chrono::duration_cast<std::chrono::milliseconds>(
end - start)
.count()
<< " ms\n";
```
I ran it 10 times on both `main` and this branch, discarded one obvious outlier from each set, and compared the average serialization time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3635251918
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -310,10 +311,26 @@ macro(resolve_dependency DEPENDENCY_NAME)
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "BUNDLED")
build_dependency(${DEPENDENCY_NAME})
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM")
- find_package(${FIND_PACKAGE_ARGUMENTS} REQUIRED)
- if(ARG_FORCE_ANY_NEWER_VERSION AND ARG_REQUIRED_VERSION)
+ if(ARG_FALLBACK_TO_BUNDLED_ON_OLD_SYSTEM_VERSION)
Review Comment:
Could you use `-Dsimdjson_SOURCE=BUNDLED` explicitly on jobs that use `-DARROW_DEPENDENCY_SOURCE=SYSTEM`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622622190
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
Review Comment:
Let's have only the `std::string_view` version? The other doesn't bring any benefit.
##########
cpp/src/arrow/json/json_writer.h:
##########
Review Comment:
Does this header have to be public? Otherwise, you can make it internal by renaming it to `json_writer_internal.h`.
##########
cpp/src/arrow/integration/json_internal.h:
##########
@@ -36,9 +36,14 @@
#include "arrow/util/visibility.h"
namespace rj = arrow::rapidjson;
-using RjWriter = rj::Writer<rj::StringBuffer>;
-using RjArray = rj::Value::ConstArray;
using RjObject = rj::Value::ConstObject;
+using RjArray = rj::Value::ConstArray;
+
+namespace arrow {
+namespace json {
Review Comment:
Can you collapse this as `namespace arrow::json`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3635330602
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -310,10 +311,26 @@ macro(resolve_dependency DEPENDENCY_NAME)
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "BUNDLED")
build_dependency(${DEPENDENCY_NAME})
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM")
- find_package(${FIND_PACKAGE_ARGUMENTS} REQUIRED)
- if(ARG_FORCE_ANY_NEWER_VERSION AND ARG_REQUIRED_VERSION)
+ if(ARG_FALLBACK_TO_BUNDLED_ON_OLD_SYSTEM_VERSION)
Review Comment:
The case should be already handled here: https://github.com/apache/arrow/blob/06c336af2150c8e69f06618902171d09f0ba6a83/cpp/cmake_modules/ThirdpartyToolchain.cmake#L299-L302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5082980717
Hi @kou! Would it be possible to assign the issue to me? I think that would help avoid duplicate work while the RapidJSON → simdjson migration is in progress. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3635300887
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -310,10 +311,26 @@ macro(resolve_dependency DEPENDENCY_NAME)
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "BUNDLED")
build_dependency(${DEPENDENCY_NAME})
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM")
- find_package(${FIND_PACKAGE_ARGUMENTS} REQUIRED)
- if(ARG_FORCE_ANY_NEWER_VERSION AND ARG_REQUIRED_VERSION)
+ if(ARG_FALLBACK_TO_BUNDLED_ON_OLD_SYSTEM_VERSION)
Review Comment:
My understanding is that `AUTO `already falls back when the system package is missing. The additional logic here was intended for the case where the system package is found but its version is older than the required simdjson v4.0.0. Without it, wouldn't configuration still fail for older system versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5105969889
@pitrou Thanks for confirming! Using that approach fixed the CI issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622377796
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
You are right. I checked the simdjson headers: `simdjson::builder::string_builder` is introduced in simdjson v4.x and is not available in `v3.0.0`. Would you prefer that I raise the minimum required simdjson version to `v4.0.0`, or would you rather I rework the implementation to remain compatible with v3.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654452377
##########
cpp/src/arrow/CMakeLists.txt:
##########
@@ -1041,10 +1042,11 @@ if(ARROW_JSON)
json/from_string.cc
json/object_parser.cc
json/object_writer.cc
+ json/json_writer_internal.cc
Review Comment:
Addressed.
##########
cpp/src/arrow/json/CMakeLists.txt:
##########
@@ -24,10 +24,12 @@ add_arrow_test(test
parser_test.cc
reader_test.cc
object_parser_test.cc
+ json_writer_internal_test.cc
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654111592
##########
cpp/src/arrow/integration/json_internal.h:
##########
@@ -80,17 +85,17 @@ namespace arrow::internal::integration::json {
/// \brief Append integration test Schema format to rapidjson writer
Review Comment:
@Reranko05 Could you check this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5099892812
@kou @rok Could you review this when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622041474
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
Review Comment:
Should `JsonWriter` be public?
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
We currently [require simdjson 3.0.0](https://github.com/Reranko05/arrow/blob/bb7fd906b7e47b413071ddb7d5e07888c26e0e7d/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2828), but builder API was introduced in [simdjson 4.0.0](https://github.com/simdjson/simdjson/releases?page=3#release-v4.0.0). Are we sure this will build for 3.0.0 <= version < 4.0.0? Can you do a (local) test to show what's the minimum simdjson version we need for introduced features?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5035713460
> If I missed an existing Conbench benchmark, please let me know and I will run this against it.
It then seems we currently don't benchmark serializing and deserializing json. It would be useful to introduce some, but perhaps that is out of scope for this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5036288457
> @pitrou did we ever discuss JSONL writer?
There's probably an issue open for it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3623007209
##########
cpp/src/arrow/integration/json_internal.h:
##########
@@ -36,9 +36,14 @@
#include "arrow/util/visibility.h"
namespace rj = arrow::rapidjson;
-using RjWriter = rj::Writer<rj::StringBuffer>;
-using RjArray = rj::Value::ConstArray;
using RjObject = rj::Value::ConstObject;
+using RjArray = rj::Value::ConstArray;
+
+namespace arrow {
+namespace json {
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5030847506
@kou could you review this PR when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5035340196
> Another thought - do we have conbench cpp micro benchmarks to monitor improvement of json parsing and writing?
I wasn't able to find any existing Conbench C++ microbenchmarks covering JSON parsing or writing. I did run a local benchmark comparing `IntegrationJsonWriter::WriteRecordBatch() + Finish()` on this branch against main and didn't observe any measurable regression. If I missed an existing Conbench benchmark, please let me know and I will run this against it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5035977154
They are testing the JSONL reader, not the JSON integration reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5039144352
> I compared the serialization performance of `IntegrationJsonWriter::WriteRecordBatch() + Finish()` using a temporary benchmark with a 1M-row `RecordBatch`. I ran the benchmark 10 times on both main and this branch. After excluding one obvious outlier from each run, the averages were:
>
> main: 2157.4 ms
> this branch: 2153.6 ms
>
> I didn't observe any measurable performance regression.
Thanks. I expected that simdjson would be faster than RapidJSON for writing too... Could you share the benchmark program you used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5105923424
> Since my PR doesn't modify `reader_test.cc`, I'm not sure whether this indicates a latent issue in the test or a Unity/MSVC-specific lookup interaction. Would you prefer that I include those qualifications in this PR, or is there another approach you'd recommend?
Yes, it's quite typical of unity builds which amalgamate several source files (`.cc` files) into a single translation unit, which might surface multiple sub-namespaces named `internal`. That's why you may have to use the fully-qualified namespace to fix the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #50568:
URL: https://github.com/apache/arrow/pull/50568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654436847
##########
cpp/src/arrow/integration/json_internal.h:
##########
@@ -80,17 +85,17 @@ namespace arrow::internal::integration::json {
/// \brief Append integration test Schema format to rapidjson writer
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5030257438
:warning: GitHub issue #50567 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5031871024
@kou I investigated the failing workflows and they appear to be unrelated to this PR. Could you please verify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5035777379
See [this PR](https://github.com/apache/arrow/pull/8703/changes#diff-361c1379b5afa5e33454412e0c3216f9cf1511e0db895c20e32783edfb1a6826) for an example how to implement a benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5055141640
@kou I have a question regarding the benchmark results. Earlier, when I benchmarked `IntegrationJsonWriter::WriteRecordBatch() + Finish()` on both the RapidJSON and simdjson implementations using a 1M-row `RecordBatch`, I measured around 2.1 s for both (in a debug build). After replacing the implementation with a naive `std::ostringstream`, I also measured around the same order of magnitude. You mentioned seeing about 65 ms for RapidJSON and 69 ms for simdjson. Were those measurements taken in a different setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3635235830
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -310,10 +311,26 @@ macro(resolve_dependency DEPENDENCY_NAME)
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "BUNDLED")
build_dependency(${DEPENDENCY_NAME})
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM")
- find_package(${FIND_PACKAGE_ARGUMENTS} REQUIRED)
- if(ARG_FORCE_ANY_NEWER_VERSION AND ARG_REQUIRED_VERSION)
+ if(ARG_FALLBACK_TO_BUNDLED_ON_OLD_SYSTEM_VERSION)
Review Comment:
Do we really need this?
I think that `AUTO` does this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5035949557
It seems we do have some - search for json on http://conbench.arrow-dev.org/c-benchmarks/.
Here are examples: [ReadJSONBlockWithSchemaSingleThread](http://conbench.arrow-dev.org/benchmark-results/06a5f65d335b743a8000b8c5e001322a/), [ReadJSONBlockWithSchemaMultiThread](http://conbench.arrow-dev.org/benchmark-results/06a5f2ffadbe7e55800085b851bae0bb/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654113010
##########
cpp/src/arrow/json/CMakeLists.txt:
##########
@@ -24,10 +24,12 @@ add_arrow_test(test
parser_test.cc
reader_test.cc
object_parser_test.cc
+ json_writer_internal_test.cc
Review Comment:
Could you sort `SOURCES`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5126438227
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 6c2c8c751430026625d12627d8c4f56e6ab80355.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/90779445836) has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3635300887
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -310,10 +311,26 @@ macro(resolve_dependency DEPENDENCY_NAME)
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "BUNDLED")
build_dependency(${DEPENDENCY_NAME})
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM")
- find_package(${FIND_PACKAGE_ARGUMENTS} REQUIRED)
- if(ARG_FORCE_ANY_NEWER_VERSION AND ARG_REQUIRED_VERSION)
+ if(ARG_FALLBACK_TO_BUNDLED_ON_OLD_SYSTEM_VERSION)
Review Comment:
My intention with the additional logic was to handle the case where the system package is found but its version is older than the required simdjson v4.0.0. But we can use `-Dsimdjson_SOURCE=BUNDLED` explicitly for the jobs that use `-DARROW_DEPENDENCY_SOURCE=SYSTEM`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5048548046
> We can require simdjson 4.0 or later. If system simdjson is old, we can fallback to bundled simdjson.
@rok Does this approach work for you as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5054637476
> I used the following to measure `IntegrationJsonWriter::WriteRecordBatch() + Finish()` on a 1M-row `RecordBatch`:
Thanks. I also tried it.
simdjson: 69 ms
RapidJSON: 65 ms
Is RapidJSON fast enough? Is simdjson not fast?
Can we try naive `std::ostringstream` implementation something like the following?
```cpp
namespace arrow::json {
class ARROW_EXPORT JsonWriter {
// ...
private:
// ...
std::ostringstream output_;
// ...
};
}
namespace arrow::json {
void JsonWriter::StartObject() {
MaybeComma();
output_ << "{";
needs_comma_ = false;
}
// ...
```
If this performance is similar to simdjson/RapidJSON, they may not be so fast for writing.
If this is slower than simdjson/RapidJSON, both of them are fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5086232490
It seems that the issue is already assigned to you: https://github.com/apache/arrow/issues/50567#event-28249087135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5078981110
Marking this as ready for review. The remaining CI failures appear to be unrelated to this change (dependency download, environment, or infrastructure issues).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622432230
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
I am not sure writing is possible pre-4.x so we likely want to raise the version. If we do so, how do we support ubuntu<25?
cc @lemire
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622432230
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
I am not sure writing is possible pre-4.x so we likely want to raise the version. How do we support ubuntu<25?
cc @lemire
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5035935528
> It then seems we currently don't benchmark serializing and deserializing json. It would be useful to introduce some, but perhaps that is out of scope for this ticket.
It doesn't sound useful to benchmark JSON parsing for integration testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5036171676
> It doesn't sound useful to benchmark JSON parsing for integration testing.
yeah, that is not very interesting indeed.
@Reranko05 sorry for bringing up benchmarks here, we likely don't need new ones here.
@pitrou did we ever discuss JSONL writer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5031302677
@kou, I compared the serialization performance of `IntegrationJsonWriter::WriteRecordBatch() + Finish()` using a temporary benchmark with a 1M-row `RecordBatch`. I ran the benchmark 10 times on both main and this branch. After excluding one obvious outlier from each run, the averages were:
main: 2157.4 ms
this branch: 2153.6 ms
I didn't observe any measurable performance regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3623005588
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
Review Comment:
Addressed.
##########
cpp/src/arrow/json/json_writer_internal.h:
##########
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622377796
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
You are right. I checked the simdjson headers: simdjson::builder::string_builder is introduced in simdjson 4.x and is not available in 3.0.0. Would you prefer that I raise the minimum required simdjson version to 4.0.0, or would you rather I rework the implementation to remain compatible with 3.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622292845
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
https://repology.org/project/simdjson/versions - ubuntu seems to lag a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3622377796
##########
cpp/src/arrow/json/json_writer.h:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <simdjson.h>
+
+#include <cstdint>
+#include <string_view>
+
+#include "arrow/util/visibility.h"
+
+namespace arrow::json {
+
+class ARROW_EXPORT JsonWriter {
+ public:
+ JsonWriter() = default;
+
+ void StartObject();
+ void EndObject();
+
+ void StartArray();
+ void EndArray();
+
+ void Key(std::string_view key);
+
+ void String(std::string_view value);
+ void String(const char* data, size_t length);
+ void RawValue(std::string_view value);
+ void Bool(bool value);
+
+ void Int(int32_t value);
+ void Int64(int64_t value);
+
+ void Uint(uint32_t value);
+ void Uint64(uint64_t value);
+
+ void Double(double value);
+
+ void Null();
+
+ std::string_view GetString() const;
+
+ void Clear();
+
+ private:
+ void MaybeComma();
+
+ simdjson::builder::string_builder builder_;
Review Comment:
Thanks, that's a good point. Ubuntu is lagging so bumping up the minimum version will not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3635377285
##########
cpp/cmake_modules/ThirdpartyToolchain.cmake:
##########
@@ -310,10 +311,26 @@ macro(resolve_dependency DEPENDENCY_NAME)
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "BUNDLED")
build_dependency(${DEPENDENCY_NAME})
elseif(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM")
- find_package(${FIND_PACKAGE_ARGUMENTS} REQUIRED)
- if(ARG_FORCE_ANY_NEWER_VERSION AND ARG_REQUIRED_VERSION)
+ if(ARG_FALLBACK_TO_BUNDLED_ON_OLD_SYSTEM_VERSION)
Review Comment:
```
flags: '-e ARROW_DEPENDENCY_SOURCE=SYSTEM -e ARROW_GCS=OFF -e xsimd_SOURCE=BUNDLED -e simdjson_SOURCE=BUNDLED'
```
Added `simdjson_SOURCE=BUNDLED` to the `test-r-depsource-system` job in `dev/tasks/tasks.yml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5055606650
@kou I reran the benchmark in a Release build using the same temporary benchmark (`IntegrationJsonWriter::WriteRecordBatch() + Finish()` on a 1M-row `RecordBatch`).
Results:
- main (RapidJSON): 324 ms
- This PR (simdjson): 490 ms
- This PR with a naive `std::ostringstream` implementation: 1323 ms
So, in this benchmark, the `std::ostringstream` implementation is about 2.7× slower than the current implementation and about 4× slower than the RapidJSON implementation. My absolute timings are still higher than the ~65–69 ms you reported, so I suspect my benchmark setup differs from yours.
<img width="400" height="250" alt="download" src="https://github.com/user-attachments/assets/9fe5b890-26b4-4464-b7d6-a9a76bd092f4" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654116287
##########
cpp/src/arrow/CMakeLists.txt:
##########
@@ -1041,10 +1042,11 @@ if(ARROW_JSON)
json/from_string.cc
json/object_parser.cc
json/object_writer.cc
+ json/json_writer_internal.cc
Review Comment:
Could you keep this list in alphabetical order?
##########
cpp/src/arrow/meson.build:
##########
@@ -517,6 +517,7 @@ if needs_json
'json/from_string.cc',
'json/object_parser.cc',
'json/object_writer.cc',
+ 'json/json_writer_internal.cc',
Review Comment:
Could you keep this list in alphabetical order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #50568:
URL: https://github.com/apache/arrow/pull/50568#discussion_r3654117444
##########
dev/tasks/tasks.yml:
##########
@@ -645,7 +645,7 @@ tasks:
ci: github
template: docker-tests/github.linux.yml
params:
- flags: '-e ARROW_DEPENDENCY_SOURCE=SYSTEM -e ARROW_GCS=OFF -e xsimd_SOURCE=BUNDLED'
+ flags: '-e ARROW_DEPENDENCY_SOURCE=SYSTEM -e ARROW_GCS=OFF -e xsimd_SOURCE=BUNDLED -e simdjson_SOURCE=BUNDLED'
Review Comment:
Could you keep this list in alphabetical order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #50568:
URL: https://github.com/apache/arrow/pull/50568#issuecomment-5086778542
Could you share the naive `std::ostringstream` implementation? I'll also try it on my environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org