You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/07/28 15:53:04 UTC
[GitHub] [arrow] LouisClt opened a new pull request, #13737: Add scalar casts to string types for list based types
LouisClt opened a new pull request, #13737:
URL: https://github.com/apache/arrow/pull/13737
Following https://lists.apache.org/thread/rp7vpjtt4lgtjxj35oyjyqh9b6on94jf discussion, here is the PR corresponding to the feature : be able to cast list-like types (maps, lists, fixed-size lists) to string type.
It produces output such as :
list<item: int64>{null, 1}
map<string, int64>{{key:string = a, value:int64 = 2}, {key:string = b, value:int64 = 45}}
map<struct<x: int16, y: bool>, int64>{{key:struct<x: int16, y: bool> = {x:int16 = 884, y:bool = true}, value:int64 = 2}, {key:struct<x: int16, y: bool> = {x:int16 = 874, y:bool = false}, value:int64 = null}}
fixed_size_list<item: float>[3]{4, 5, 6}
I tried to be coherent with the rest of the CastTo methods.
Feel free to comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LouisClt commented on a diff in pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
LouisClt commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r933580775
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
OK, I did the modification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1202489419
Merged.
If you have a JIRA account, let me know so I can assign you to the issue there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1199870783
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13737: Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1198346728
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LouisClt commented on a diff in pull request #13737: Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
LouisClt commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r932964877
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
Hello, yes maybe indeed it will be better, and more coherent with languages such as Python (I see there was also a JSON format representing Arrays, maybe it will be more coherent with that too).
As for the tests, maybe something like that will do :
```
ASSERT_OK_AND_ASSIGN(auto str,
ListScalar(ArrayFromJSON(int16(), "[1, 2, 5]"), list(int16())).CastTo(utf8()));
EXPECT_EQ(*str, StringScalar("list<item: int16>[1, 2, 3]"));
```
But I must admit I can't test this for the moment, I first need to set up better my dev environment for Arrow, because for the moment I do not build the tests (I am on windows building with vcpkg)
So feel free to help as you want.
I accepted your suggestion, which makes the code simpler indeed (and might correct the workflows that did not pass with the first version)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1199870765
https://issues.apache.org/jira/browse/ARROW-17214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LouisClt commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
LouisClt commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1202965430
Hello, thanks for the merging.
My jira account is "louisc" if you find it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1203122027
['Python', 'R'] benchmarks have high level of regressions.
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/bd6cd6234c8f4b53ae8caf565b3623f5...a3900c8d229a465488aa0d4d6b532a95/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1199926251
Hmm, from the Ruby CI, it seems we may actually have already had a ToString for list scalars…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LouisClt commented on a diff in pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
LouisClt commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r933570685
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
Thanks, I merged your PR. Do you want to change "{}" for "[]" or is this OK ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #13737: Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r932436063
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
+ for (int64_t i = 0; i < from.value->length(); i++) {
+ if (i > 0) ss << ", ";
+ auto result = from.value->GetScalar(i);
+ if (!result.ok()) {
+ return(result.status());
+ } else {
+ ss << result.ValueOrDie()->ToString();
+ }
Review Comment:
```suggestion
ARROW_ASSIGN_OR_RAISE(auto value, from.value->GetScalar(i));
ss << value->ToString();
```
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
kind of a nit, but maybe square brackets are preferable here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #13737: Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r933221774
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
Got it, thanks. I'll open a PR to your branch when I find time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1203121708
Benchmark runs are scheduled for baseline = 2a027571d7a10526939c4c37bf84b2368e6f4b74 and contender = 4bd3d2ecbb6f4f311d59098892b7dca5421c15e4. 4bd3d2ecbb6f4f311d59098892b7dca5421c15e4 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/6ecd038902ae457fa3b4e17befdd3faa...efad7929ff904187a970149063a5a39e/)
[Finished :arrow_down:0.54% :arrow_up:0.03%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/e65f8f451f634d02b5535480d6b3f6d5...693dd0441da54882b8e789ad2f423d24/)
[Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/bd6cd6234c8f4b53ae8caf565b3623f5...a3900c8d229a465488aa0d4d6b532a95/)
[Finished :arrow_down:0.46% :arrow_up:0.21%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/64330e80b2ae4bf3949739ac2a638a7b...379f497046cd48f8822c7a4d25a65224/)
Buildkite builds:
[Failed] [`4bd3d2ec` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1241)
[Finished] [`4bd3d2ec` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1253)
[Finished] [`4bd3d2ec` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1235)
[Finished] [`4bd3d2ec` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1255)
[Failed] [`2a027571` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1240)
[Finished] [`2a027571` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1252)
[Finished] [`2a027571` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1234)
[Finished] [`2a027571` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1254)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm merged pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm merged PR #13737:
URL: https://github.com/apache/arrow/pull/13737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1199930951
ah, it's because if CastTo(utf8()) fails, we turn the scalar into an Array and print that instead. but this means the string repr changed; I think the new repr is probably better (at least, for short values) so I'll put up a PR to adjust the remaining tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LouisClt commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
LouisClt commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1200108034
I merged your PR, let's see if tests are OK now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r933571763
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
Hmm, yes, let's just change it to be consistent with (for instance) how arrays themselves print out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #13737:
URL: https://github.com/apache/arrow/pull/13737#issuecomment-1199950904
See https://github.com/LouisClt/arrow/pull/2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #13737: ARROW-17214: [C++] Add scalar casts to string types for list based types
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #13737:
URL: https://github.com/apache/arrow/pull/13737#discussion_r933561633
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1041,6 +1041,24 @@ Status CastImpl(const StructScalar& from, StringScalar* to) {
return Status::OK();
}
+// list based types (list, large list and map (fixed sized list too)) to string
+Status CastImpl(const BaseListScalar& from, StringScalar* to) {
+ std::stringstream ss;
+ ss << from.type->ToString() << "{";
Review Comment:
See https://github.com/LouisClt/arrow/pull/1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org