You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/06/13 01:31:30 UTC
[GitHub] [arrow-datafusion] alamb opened a new pull request, #6656: minor: use sql to setup test data for joins.slt rather than rust
alamb opened a new pull request, #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656
# Which issue does this PR close?
Follow on to https://github.com/apache/arrow-datafusion/pull/6642 from @jiangzhx
related to https://github.com/apache/arrow-datafusion/issues/6302
# Rationale for this change
I noticed we could port more of the rust based setup code in https://github.com/apache/arrow-datafusion/pull/6642 to use sql which makes it clearer what is going on as well as is more concise
# What changes are included in this PR?
Port 4 test setups from Rust to SQL
# Are these changes tested?
covered by existing tests
# Are there any user-facing changes?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on pull request #6656: minor: use sql to setup test data for joins.slt rather than rust
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on PR #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656#issuecomment-1588427676
Sorry for the late reply.
there are places similar to #6643, should we also migrate them from rs to SQL?
https://github.com/apache/arrow-datafusion/blob/071a2a6dfd43b3b225fce1af8838fa95c79f9ede/datafusion/core/tests/sqllogictests/src/main.rs#L266-L287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #6656: minor: use sql to setup test data for joins.slt rather than rust
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656#issuecomment-1589221246
@jiangzhx
> there are places similar to https://github.com/apache/arrow-datafusion/issues/6643, should we also migrate them from rs to SQL?
Yes, that would be awesome. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jiangzhx commented on pull request #6656: minor: use sql to setup test data for joins.slt rather than rust
Posted by "jiangzhx (via GitHub)" <gi...@apache.org>.
jiangzhx commented on PR #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656#issuecomment-1590363183
> @jiangzhx
>
> > there are places similar to #6643, should we also migrate them from rs to SQL?
>
> Yes, that would be awesome. Thank you.
OK, I will take the rest of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] aprimadi commented on a diff in pull request #6656: minor: use sql to setup test data for joins.slt rather than rust
Posted by "aprimadi (via GitHub)" <gi...@apache.org>.
aprimadi commented on code in PR #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656#discussion_r1227472472
##########
datafusion/core/tests/sqllogictests/test_files/joins.slt:
##########
@@ -75,6 +75,113 @@ AS VALUES
(55, 'w', 3),
(NULL, 'v', 0);
+statement ok
+CREATE TABLE left_semi_anti_join_table_t1(t1_id INT UNSIGNED, t1_name VARCHAR, t1_int INT UNSIGNED)
+AS VALUES
+(11, 'a', 1),
+(11, 'a', 1),
+(22, 'b', 2),
+(33, 'c', 3),
+(44, 'd', 4),
+(NULL, 'e', 0);
+
+statement ok
+CREATE TABLE left_semi_anti_join_table_t2(t2_id INT UNSIGNED, t2_name VARCHAR, t2_int INT UNSIGNED)
+AS VALUES
+(11, 'z', 3),
+(11, 'z', 3),
+(22, 'y', 1),
+(44, 'x', 3),
+(55, 'w', 3),
+(NULL, 'v', 0);
+
+
+statement ok
+CREATE TABLE right_semi_anti_join_table_t1(t1_id INT UNSIGNED, t1_name VARCHAR, t1_int INT UNSIGNED)
+AS VALUES
+(11, 'a', 1),
+(22, 'b', 2),
+(33, 'c', 3),
+(44, 'd', 4),
+(NULL, 'e', 0);
+
+statement ok
+CREATE TABLE right_semi_anti_join_table_t2(t2_id INT UNSIGNED, t2_name VARCHAR)
+AS VALUES
+(11, 'a'),
+(11, 'x'),
+(NULL, NULL);
+
+# Table with all of the supported timestamp types values
+#
+# Columns are named:
+# "nanos" --> TimestampNanosecondArray
+# "micros" --> TimestampMicrosecondArray
+# "millis" --> TimestampMillisecondArray
+# "secs" --> TimestampSecondArray
+# "names" --> StringArray
+statement ok
+CREATE TABLE test_timestamps_table_source(ts varchar, names varchar)
+AS VALUES
+('2018-11-13T17:11:10.011375885995', 'Row 0'),
+('2011-12-13T11:13:10.12345', 'Row 1'),
+(NULL, 'Row 2'),
+('2021-01-01T05:11:10.432', 'Row 3');
+
+
+statement ok
+CREATE TABLE test_timestamps_table as
+SELECT
+ arrow_cast(ts::timestamp::bigint, 'Timestamp(Nanosecond, None)') as nanos,
+ arrow_cast(ts::timestamp::bigint / 1000, 'Timestamp(Microsecond, None)') as micros,
+ arrow_cast(ts::timestamp::bigint / 1000000, 'Timestamp(Millisecond, None)') as millis,
+ arrow_cast(ts::timestamp::bigint / 1000000000, 'Timestamp(Second, None)') as secs,
+ names
+FROM
+ test_timestamps_table_source;
+
+
+
+statement ok
+CREATE TABLE hashjoin_datatype_table_t1_source(c1 INT, c2 BIGINT, c3 DECIMAL(5,2), c4 VARCHAR)
+AS VALUES
+(1, 86400000, 1.23, 'abc'),
+(2, 172800000, 456.00, 'def'),
+(null, 259200000, 789.000, 'ghi'),
+(3, null, -123.12, 'jkl')
+;
+
+statement ok
+CREATE TABLE hashjoin_datatype_table_t1
+AS SELECT
+ arrow_cast(c1, 'Date32') as c1,
+ arrow_cast(c2, 'Date64') as c2,
+ c3,
+ arrow_cast(c4, 'Dictionary(Int32, Utf8)') as c4
Review Comment:
Nice, so that's how to cast a dictionary array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #6656: minor: use sql to setup test data for joins.slt rather than rust
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656#discussion_r1228060482
##########
datafusion/core/tests/sqllogictests/test_files/joins.slt:
##########
@@ -75,6 +75,113 @@ AS VALUES
(55, 'w', 3),
(NULL, 'v', 0);
+statement ok
+CREATE TABLE left_semi_anti_join_table_t1(t1_id INT UNSIGNED, t1_name VARCHAR, t1_int INT UNSIGNED)
+AS VALUES
+(11, 'a', 1),
+(11, 'a', 1),
+(22, 'b', 2),
+(33, 'c', 3),
+(44, 'd', 4),
+(NULL, 'e', 0);
+
+statement ok
+CREATE TABLE left_semi_anti_join_table_t2(t2_id INT UNSIGNED, t2_name VARCHAR, t2_int INT UNSIGNED)
+AS VALUES
+(11, 'z', 3),
+(11, 'z', 3),
+(22, 'y', 1),
+(44, 'x', 3),
+(55, 'w', 3),
+(NULL, 'v', 0);
+
+
+statement ok
+CREATE TABLE right_semi_anti_join_table_t1(t1_id INT UNSIGNED, t1_name VARCHAR, t1_int INT UNSIGNED)
+AS VALUES
+(11, 'a', 1),
+(22, 'b', 2),
+(33, 'c', 3),
+(44, 'd', 4),
+(NULL, 'e', 0);
+
+statement ok
+CREATE TABLE right_semi_anti_join_table_t2(t2_id INT UNSIGNED, t2_name VARCHAR)
+AS VALUES
+(11, 'a'),
+(11, 'x'),
+(NULL, NULL);
+
+# Table with all of the supported timestamp types values
+#
+# Columns are named:
+# "nanos" --> TimestampNanosecondArray
+# "micros" --> TimestampMicrosecondArray
+# "millis" --> TimestampMillisecondArray
+# "secs" --> TimestampSecondArray
+# "names" --> StringArray
+statement ok
+CREATE TABLE test_timestamps_table_source(ts varchar, names varchar)
+AS VALUES
+('2018-11-13T17:11:10.011375885995', 'Row 0'),
+('2011-12-13T11:13:10.12345', 'Row 1'),
+(NULL, 'Row 2'),
+('2021-01-01T05:11:10.432', 'Row 3');
+
+
+statement ok
+CREATE TABLE test_timestamps_table as
+SELECT
+ arrow_cast(ts::timestamp::bigint, 'Timestamp(Nanosecond, None)') as nanos,
+ arrow_cast(ts::timestamp::bigint / 1000, 'Timestamp(Microsecond, None)') as micros,
+ arrow_cast(ts::timestamp::bigint / 1000000, 'Timestamp(Millisecond, None)') as millis,
+ arrow_cast(ts::timestamp::bigint / 1000000000, 'Timestamp(Second, None)') as secs,
+ names
+FROM
+ test_timestamps_table_source;
+
+
+
+statement ok
+CREATE TABLE hashjoin_datatype_table_t1_source(c1 INT, c2 BIGINT, c3 DECIMAL(5,2), c4 VARCHAR)
+AS VALUES
+(1, 86400000, 1.23, 'abc'),
+(2, 172800000, 456.00, 'def'),
+(null, 259200000, 789.000, 'ghi'),
+(3, null, -123.12, 'jkl')
+;
+
+statement ok
+CREATE TABLE hashjoin_datatype_table_t1
+AS SELECT
+ arrow_cast(c1, 'Date32') as c1,
+ arrow_cast(c2, 'Date64') as c2,
+ c3,
+ arrow_cast(c4, 'Dictionary(Int32, Utf8)') as c4
Review Comment:
👍 I think `arrow_cast` is the key if you want to control the arrow types specifically (otherwise you are stuck with the types that have an SQL mapping)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #6656: minor: use sql to setup test data for joins.slt rather than rust
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6656:
URL: https://github.com/apache/arrow-datafusion/pull/6656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org