You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2023/03/14 08:47:29 UTC
[GitHub] [spark] wangyum opened a new pull request, #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
wangyum opened a new pull request, #40409:
URL: https://github.com/apache/spark/pull/40409
### What changes were proposed in this pull request?
This PR ports the [tests](https://github.com/apache/hive/blob/rel/release-3.1.3/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFJson.java) for `get_json_object` from the Apache Hive project.
### Why are the changes needed?
Increase test coverage.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
N/A.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Hive compatibility check for get_json_object
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135395524
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
Review Comment:
Renamed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135384034
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
Review Comment:
Do you prefer "[SPARK-42782](https://issues.apache.org/jira/browse/SPARK-42782): Hive compatibility check for get_json_object"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135380192
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
Review Comment:
Why don't you name this test title like "SPARK-42782: Hive compatibility check for get_json_object"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #40409: [SPARK-42782][SQL][TESTS] Hive compatibility check for get_json_object
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #40409: [SPARK-42782][SQL][TESTS] Hive compatibility check for get_json_object
URL: https://github.com/apache/spark/pull/40409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135202529
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,93 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
Review Comment:
Spark does not support.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,93 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
+ runTest("[1,2,3]", "$[1]", "2")
+
+ runTest("[1,2,3]", "$[3]", null)
+ runTest("[1,2,3]", "$.[*]", null) // Not supported
Review Comment:
Spark does not support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135199655
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,93 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
+ runTest("[1,2,3]", "$[1]", "2")
+
+ runTest("[1,2,3]", "$[3]", null)
+ runTest("[1,2,3]", "$.[*]", null) // Not supported
+ runTest("[1,2,3]", "$[*]", "[1,2,3]")
+ runTest("[1,2,3]", "$", "[1,2,3]")
+
+ runTest("[{\"k1\":\"v1\"},{\"k2\":\"v2\"},{\"k3\":\"v3\"}]", "$[2]", "{\"k3\":\"v3\"}")
+ runTest("[{\"k1\":\"v1\"},{\"k2\":\"v2\"},{\"k3\":\"v3\"}]", "$[2].k3", "v3")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1[0].k11[1]", "2")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1[0].k11", "[1,2,3]")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1[0]", "{\"k11\":[1,2,3]}")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1", "[{\"k11\":[1,2,3]}]")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0]", "{\"k1\":[{\"k11\":[1,2,3]}]}")
+ runTest("[[1,2,3],[4,5,6],[7,8,9]]", "$[1]", "[4,5,6]")
+ runTest("[[1,2,3],[4,5,6],[7,8,9]]", "$[1][0]", "4")
+ runTest("[\"a\",\"b\"]", "$[1]", "b")
+ runTest("[[\"a\",\"b\"]]", "$[0][1]", "b")
+
+ runTest("[1,2,3]", "[0]", null)
+ runTest("[1,2,3]", "$0", null)
+ runTest("[1,2,3]", "0", null)
+ runTest("[1,2,3]", "$.", null)
+
+ runTest("[1,2,3]", "$", "[1,2,3]")
+ runTest("{\"a\":4}", "$", "{\"a\":4}")
Review Comment:
Hive's test: https://github.com/apache/hive/blob/rel/release-3.1.3/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFJson.java#L31-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135199655
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,93 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
+ runTest("[1,2,3]", "$[1]", "2")
+
+ runTest("[1,2,3]", "$[3]", null)
+ runTest("[1,2,3]", "$.[*]", null) // Not supported
+ runTest("[1,2,3]", "$[*]", "[1,2,3]")
+ runTest("[1,2,3]", "$", "[1,2,3]")
+
+ runTest("[{\"k1\":\"v1\"},{\"k2\":\"v2\"},{\"k3\":\"v3\"}]", "$[2]", "{\"k3\":\"v3\"}")
+ runTest("[{\"k1\":\"v1\"},{\"k2\":\"v2\"},{\"k3\":\"v3\"}]", "$[2].k3", "v3")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1[0].k11[1]", "2")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1[0].k11", "[1,2,3]")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1[0]", "{\"k11\":[1,2,3]}")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0].k1", "[{\"k11\":[1,2,3]}]")
+ runTest("[{\"k1\":[{\"k11\":[1,2,3]}]}]", "$[0]", "{\"k1\":[{\"k11\":[1,2,3]}]}")
+ runTest("[[1,2,3],[4,5,6],[7,8,9]]", "$[1]", "[4,5,6]")
+ runTest("[[1,2,3],[4,5,6],[7,8,9]]", "$[1][0]", "4")
+ runTest("[\"a\",\"b\"]", "$[1]", "b")
+ runTest("[[\"a\",\"b\"]]", "$[0][1]", "b")
+
+ runTest("[1,2,3]", "[0]", null)
+ runTest("[1,2,3]", "$0", null)
+ runTest("[1,2,3]", "0", null)
+ runTest("[1,2,3]", "$.", null)
+
+ runTest("[1,2,3]", "$", "[1,2,3]")
+ runTest("{\"a\":4}", "$", "{\"a\":4}")
Review Comment:
Hive's test: https://github.com/apache/hive/blob/rel/release-3.1.3/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFJson.java#L31-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135202529
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,93 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
Review Comment:
Spark does not support.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,93 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
+ runTest("[1,2,3]", "$[1]", "2")
+
+ runTest("[1,2,3]", "$[3]", null)
+ runTest("[1,2,3]", "$.[*]", null) // Not supported
Review Comment:
Spark does not support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135237148
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
Review Comment:
Spark does not support.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
+ val book0 = "{\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\"" +
+ ",\"category\":\"reference\",\"price\":8.95}"
+ val backet0 = "[1,2,{\"b\":\"y\",\"a\":\"x\"}]"
+ val backet = "[" + backet0 + ",[3,4],[5,6]]"
+ val backetFlat = backet0.substring(0, backet0.length() - 1) + ",3,4,5,6]"
+
+ val book = "[" + book0 + ",{\"author\":\"Herman Melville\",\"title\":\"Moby Dick\"," +
+ "\"category\":\"fiction\",\"price\":8.99" +
+ ",\"isbn\":\"0-553-21311-3\"},{\"author\":\"J. R. R. Tolkien\"" +
+ ",\"title\":\"The Lord of the Rings\",\"category\":\"fiction\"" +
+ ",\"reader\":[{\"age\":25,\"name\":\"bob\"},{\"age\":26,\"name\":\"jack\"}]" +
+ ",\"price\":22.99,\"isbn\":\"0-395-19395-8\"}]"
+
+ val json = "{\"store\":{\"fruit\":[{\"weight\":8,\"type\":\"apple\"}," +
+ "{\"weight\":9,\"type\":\"pear\"}],\"basket\":" + backet + ",\"book\":" + book +
+ ",\"bicycle\":{\"price\":19.95,\"color\":\"red\"}}" +
+ ",\"email\":\"amy@only_for_json_udf_test.net\"" +
+ ",\"owner\":\"amy\",\"zip code\":\"94025\",\"fb:testid\":\"1234\"}"
+
+ // Basic test
+ runTest(json, "$.owner", "amy")
+ runTest(json, "$.store.bicycle", "{\"price\":19.95,\"color\":\"red\"}")
+ runTest(json, "$.store.book", book)
+ runTest(json, "$.store.book[0]", book0)
+ runTest(json, "$.store.book[*]", book)
+ runTest(json, "$.store.book[0].category", "reference")
+ runTest(json, "$.store.book[*].category", "[\"reference\",\"fiction\",\"fiction\"]")
+ runTest(json, "$.store.book[*].reader[0].age", "25")
+ runTest(json, "$.store.book[*].reader[*].age", "[25,26]")
+ runTest(json, "$.store.basket[0][1]", "2")
+ runTest(json, "$.store.basket[*]", backet)
+ runTest(json, "$.store.basket[*][0]", "[1,3,5]")
+ runTest(json, "$.store.basket[0][*]", backet0)
+ runTest(json, "$.store.basket[*][*]", backetFlat)
+ runTest(json, "$.store.basket[0][2].b", "y")
+ runTest(json, "$.store.basket[0][*].b", "[\"y\"]")
+ runTest(json, "$.non_exist_key", null)
+ runTest(json, "$.store.book[10]", null)
+ runTest(json, "$.store.book[0].non_exist_key", null)
+ runTest(json, "$.store.basket[*].non_exist_key", null)
+ runTest(json, "$.store.basket[0][*].non_exist_key", null)
+ runTest(json, "$.store.basket[*][*].non_exist_key", null)
+ runTest(json, "$.zip code", "94025")
+ runTest(json, "$.fb:testid", "1234")
+ runTest("{\"a\":\"b\nc\"}", "$.a", "b\nc")
+
+ // Test root array
+ runTest("[1,2,3]", "$[0]", "1")
+ runTest("[1,2,3]", "$.[0]", null) // Not supported
+ runTest("[1,2,3]", "$.[1]", null) // Not supported
+ runTest("[1,2,3]", "$[1]", "2")
+
+ runTest("[1,2,3]", "$[3]", null)
+ runTest("[1,2,3]", "$.[*]", null) // Not supported
Review Comment:
Spark does not support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135238635
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
Review Comment:
Hive's test: https://github.com/apache/hive/blob/rel/release-3.1.3/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFJson.java#L31-L110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40409: [SPARK-42782][SQL][TESTS] Port the tests for get_json_object from the Apache Hive project
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40409:
URL: https://github.com/apache/spark/pull/40409#discussion_r1135386336
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala:
##########
@@ -81,6 +81,90 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession {
expected)
}
+ test("SPARK-42782: Port get_json_object's tests from the Apache Hive project") {
Review Comment:
Yeah but I don't mind if you don't prefer that. Will leave it to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org