You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/21 13:44:25 UTC
[GitHub] [iceberg] felixYyu opened a new issue, #4603: Spark: Ignore SQL comments when testing error
felixYyu opened a new issue, #4603:
URL: https://github.com/apache/iceberg/issues/4603
link #4289 and #4396
- Test step in spark sql(v3.2.1) in Latest Master branch
```
def testStep(spark: SparkSession): Unit = {
spark.sql(
s"""
|
|--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (1, 11, 21)
|ALTER TABLE hadoop_prod.default.test_evolution5 ADD PARTITION FIELD col2
|--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (2, 12, 22)
|
|--ALTER TABLE hadoop_prod.default.test_evolution5 DROP PARTITION FIELD col2
|--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (3, 13, 23)
|--ALTER TABLE hadoop_prod.default.test_evolution5 DROP COLUMN col2
|""".stripMargin)
}
```
- exception log
```
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
missing '(' at 'FIELD'(line 4, pos 62)
== SQL ==
--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (1, 11, 21)
ALTER TABLE hadoop_prod.default.test_evolution5 ADD PARTITION FIELD col2
--------------------------------------------------------------^^^
--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (2, 12, 22)
--ALTER TABLE hadoop_prod.default.test_evolution5 DROP PARTITION FIELD col2
--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (3, 13, 23)
--ALTER TABLE hadoop_prod.default.test_evolution5 DROP COLUMN col2
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:266)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:127)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:51)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:77)
at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:123)
at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at com.jiuye.data.lake.api.modle.DDLPartitionSQL$.testStep(DDLPartitionSQL.scala:176)
at com.jiuye.data.lake.api.modle.DDLPartitionSQL$.main(DDLPartitionSQL.scala:44)
at com.jiuye.data.lake.api.modle.DDLPartitionSQL.main(DDLPartitionSQL.scala)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105866438
thanks @kbendick for detail description. I just want not to write multiple spark sql commands during single step debugging.
like the following is ok
```
spark.sql(
s"""
|
|/*INSERT INTO hadoop_prod.default.test_evolution5 VALUES (1, 11, 21)*/
|--ALTER TABLE hadoop_prod.default.test_evolution5 ADD PARTITION FIELD col2
|INSERT INTO hadoop_prod.default.test_evolution5 VALUES (2, 12, 22)
|
|--ALTER TABLE hadoop_prod.default.test_evolution5 DROP PARTITION FIELD col2
|--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (3, 13, 23)
|--ALTER TABLE hadoop_prod.default.test_evolution5 DROP COLUMN col2
|""".stripMargin)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105817050
I tested with your multiline string and it does result in SQL that has a leading `"`, which would cause the `isIcebergCommand` function to fail as it just checks the text starts with `call` or `alter table` after removing comments and new lines.
That's also why your stack trace doesn't have the Iceberg spark session extension stuff in it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105820871
Check out this multi-line string Scala example, which yours should look more like (no leading `|`): https://www.oreilly.com/library/view/scala-cookbook/9781449340292/ch01s03.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105811945
In fact, I think that your multiline string is winding up with a leading `"` character.
It should have the first line without `|` and then all following lines should have that.
Compare when placing the first text on the first line when using strip-margin
```scala
scala> :paste
// Entering paste mode (ctrl-D to finish)
val speech =
s"""Four score and
|seven years ago""".stripMargin
// Exiting paste mode, now interpreting.
speech: String =
Four score and
seven years ago
```
vs
```scala
scala> :paste
// Entering paste mode (ctrl-D to finish)
val speech =
s"""
| Four score and
|seven years ago""".stripMargin
// Exiting paste mode, now interpreting.
speech: String =
"
Four score and
seven years ago"
```
Notice that the second one has an additional leading `"`. I think that's what's causing problems in your multiline string.
Can you try evaluating the multiline string in a shell and seeing what it comes out to? I think that's the problem (the text needs to start on the first line for multiline strings in Scala).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105795552
Does this work without the comments @felixYyu?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105232502
cc: @kbendick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105869035
the following also don't work, exception log is:
```
spark.sql(
s"""
| CREATE TABLE tbl(id bigint, data string) using parquet;
| INSERT INTO tbl VALUES (0, 'kyle');
|""".stripMargin
```
```
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input 'INSERT' expecting {<EOF>, ';'}(line 3, pos 1)
== SQL ==
CREATE TABLE tbl(id bigint, data string) using parquet;
INSERT INTO tbl VALUES (0, 'kyle');
-^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:266)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:127)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:51)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:77)
at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:123)
at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at com.jiuye.data.lake.api.modle.DDLPartitionSQL$.testStep(DDLPartitionSQL.scala:188)
at com.jiuye.data.lake.api.modle.DDLPartitionSQL$.main(DDLPartitionSQL.scala:44)
at com.jiuye.data.lake.api.modle.DDLPartitionSQL.main(DDLPartitionSQL.scala)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105800583
Oh. Actually, I believe that you can't have multiple Iceberg DDL statements in one call to `spark.sql`.
I think that's true for normal spark sql as well (e.g. without using Iceberg in it at all). Like you couldn't do something like the following:
```scala
spark.sql(
s"""
| CREATE TABLE tbl(id bigint, data string) using parquet;
| INSERT INTO tbl VALUES (0, 'kyle');
|""".stripMargin
)
```
If you try to do the above and it works, let me know. Otherwise, I think the issue is that you need to have only executed statement in each call to `spark.sql`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
felixYyu commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105875533
put the first statement on the first line with no leading |, does that work? I suspect it will.
> Really, thanks. then I'll close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] felixYyu closed issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
felixYyu closed issue #4603: Spark: Ignore SQL comments when testing error
URL: https://github.com/apache/iceberg/issues/4603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4603: Spark: Ignore SQL comments when testing error
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4603:
URL: https://github.com/apache/iceberg/issues/4603#issuecomment-1105869243
So if you took the original multiline SQL you had, but put the first statement on the first line with no leading `|`, does that work? I suspect it will.
I’m just trying to clarify as I don’t think the issue has anything to do with the comment stripping. It would have happened before that.
Even try just
```
spark.sql(
s"""
|
|ALTER TABLE hadoop_prod.default.test_evolution5 ADD PARTITION FIELD col2
|/*INSERT INTO hadoop_prod.default.test_evolution5 VALUES (1, 11, 21)*/
|--ALTER TABLE hadoop_prod.default.test_evolution5 ADD PARTITION FIELD col2
|--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (2, 12, 22)
|
|--ALTER TABLE hadoop_prod.default.test_evolution5 DROP PARTITION FIELD col2
|--INSERT INTO hadoop_prod.default.test_evolution5 VALUES (3, 13, 23)
|--ALTER TABLE hadoop_prod.default.test_evolution5 DROP COLUMN col2
|""".stripMargin)
```
I know why it’s failing but I want to verify that before going on a deep explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org