You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/21 01:34:02 UTC
[GitHub] [hudi] jiezi2026 opened a new issue, #6156: [SUPPORT] DISTRIBUTE BY is not supported(line 59:undefined, pos 0) when using hudi-0.11.1 & spark-3.2.1
jiezi2026 opened a new issue, #6156:
URL: https://github.com/apache/hudi/issues/6156
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
When without conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' ,start a sparksql application by "/opt/apache/SPARK/SPARK-CURRENT/bin/spark-sql --num-executors 5 --queue=root.bi --conf spark.executor.cores=3 --conf spark.driver.memory=2G --conf spark.executor.memory=5G --conf spark.executor.memoryOverhead=2G"
-------------------[sparksql]---------------------------
select 1 distribute by rand()
-------------------[sparksql]---------------------------
The SQL execution results are as follows:
But when conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' is added,start another application by "/opt/apache/SPARK/SPARK-CURRENT/bin/spark-sql --num-executors 5 --queue=root.bi --conf spark.executor.cores=3 --conf spark.driver.memory=2G --conf spark.executor.memory=5G --conf spark.executor.memoryOverhead=2G --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
-------------------[sparksql]---------------------------
select 1 distribute by rand()
-------------------[sparksql]---------------------------
Error operating EXECUTE_STATEMENT: org.apache.spark.sql.catalyst.parser.ParseException: DISTRIBUTE BY is not supported(line 1:undefined, pos 9)
**It makes it impossible for me to use distribute by on other non Hudi tables**
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.11.1
* Spark version :3.2.1
* Hive version :2.1.1-cdh6.3.2
* Hadoop version :3.0.0-cdh6.3.2
* Storage (HDFS/S3/GCS..) :HDFS
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6156: [SUPPORT] DISTRIBUTE BY is not supported(line 59:undefined, pos 0) when using hudi-0.11.1 & spark-3.2.1
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6156:
URL: https://github.com/apache/hudi/issues/6156#issuecomment-1229344195
closing it out since the PR is landed. thanks @KnightChess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #6156: [SUPPORT] DISTRIBUTE BY is not supported(line 59:undefined, pos 0) when using hudi-0.11.1 & spark-3.2.1
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #6156:
URL: https://github.com/apache/hudi/issues/6156#issuecomment-1203486481
#6033 will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #6156: [SUPPORT] DISTRIBUTE BY is not supported(line 59:undefined, pos 0) when using hudi-0.11.1 & spark-3.2.1
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #6156: [SUPPORT] DISTRIBUTE BY is not supported(line 59:undefined, pos 0) when using hudi-0.11.1 & spark-3.2.1
URL: https://github.com/apache/hudi/issues/6156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org