You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/28 03:07:07 UTC
[GitHub] [incubator-seatunnel] 2416210017 opened a new issue, #2926: Seatunnel-2.1.3 How to specify a spark file?
2416210017 opened a new issue, #2926:
URL: https://github.com/apache/incubator-seatunnel/issues/2926
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Seatunnel-2.1.3 How to specify a spark file? Spark2.4 is used. The hive table cannot be found. The hive site is specified Xml is normal.
![Uploading image.png…]()
### SeaTunnel Version
Seatunnel-2.1.3
### SeaTunnel Config
```conf
cat ads_product_sale_d.conf
env {
spark.app.name = "ads_product_sale_d"
spark.executor.instances = 1
spark.dynamicAllocation.maxExecutors = 20
spark.executor.cores = 1
spark.executor.memory = "8g"
spark.sql.catalogImplementation = "hive"
spark.sql.hive.verifyPartitionPath = "true"
}
source {
hive {
pre_sql = "select platform_id,cat_id from dws.dws_prd_product_detail_d where day_id ='2022-05-31' "
result_table_name = "ads_product_sale_d"
}
}
transform {
sql {
sql = "select platform_id,cat_id from ads_product_sale_d",
table_name = "ads_product_sale_d"
}
}
sink {
Doris {
fenodes="172.17.18.95:8030"
database="example_db"
table="ads_product_sale_d"
user="test"
password="123456"
batch_size=10000
doris.column_separator="\t"
doris.columns="platform_id,cat_id"
}
}
```
### Running Command
```shell
su hive -s /bin/bash -c "/app/seatunnel-2.1.3/bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config /tmp/mysql_tt3.sh"
```
### Error Exception
```log
Caused by: org.apache.spark.sql.AnalysisException: Table or view not found: `mid`.`player_parquet`; line 1 pos 14;
```
### Flink or Spark Version
spark-2.4
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #2926: Seatunnel-2.1.3 How to specify a spark file?
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #2926: Seatunnel-2.1.3 How to specify a spark file?
URL: https://github.com/apache/incubator-seatunnel/issues/2926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 2416210017 commented on issue #2926: Seatunnel-2.1.3 How to specify a spark file?
Posted by GitBox <gi...@apache.org>.
2416210017 commented on issue #2926:
URL: https://github.com/apache/incubator-seatunnel/issues/2926#issuecomment-1260451574
Yes, can I help you? thank you
> Do you want some other people help you fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2926: Seatunnel-2.1.3 How to specify a spark file?
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2926:
URL: https://github.com/apache/incubator-seatunnel/issues/2926#issuecomment-1304356173
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2926: Seatunnel-2.1.3 How to specify a spark file?
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2926:
URL: https://github.com/apache/incubator-seatunnel/issues/2926#issuecomment-1260448865
Do you want some other people help you fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #2926: Seatunnel-2.1.3 How to specify a spark file?
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2926:
URL: https://github.com/apache/incubator-seatunnel/issues/2926#issuecomment-1295659769
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org