You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 张健BJ <zh...@datagrand.com> on 2023/07/25 11:42:21 UTC
a problem about pyspark writing to trino
I can't find any way to write trino from pyspark now, pyspark 3.3.0, trino-jdbc-405 or trino-jdbc-379.but it can read.
Method 1: I tried to write through 'insert into' and found that the syntax did not report an error, but the data was not successfully written, and the submitted insert command was not visible on the cluster.
Method 2: I attempted to write through the 'save()' command and found that it directly reported an error. When I looked at the cluster log, I found that the code in jdbc first deleted the original table and then created a new table. However, the creation of the new table failed, and I tried this linking(https://stackoverflow.com/questions/72174037/pyspark-write-to-trino-with-jdbc-connector) method, it did not work.
The table creation statement is:
create table intf_nlp.nlp_platform_output_data (
id string,
result string
) PARTITIONED BY (date int, type int)
The screenshot is as follows:
1:
2: