You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Rahul Kumar <rk...@gmail.com> on 2021/02/02 19:29:09 UTC

Spark 3 datasource v2: Can't extract user provided schema Dataframewriter save operation

Dear friends,I'm implementing datasource v2 for a custom NoSql database. I'm
facing following issuea) It seems while doing save operation, there is no
way to access user specified schema on dataframe. There is an existing
unresolved ongoing conversation  here
<http://apache-spark-user-list.1001560.n3.nabble.com/Implementing-TableProvider-in-Spark-3-0-td38113.html> 
.b)I have also noticed that it is not possible to extract write modes
etc(save, append, overwrite, error_if_exists ..)  from the dataframewriter.
I believe, I can force usesrs to supply these things through options I'm
still curious if there is cleaner way to access this information.Please shed
some light. Thank you very much, my next release is dependent on it lol
Best,Rahul  



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/