Allowing all Reader or Writer settings to be provided as options

[Nicholas Chammas <ni...@gmail.com>]

Hello people,

I want to bring some attention to SPARK-39630 <https://issues.apache.org/jira/browse/SPARK-39630> and ask if there are any design objections to the idea proposed there.

The gist of the proposal is that there are some reader or writer directives that cannot be supplied as options, like the format, write mode, or partitioning settings. Allowing those directives to be specified as options too means that it will become possible to fully represent a reader or writer as a map of options and reconstruct it from that.

This makes certain workflows more natural, especially when you are trying to manage reader or writer configurations declaratively.

Is there some design reason not to enable this, or is it just a matter of doing the work?

Feel free to comment either here or on the ticket.

Nick