ensure that format specification in Metadata is respected

[ekoifman]

ensure that Metadata.format.options is respected - closes #362

[databricks-cla-assistant]

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[ekoifman]

@rahulsmahadev @tdas could someone take a look at this please?

[tdas]

@ekoifman Sorry for not responding to this back then. My thought on this change is that it is risky to transparently pass all options down into the parquet format. There are a few subtle assumptions that delta lake makes about parquet format (some details about how ints and longs are stored) which are intricately tied to delta... so it might be possible to break the compatibility of the delta table by making arbitrary changes to parquet by allowing arbitrary options. To safeguard against such corner case we are currently not allowing arbitrary options to be passed down.

However, that said, I totally buy that there are options that can super useful to pass down. We could built a allow-list of options to be passed down. If you are still interested in working on this, we can think in that direction. What do you think?

[ekoifman]

@tdas Sure let me take a look. Do you have a list in mind?
Also, I'd expect the list to be hardcoded since no end user is expected to configure the list.

[felipepessoto]

Related:
#6324
#6535

Seems only supported by Kernel