Alter table concatenate - unsupported

[Arun Manivannan <ar...@arunma.com>]

Hi,

Good morning.

I am using Spark batch to process and ingest extracts of several RDBMS
tables/Filebased systems arriving in regular intervals into a Datalake as
ORC backed Hive tables. Considering that the input data file size, file
count, row count and feature counts vary quite a lot, I am unable to come
up with an optimal number for the coalesce.

I felt that the "alter table concatenate" is an easy way out to work around
the small files issue on NN that we are facing.

Sorry about the long story -  I bumped into this issue earlier today
- Alter table concatenate is not working as expected (SPARK-20592
<https://issues.apache.org/jira/browse/SPARK-20592>).  After some analysis
of the sql module, I found that the concatenate operation is consciously
marked as one of unsupportedHiveNativeCommands in the Antlr grammar.

Please let me know if you have strong reservations against enabling this? I
can take a stab at it and have a PR for review.

Cheers,
Arun