Dropping support for older Spark versions

[Eyal Allweil <ey...@apache.org>]

Hello DataFu,

The fix for DATAFU-167 (which allows us to support newer Spark 2.x
versions) requires the use of a py4j API that isn't used in the py4j
versions in older Spark versions. Therefore, we need to decide whether
we're fine with dropping support for 2.2.0, 2.2.1, and 2.3.0.

In exchange, we'll be able to support 2.2.2 - 2.2.3, 2.3.1 - 2.3.4 and
2.4.0 - 2.4.5.

I think this is fine, and the solution proposed in DATAFU-167 (by new
contributor Arpit!) should be accepted, and we can put the fix in upcoming
release 1.7.0.

You can look at the proposed solution easily here:
https://github.com/apache/datafu/compare/master...arpitbhardwaj:datafu:spark_upgrade_fix

Does anyone have any reservations? If not, I'd like to release 1.7.0 soon.

Cheers,
Eyal