[GitHub] [beam] mosche commented on a diff in pull request #23075: add a new IO named DataLakeIO (#23074)

[GitBox <gi...@apache.org>]

mosche commented on code in PR #23075:
URL: https://github.com/apache/beam/pull/23075#discussion_r978335634


##########
sdks/java/io/datalake/build.gradle:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.datalake')
+provideIntegrationTestingDependencies()
+enableJavaPerformanceTesting()
+
+description = "Apache Beam :: SDKs :: Java :: IO :: Datalake"
+ext.summary = "IO to read from and write to Data Lake"
+
+dependencies {
+    implementation project(path: ":sdks:java:core", configuration: "shadow")
+//    implementation library.java.slf4j_api
+    implementation library.java.vendored_guava_26_0_jre
+    implementation "org.apache.spark:spark-sql_2.12:3.1.2"
+    implementation "org.apache.spark:spark-core_2.12:3.1.2"
+    implementation "org.apache.spark:spark-streaming_2.12:3.1.2"

Review Comment:
   This IO would be really neat and I understand the motivation of using Spark underneath.
   Nevertheless, the spark dependency is rather problematic and I'm very concerned about the consequences ... 
   
   There's also a Spark runner, which supports both Spark 2.4 and Spark >= 3.1. This IO would certainly conflict with the Spark 2.4 runner. The Spark 3 runner is build in a way that it supports various versions of Spark 3 (the path from 3.1 to 3.3 is full of breaking changes), Spark dependencies are typically provided (as available on the cluster). Even further, Spark comes with a massive tail of dependencies prone to causing conflicts with versions used in Beam.
   
   The one common candidate to mention here is Avro. Spark 3.1 is still using Avro 1.8 matching Beam's version, Spark 3.2 bumps Avro to 1.10 which is incompatible with Beam :/ This kinda exemplifies the maintenance headache ahead.
   
   Have you evaluated any alternative to using Spark underneath?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org