[GitHub] [kylin] zzcclp commented on a change in pull request #1514: KYLIN-4818 Check lookup table duplicate key when building job

[GitBox <gi...@apache.org>]

zzcclp commented on a change in pull request #1514:
URL: https://github.com/apache/kylin/pull/1514#discussion_r544857040



##########
File path: kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/CubeSnapshotBuilder.scala
##########
@@ -183,6 +185,22 @@ class CubeSnapshotBuilder extends Logging {
   }
   import org.apache.kylin.engine.spark.utils.SparkDataSource._
 
+  def checkDupKey() = {
+    val joinDescs = seg.joindescs
+    joinDescs.foreach {
+      joinDesc =>
+        val tableInfo = joinDesc.lookupTable
+        val lookupTableName = tableInfo.tableName
+        val df = ss.table(tableInfo)
+        val countColumn = df.count()
+        val lookupTablePKS = joinDesc.PKS.map(lookupTablePK => lookupTablePK.columnName)
+        val countDistinctColumn = df.agg(countDistinct(joinDesc.PKS(0).columnName, lookupTablePKS: _*)).collect().map(_(0)).toList(0)

Review comment:
       better change to : `df.agg(countDistinct(lookupTablePKS(0), lookupTablePKS.tail : _*)).collect().map(_.getLong(0)).head`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org