[GitHub] [spark] steveloughran commented on a change in pull request #30319: [SPARK-33402][CORE] SparkHadoopWriter to set unique job ID in "spark.sql.sources.writeJobUUID"

[GitBox <gi...@apache.org>]

steveloughran commented on a change in pull request #30319:
URL: https://github.com/apache/spark/pull/30319#discussion_r521441512



##########
File path: core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriterUtils.scala
##########
@@ -37,14 +36,39 @@ private[spark]
 object SparkHadoopWriterUtils {
 
   private val RECORDS_BETWEEN_BYTES_WRITTEN_METRIC_UPDATES = 256
+  private val RAND = new Random()
 
+  /**
+   * Create a job ID.
+   *
+   * @param time (current) time
+   * @param id job number
+   * @return a job ID
+   */
   def createJobID(time: Date, id: Int): JobID = {
+    if (id < 0) {
+      throw new IllegalArgumentException("Job number is negative")
+    }
     val jobtrackerID = createJobTrackerID(time)
     new JobID(jobtrackerID, id)
   }
 
+  /**
+   * Generate an ID for a job tracker.
+   * @param time (current) time
+   * @return a string for a job ID
+   */
   def createJobTrackerID(time: Date): String = {
-    new SimpleDateFormat("yyyyMMddHHmmss", Locale.US).format(time)
+    var l1 = RAND.nextLong()
+    if (l1 < 0) {
+      l1 = -l1
+    }
+    var l2 = RAND.nextLong()
+    if (l2 < 0) {
+      l2 = -l2
+    }
+    // use leading zeros to ensure the id is always at least four digits long
+    f"$l1%04d$l2"

Review comment:
       I did think about doing something here retaining date and time. If you want a random number at the end, happy to oblige. What's key is: it must be the digits 0-9 only.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org