[jira] [Commented] (SPARK-12333) Support shuffle spill encryption in Spark

["Krish Dey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15652263#comment-15652263 ] 

Krish Dey commented on SPARK-12333:
-----------------------------------

The constructor still seems to be the same as it is. Doesn't this to be changed to accommodate encryption of spill to disk? Moreover passing the DummySerializerInstance it should be allowed to pass any Serializer

public UnsafeSorterSpillWriter(BlockManager blockManager, int fileBufferSize, ShuffleWriteMetrics writeMetrics, int numRecordsToWrite) throws IOException{
   final Tuple2<TempLocalBlockId, File> spilledFileInfo =
      blockManager.diskBlockManager().createTempLocalBlock();
    this.file = spilledFileInfo._2();
    this.blockId = spilledFileInfo._1();
    this.numRecordsToWrite = numRecordsToWrite;
    // Unfortunately, we need a serializer instance in order to construct a DiskBlockObjectWriter.
    // Our write path doesn't actually use this serializer (since we end up calling the `write()`
    // OutputStream methods), but DiskBlockObjectWriter still calls some methods on it. To work
    // around this, we pass a dummy no-op serializer.
    writer = blockManager.getDiskWriter(
      blockId, file, DummySerializerInstance.INSTANCE, fileBufferSize, writeMetrics);
    // Write the number of records
    writeIntToBuffer(numRecordsToWrite, 0);
    writer.write(writeBuffer, 0, 4);
  }


> Support shuffle spill encryption in Spark
> -----------------------------------------
>
>                 Key: SPARK-12333
>                 URL: https://issues.apache.org/jira/browse/SPARK-12333
>             Project: Spark
>          Issue Type: New Feature
>          Components: Shuffle
>            Reporter: Ferdinand Xu
>
> Like shuffle file encryption in SPARK-5682, spills data should also be encrypted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org