You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Erik Krogen (Jira)" <ji...@apache.org> on 2021/04/28 20:13:00 UTC

[jira] [Created] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code

Erik Krogen created SPARK-35263:
-----------------------------------

             Summary: Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
                 Key: SPARK-35263
                 URL: https://issues.apache.org/jira/browse/SPARK-35263
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle, Tests
    Affects Versions: 3.1.1
            Reporter: Erik Krogen


{{ShuffleFetcherBlockIteratorSuite}} has tons of duplicate code, like:
{code}
    val iterator = new ShuffleBlockFetcherIterator(
      taskContext,
      transfer,
      blockManager,
      blocksByAddress,
      (_, in) => in,
      48 * 1024 * 1024,
      Int.MaxValue,
      Int.MaxValue,
      Int.MaxValue,
      true,
      false,
      metrics,
      false)
{code}
It's challenging to tell what the interesting parts are vs. what is just being set to some default/unused value.

Similarly but not as bad, there are 10 calls like:
{code}
verify(transfer, times(1)).fetchBlocks(any(), any(), any(), any(), any(), any())
{code}
and 7 like
{code}
when(transfer.fetchBlocks(any(), any(), any(), any(), any(), any())).thenAnswer ...
{code}

This can result in about 10% reduction in both lines and characters in the file:
{code}
# Before
> wc core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
    1063    3950   43201 core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala

# After
> wc core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
     928    3609   39053 core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
{code}

It also helps readability:
{code}
    val iterator = createShuffleBlockIteratorWithDefaults(
      transfer,
      blocksByAddress,
      maxBytesInFlight = 1000L
    )
{code}
Now I can clearly tell that {{maxBytesInFlight}} is the main parameter we're interested in here.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org