[GitHub] spark pull request #23034: [SPARK-26035][PYTHON] Break large streaming/tests...

[HyukjinKwon <gi...@git.apache.org>]

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/23034#discussion_r233829511
  
    --- Diff: python/pyspark/testing/streamingutils.py ---
    @@ -0,0 +1,189 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +# http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +import glob
    +import os
    +import tempfile
    +import time
    +import unittest
    +
    +from pyspark import SparkConf, SparkContext, RDD
    +from pyspark.streaming import StreamingContext
    +
    +
    +def search_kinesis_asl_assembly_jar():
    +    kinesis_asl_assembly_dir = os.path.join(
    +        os.environ["SPARK_HOME"], "external/kinesis-asl-assembly")
    +
    +    # We should ignore the following jars
    +    ignored_jar_suffixes = ("javadoc.jar", "sources.jar", "test-sources.jar", "tests.jar")
    +
    +    # Search jar in the project dir using the jar name_prefix for both sbt build and maven
    +    # build because the artifact jars are in different directories.
    +    name_prefix = "spark-streaming-kinesis-asl-assembly"
    +    sbt_build = glob.glob(os.path.join(
    +        kinesis_asl_assembly_dir, "target/scala-*/%s-*.jar" % name_prefix))
    +    maven_build = glob.glob(os.path.join(
    +        kinesis_asl_assembly_dir, "target/%s_*.jar" % name_prefix))
    +    jar_paths = sbt_build + maven_build
    +    jars = [jar for jar in jar_paths if not jar.endswith(ignored_jar_suffixes)]
    +
    +    if not jars:
    +        return None
    +    elif len(jars) > 1:
    +        raise Exception(("Found multiple Spark Streaming Kinesis ASL assembly JARs: %s; please "
    +                         "remove all but one") % (", ".join(jars)))
    +    else:
    +        return jars[0]
    +
    +
    +# Must be same as the variable and condition defined in KinesisTestUtils.scala and modules.py
    +kinesis_test_environ_var = "ENABLE_KINESIS_TESTS"
    +should_skip_kinesis_tests = not os.environ.get(kinesis_test_environ_var) == '1'
    +kinesis_asl_assembly_jar = search_kinesis_asl_assembly_jar()
    +
    +if should_skip_kinesis_tests:
    --- End diff --
    
    I simplified this logic and tested each if-else branch.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org