[jira] [Created] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

["James Porritt (JIRA)" <ji...@apache.org>]

James Porritt created SPARK-22468:
-------------------------------------

             Summary: subtract creating empty DataFrame that isn't initialised properly 
                 Key: SPARK-22468
                 URL: https://issues.apache.org/jira/browse/SPARK-22468
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.2.0
            Reporter: James Porritt


I have an issue whereby a subtract between two DataFrames that will correctly end up with an empty DataFrame, seemingly has the DataFrame not initialised properly.

In my code I try and do a subtract both ways:

x = a.subtract(b)
y = b.subtract(a)

I then do an .rdd.isEmpty() on both of them to check if I need to do something with the results. Often the 'y' subtract will fail if the 'x' subtract is non-empty. It's hard to reproduce however. The error I will get is:

  File "<my spark script>", line 642, in <my function>
    if y.rdd.isEmpty():
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 1377, in isEmpty
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 1343, in take
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 992, in runJob
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 2455, in _jrdd
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 2390, in _wrap_function
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1386, in __call__
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1372, in _get_args
  File "<my base dir>/spark-2.2.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_collections.py", line 501, in convert
AttributeError: 'NoneType' object has no attribute 'add'

Sometimes the error will complain about it not having a 'size' parameter.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org