You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Paul Wais <pw...@yelp.com> on 2014/07/17 01:03:08 UTC

Release date for new pyspark

Dear List,

The version of pyspark on master has a lot of nice new features, e.g.
SequenceFile reading, pickle i/o, etc:
https://github.com/apache/spark/blob/master/python/pyspark/context.py#L353

I downloaded the recent 1.0.1 release and was surprised to see the
distribution did not include these changes in master.  (I've tried pulling
master [ 9c249743ea ] and compiling from source, but I get a build failure
in TestSQLContext.scala FWIW).

Is an updated pyspark scheduled for the next release?  (Also, am I wrong in
expecting HEAD on master should probably compile and run?)

Best Regards,
-Paul Wais

Re: Release date for new pyspark

Posted by Paul Wais <pw...@yelp.com>.
Thanks all!  (And thanks Matei for the developer link!)  I was able to
build using maven[1] but `./sbt/sbt assembly` results in build errors.
(Not familiar enough with the build to know why; in the past sbt
worked for me and maven did not).

I was able to run the master version of pyspark, which was what I
wanted, though I discovered a bug when trying to read spark-pickled
data from HDFS.  (Looks similar to
https://spark-project.atlassian.net/browse/SPARK-1034 from my naive
point of view).  For the curious:

Code:

conf = SparkConf()
conf.set('spark.local.dir', '/nail/tmp')
conf.set('spark.executor.memory', '28g')
conf.set('spark.app.name', 'test')

sc = SparkContext(conf=conf)

sc.parallelize(range(10)).saveAsPickleFile("hdfs://host:9000/test_pickle")
unpickled_rdd = sc.pickleFile("hdfs://host:9000/test_pickle")
print unpickled_rdd.takeSample(False, 3)

Traceback (most recent call last):
  File "/path/to/my/home/spark-master/tast.py", line 33, in <module>
    print unpickled_rdd.takeSample(False, 3)
  File "/path/to/my/home/spark-master/python/pyspark/rdd.py", line
391, in takeSample
    initialCount = self.count()
  File "/path/to/my/home/spark-master/python/pyspark/rdd.py", line 791, in count
    return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
  File "/path/to/my/home/spark-master/python/pyspark/rdd.py", line 782, in sum
    return self.mapPartitions(lambda x: [sum(x)]).reduce(operator.add)
  File "/path/to/my/home/spark-master/python/pyspark/rdd.py", line
703, in reduce
    vals = self.mapPartitions(func).collect()
  File "/path/to/my/home/spark-master/python/pyspark/rdd.py", line
667, in collect
    bytesInJava = self._jrdd.collect().iterator()
  File "/path/to/my/home/spark-master/python/pyspark/rdd.py", line
1600, in _jrdd
    class_tag)
  File "/path/to/my/home/spark-master/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
line 669, in __call__
  File "/path/to/my/home/spark-master/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py",
line 304, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling
None.org.apache.spark.api.python.PythonRDD. Trace:
py4j.Py4JException: Constructor
org.apache.spark.api.python.PythonRDD([class
org.apache.spark.rdd.FlatMappedRDD, class [B, class java.util.HashMap,
class java.util.ArrayList, class java.lang.Boolean, class
java.lang.String, class java.util.ArrayList, class
org.apache.spark.Accumulator, class
scala.reflect.ManifestFactory$$anon$2]) does not exist
        at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:184)
        at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202)
        at py4j.Gateway.invoke(Gateway.java:213)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
        at py4j.GatewayConnection.run(GatewayConnection.java:207)
        at java.lang.Thread.run(Thread.java:662)


[1] mvn -Phadoop-2.3 -Dhadoop.verson=2.3.0 -DskipTests clean package

On Wed, Jul 16, 2014 at 8:39 PM, Michael Armbrust
<mi...@databricks.com> wrote:
> You should try cleaning and then building.  We have recently hit a bug in
> the scala compiler that sometimes causes non-clean builds to fail.
>
>
> On Wed, Jul 16, 2014 at 7:56 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
>>
>> Yeah, we try to have a regular 3 month release cycle; see
>> https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage for the
>> current window.
>>
>> Matei
>>
>> On Jul 16, 2014, at 4:21 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:
>>
>> You should expect master to compile and run: patches aren't merged unless
>> they build and pass tests on Jenkins.
>>
>> You shouldn't expect new features to be added to stable code in
>> maintenance releases (e.g. 1.0.1).
>>
>> AFAIK, we're still on track with Spark 1.1.0 development, which means that
>> it should be released sometime in the second half of next month (or shortly
>> thereafter).
>>
>>
>> On Wed, Jul 16, 2014 at 4:03 PM, Paul Wais <pw...@yelp.com> wrote:
>>>
>>> Dear List,
>>>
>>> The version of pyspark on master has a lot of nice new features, e.g.
>>> SequenceFile reading, pickle i/o, etc:
>>> https://github.com/apache/spark/blob/master/python/pyspark/context.py#L353
>>>
>>> I downloaded the recent 1.0.1 release and was surprised to see the
>>> distribution did not include these changes in master.  (I've tried pulling
>>> master [ 9c249743ea ] and compiling from source, but I get a build failure
>>> in TestSQLContext.scala FWIW).
>>>
>>> Is an updated pyspark scheduled for the next release?  (Also, am I wrong
>>> in expecting HEAD on master should probably compile and run?)
>>>
>>> Best Regards,
>>> -Paul Wais
>>
>>
>>
>

Re: Release date for new pyspark

Posted by Michael Armbrust <mi...@databricks.com>.
You should try cleaning and then building.  We have recently hit a bug in
the scala compiler that sometimes causes non-clean builds to fail.


On Wed, Jul 16, 2014 at 7:56 PM, Matei Zaharia <ma...@gmail.com>
wrote:

> Yeah, we try to have a regular 3 month release cycle; see
> https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage for the
> current window.
>
> Matei
>
> On Jul 16, 2014, at 4:21 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:
>
> You should expect master to compile and run: patches aren't merged unless
> they build and pass tests on Jenkins.
>
> You shouldn't expect new features to be added to stable code in
> maintenance releases (e.g. 1.0.1).
>
> AFAIK, we're still on track with Spark 1.1.0 development, which means that
> it should be released sometime in the second half of next month (or shortly
> thereafter).
>
>
> On Wed, Jul 16, 2014 at 4:03 PM, Paul Wais <pw...@yelp.com> wrote:
>
>> Dear List,
>>
>> The version of pyspark on master has a lot of nice new features, e.g.
>> SequenceFile reading, pickle i/o, etc:
>> https://github.com/apache/spark/blob/master/python/pyspark/context.py#L353
>>
>> I downloaded the recent 1.0.1 release and was surprised to see the
>> distribution did not include these changes in master.  (I've tried pulling
>> master [ 9c249743ea ] and compiling from source, but I get a build failure
>> in TestSQLContext.scala FWIW).
>>
>> Is an updated pyspark scheduled for the next release?  (Also, am I wrong
>> in expecting HEAD on master should probably compile and run?)
>>
>> Best Regards,
>> -Paul Wais
>>
>
>
>

Re: Release date for new pyspark

Posted by Matei Zaharia <ma...@gmail.com>.
Yeah, we try to have a regular 3 month release cycle; see https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage for the current window.

Matei

On Jul 16, 2014, at 4:21 PM, Mark Hamstra <ma...@clearstorydata.com> wrote:

> You should expect master to compile and run: patches aren't merged unless they build and pass tests on Jenkins.
> 
> You shouldn't expect new features to be added to stable code in maintenance releases (e.g. 1.0.1).
> 
> AFAIK, we're still on track with Spark 1.1.0 development, which means that it should be released sometime in the second half of next month (or shortly thereafter).
> 
> 
> On Wed, Jul 16, 2014 at 4:03 PM, Paul Wais <pw...@yelp.com> wrote:
> Dear List,
> 
> The version of pyspark on master has a lot of nice new features, e.g. SequenceFile reading, pickle i/o, etc: https://github.com/apache/spark/blob/master/python/pyspark/context.py#L353
> 
> I downloaded the recent 1.0.1 release and was surprised to see the distribution did not include these changes in master.  (I've tried pulling master [ 9c249743ea ] and compiling from source, but I get a build failure in TestSQLContext.scala FWIW).
> 
> Is an updated pyspark scheduled for the next release?  (Also, am I wrong in expecting HEAD on master should probably compile and run?)
> 
> Best Regards,
> -Paul Wais
> 


Re: Release date for new pyspark

Posted by Mark Hamstra <ma...@clearstorydata.com>.
You should expect master to compile and run: patches aren't merged unless
they build and pass tests on Jenkins.

You shouldn't expect new features to be added to stable code in maintenance
releases (e.g. 1.0.1).

AFAIK, we're still on track with Spark 1.1.0 development, which means that
it should be released sometime in the second half of next month (or shortly
thereafter).


On Wed, Jul 16, 2014 at 4:03 PM, Paul Wais <pw...@yelp.com> wrote:

> Dear List,
>
> The version of pyspark on master has a lot of nice new features, e.g.
> SequenceFile reading, pickle i/o, etc:
> https://github.com/apache/spark/blob/master/python/pyspark/context.py#L353
>
> I downloaded the recent 1.0.1 release and was surprised to see the
> distribution did not include these changes in master.  (I've tried pulling
> master [ 9c249743ea ] and compiling from source, but I get a build failure
> in TestSQLContext.scala FWIW).
>
> Is an updated pyspark scheduled for the next release?  (Also, am I wrong
> in expecting HEAD on master should probably compile and run?)
>
> Best Regards,
> -Paul Wais
>