You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by orenmazor <gi...@git.apache.org> on 2014/12/08 23:14:51 UTC

[GitHub] spark pull request: Cdh5

GitHub user orenmazor opened a pull request:

    https://github.com/apache/spark/pull/3639

    Cdh5

    https://github.com/Shopify/dataops/issues/2

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Shopify/spark cdh5

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/3639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3639
    
----
commit 422de4cc2a823e16b86fd22095e35d1ebe842a12
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-15T01:29:43Z

    Add compile script for packserv

commit 4ffa04cc6cc7bb8086a422a94d4f2e4105a69786
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-15T02:18:29Z

    Don't compile streaming when assembling cause it doesn't build against CDH4.4.0

commit b7bf08171e8eb796d86408ce5712175d781e0f8d
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-15T02:22:10Z

    Make script compile executable

commit 65033e665c75f4e82b56c8113c99308f8b419704
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-15T02:22:57Z

    Make script compile bash

commit 9e6fc96f461864f4ffdd6c8aefaa53b6fd8c4ae0
Author: Mark Cooper <mc...@quantcast.com>
Date:   2013-11-20T22:26:42Z

    Add a environment variable that allows for configuring a different path to Spark binaries when running Spark from a different location locally

commit fdb0ce298048832f75b24b464fdf59fb791f869f
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-15T19:24:05Z

    Add fixed conf file with proper master and remote spark home

commit a837356d7d84641ab504522e74cedc4b5d865aa3
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T01:50:22Z

    Copy in hadoop core-site.xml so local clones know where to find hdfs.

commit 4d0f3682e0931c21ba6e5b01fc42ee33a44453e1
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T02:12:36Z

    Update the given spark env to actually work, and only if a custom master isn't provided.

commit 01cf4c51f2c3c3089ee91dd64d6cab32dd17aa70
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T04:22:11Z

    Allow controlling the number of cores pyspark uses using the `-c` option, like spark-shell.
     
     - Turns out there isn't actually a way right now to control the number of cores an interactive pyspark session uses, which is annoying if more than one person is trying to work on a cluster interactively at once. 
     - Use the python 2.7 stdlib argparse library to pull out the -c option
     - This requires changing the bin/pyspark shell script to pass all arguments to the python script instead of allowing the python interpreter program to parse any of them.

commit 91ddfb4c43a88a4cf0082e445e2e82bcde069969
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T04:22:34Z

    Merge branch 'pyspark_cores'

commit 0b44511492131b60f744527eee467fd147e4f4c0
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T14:51:05Z

    Revert "Merge branch 'pyspark_cores'"
    
    This reverts commit 91ddfb4c43a88a4cf0082e445e2e82bcde069969, reversing
    changes made to 4d0f3682e0931c21ba6e5b01fc42ee33a44453e1.

commit b4c5ff7e7d6d550743e3aa97710fa514744b0c6e
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T15:22:09Z

    Auto setup python and warn if the vpn isn't connected

commit 4c2c45eaf14197b79cf5949bb370a74c52a38ff0
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T15:34:05Z

    Add an applescript to the spark conf file that autoconnects the VPN if it can't find the interface the VPN should create

commit 712b8856e4b14f88d34da569505c59884d8e8155
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T19:24:21Z

    Check to see if Viscosity is a thing before trying to tell it to connect in spark env setup

commit 986a60c0b9a880ee0fd7242e53458efbebfab73e
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-21T21:21:18Z

    Merge pull request #1 from Shopify/autoconnect_vpn
    
    Autoconnect VPN

commit b944fc6ff5f5866b93e937f7d7629370c24944f0
Author: Dana Klassen <kl...@gmail.com>
Date:   2014-01-23T01:36:53Z

    change configuration to be set through environment variable

commit 5eace91604360da5b446a96582f141b09ab109c1
Author: Erik Selin <er...@jadedpixel.com>
Date:   2014-01-23T04:14:17Z

    apply pr 494 and 496

commit 42dc1708daec21a3ba302f61f473afa57fb5c12c
Author: Dana Klassen <kl...@gmail.com>
Date:   2014-01-23T12:15:27Z

    Merge pull request #2 from Shopify/config_hdfs
    
    Config hdfs

commit 2b6c170b50f58ccdbe1e2faaf4ff3439bdf9e01e
Author: Erik Selin <ty...@gmail.com>
Date:   2014-01-23T15:28:32Z

    Merge pull request #3 from Shopify/apply_494_and_496
    
    apply pr 494 and 496

commit 25c5a0d90c5926133b32e43f5e6a8d1a58c0685c
Author: Patrick Wendell <pw...@gmail.com>
Date:   2014-01-15T18:58:02Z

    Workers should use working directory as spark home if it's not specified

commit c0cd09186b5cef2f15f3dfb4bd9dadea6392385c
Author: Erik Selin <ty...@gmail.com>
Date:   2014-01-24T00:37:45Z

    Merge pull request #5 from Shopify/apply_commit_00a3f7e
    
    Workers should use working directory as spark home if it's not specified

commit df6098e1254a5d7128574d2c2fc31dae005c8fd7
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-24T00:14:06Z

    Fix broken quoting in spark-env.sh and echo to stderr to not break stdout communication with pyspark

commit 1b71ba315487b3bb6214631fece93cbd8ef27fb3
Author: Erik Selin <er...@jadedpixel.com>
Date:   2014-01-26T19:47:37Z

    Add custom writeUTF method to PythonRDD and make it replace DataOutputStream.writeUTF so that we can use items with greater size in pyspark. Also modify pyspark MUTF8Deserializer to expect the size of the data blocks to be represented by an integer instead of an unsigned short.

commit af58a246f3babb5848c248eb133e88336c15d3ec
Author: Erik Selin <er...@jadedpixel.com>
Date:   2014-01-26T22:21:39Z

    remove alignment

commit 39330f3a41dd6ccdd8a8211ea36e33b5600e27ec
Author: Erik Selin <er...@jadedpixel.com>
Date:   2014-01-26T22:38:27Z

    remove using MUTF to send data to pyspark and use UTF from getBytes instead.

commit 80c976e3c5befcc02f5581a15058afd825fe16e2
Author: Bouke van der Bijl <bo...@gmail.com>
Date:   2014-01-27T16:11:23Z

    Add test for writing large strings

commit 9de42052c8c622d11d20e82bdf18f312fc1e0d0e
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-01-27T21:34:26Z

    Merge pull request #6 from Shopify/conf_fixes_for_standalone_pyspark
    
    Fix broken quoting in spark-env.sh and echo to stderr to not break stdout communication with pyspark

commit 57d86ee81ac02a9c8357ab56c16c0577491bc4c4
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-02-05T02:57:38Z

    Remove erroneous local SPARK_HOME environment variable setting

commit 47e44f87963e5ba0c9fa51f14db55d1a06f6fa38
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-02-05T03:24:15Z

    Don't let spark spend time figuring out what IP to bind the web server to when executing in local mode

commit 812a7880938a415e7aa89d9e82e13dc827b7d119
Author: Harry Brundage <ha...@gmail.com>
Date:   2014-02-05T03:25:09Z

    Get rid of kerberos auth testing. No more "Unable to load realm info from SCDynamicStore"! Woo!
    
    See http://stackoverflow.com/questions/7134723/hadoop-on-osx-unable-to-load-realm-info-from-scdynamicstore

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: Cdh5

Posted by orenmazor <gi...@git.apache.org>.

Github user orenmazor closed the pull request at:

    https://github.com/apache/spark/pull/3639


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org