You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2015/02/11 23:27:29 UTC
1.2.1 start-all.sh broken?
I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran sbin/start-all.sh
on my OS X.
Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib
You need to build Spark before running this program.
Did the same for 1.2.0 and it worked fine.
Nick
Re: 1.2.1 start-all.sh broken?
Posted by Sean Owen <so...@cloudera.com>.
Seems to work OK for me on OS X. I ran ./sbin/start-all.sh from the
root. Both processes say they started successfully.
On Wed, Feb 11, 2015 at 10:27 PM, Nicholas Chammas
<ni...@gmail.com> wrote:
> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran sbin/start-all.sh
> on my OS X.
>
> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib
> You need to build Spark before running this program.
>
> Did the same for 1.2.0 and it worked fine.
>
> Nick
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: 1.2.1 start-all.sh broken?
Posted by Nicholas Chammas <ni...@gmail.com>.
SPARK-5747 <https://issues.apache.org/jira/browse/SPARK-5747>: Review all
Bash scripts for word splitting bugs
I’ll file sub-tasks under this issue. Feel free to pitch in people!
Nick
On Wed Feb 11 2015 at 3:07:51 PM Ted Yu <yu...@gmail.com> wrote:
> After some googling / trial and error, I got the following working
> (against a directory with space in its name):
>
> #!/usr/bin/env bash
> OLDIFS="$IFS" # save it
> IFS="" # don't split on any white space
> dir="$1/*"
> for f in "$dir"; do
> cat $f
> done
> IFS=$OLDIFS # restore IFS
>
> Cheers
>
> On Wed, Feb 11, 2015 at 2:47 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> The tragic thing here is that I was asked to review the patch that
>> introduced this
>> <https://github.com/apache/spark/pull/3377#issuecomment-68077315>, and
>> totally missed it... :(
>>
>> On Wed Feb 11 2015 at 2:46:35 PM Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> lol yeah, I changed the path for the email... turned out to be the issue
>>> itself.
>>>
>>>
>>> On Wed Feb 11 2015 at 2:43:09 PM Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> I see.
>>>> '/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-)
>>>>
>>>> On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> Found it:
>>>>>
>>>>> https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-
>>>>> 73058f8e51951ec0b4cb3d48ade91a1fR73
>>>>>
>>>>> GRRR BASH WORD SPLITTING
>>>>>
>>>>> My path has a space in it...
>>>>>
>>>>> Nick
>>>>>
>>>>> On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas <
>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>
>>>>>> This is what get:
>>>>>>
>>>>>> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
>>>>>> datanucleus-api-jdo-3.2.6.jar
>>>>>> datanucleus-core-3.2.10.jar
>>>>>> datanucleus-rdbms-3.2.9.jar
>>>>>> spark-1.2.1-yarn-shuffle.jar
>>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>>>
>>>>>> So that looks correct… Hmm.
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> I downloaded 1.2.1 tar ball for hadoop 2.4
>>>>>>> I got:
>>>>>>>
>>>>>>> ls lib/
>>>>>>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
>>>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>>>>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
>>>>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>>>>
>>>>>>> FYI
>>>>>>>
>>>>>>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
>>>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>>>
>>>>>>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>>>>>>>> sbin/start-all.sh
>>>>>>>> on my OS X.
>>>>>>>>
>>>>>>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoo
>>>>>>>> p2.4/lib
>>>>>>>> You need to build Spark before running this program.
>>>>>>>>
>>>>>>>> Did the same for 1.2.0 and it worked fine.
>>>>>>>>
>>>>>>>> Nick
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>
Re: 1.2.1 start-all.sh broken?
Posted by Ted Yu <yu...@gmail.com>.
After some googling / trial and error, I got the following working (against
a directory with space in its name):
#!/usr/bin/env bash
OLDIFS="$IFS" # save it
IFS="" # don't split on any white space
dir="$1/*"
for f in "$dir"; do
cat $f
done
IFS=$OLDIFS # restore IFS
Cheers
On Wed, Feb 11, 2015 at 2:47 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> The tragic thing here is that I was asked to review the patch that
> introduced this
> <https://github.com/apache/spark/pull/3377#issuecomment-68077315>, and
> totally missed it... :(
>
> On Wed Feb 11 2015 at 2:46:35 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> lol yeah, I changed the path for the email... turned out to be the issue
>> itself.
>>
>>
>> On Wed Feb 11 2015 at 2:43:09 PM Ted Yu <yu...@gmail.com> wrote:
>>
>>> I see.
>>> '/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-)
>>>
>>> On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> Found it:
>>>>
>>>> https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-
>>>> 73058f8e51951ec0b4cb3d48ade91a1fR73
>>>>
>>>> GRRR BASH WORD SPLITTING
>>>>
>>>> My path has a space in it...
>>>>
>>>> Nick
>>>>
>>>> On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> This is what get:
>>>>>
>>>>> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
>>>>> datanucleus-api-jdo-3.2.6.jar
>>>>> datanucleus-core-3.2.10.jar
>>>>> datanucleus-rdbms-3.2.9.jar
>>>>> spark-1.2.1-yarn-shuffle.jar
>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>>
>>>>> So that looks correct… Hmm.
>>>>>
>>>>> Nick
>>>>>
>>>>>
>>>>> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> I downloaded 1.2.1 tar ball for hadoop 2.4
>>>>>> I got:
>>>>>>
>>>>>> ls lib/
>>>>>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
>>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>>>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
>>>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>>>
>>>>>> FYI
>>>>>>
>>>>>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
>>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>>
>>>>>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>>>>>>> sbin/start-all.sh
>>>>>>> on my OS X.
>>>>>>>
>>>>>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoo
>>>>>>> p2.4/lib
>>>>>>> You need to build Spark before running this program.
>>>>>>>
>>>>>>> Did the same for 1.2.0 and it worked fine.
>>>>>>>
>>>>>>> Nick
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>
Re: 1.2.1 start-all.sh broken?
Posted by Nicholas Chammas <ni...@gmail.com>.
The tragic thing here is that I was asked to review the patch that
introduced this
<https://github.com/apache/spark/pull/3377#issuecomment-68077315>, and
totally missed it... :(
On Wed Feb 11 2015 at 2:46:35 PM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> lol yeah, I changed the path for the email... turned out to be the issue
> itself.
>
>
> On Wed Feb 11 2015 at 2:43:09 PM Ted Yu <yu...@gmail.com> wrote:
>
>> I see.
>> '/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-)
>>
>> On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Found it:
>>>
>>> https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-
>>> 73058f8e51951ec0b4cb3d48ade91a1fR73
>>>
>>> GRRR BASH WORD SPLITTING
>>>
>>> My path has a space in it...
>>>
>>> Nick
>>>
>>> On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> This is what get:
>>>>
>>>> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
>>>> datanucleus-api-jdo-3.2.6.jar
>>>> datanucleus-core-3.2.10.jar
>>>> datanucleus-rdbms-3.2.9.jar
>>>> spark-1.2.1-yarn-shuffle.jar
>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>
>>>> So that looks correct… Hmm.
>>>>
>>>> Nick
>>>>
>>>>
>>>> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> I downloaded 1.2.1 tar ball for hadoop 2.4
>>>>> I got:
>>>>>
>>>>> ls lib/
>>>>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
>>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
>>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>>
>>>>> FYI
>>>>>
>>>>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
>>>>> nicholas.chammas@gmail.com> wrote:
>>>>>
>>>>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>>>>>> sbin/start-all.sh
>>>>>> on my OS X.
>>>>>>
>>>>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoo
>>>>>> p2.4/lib
>>>>>> You need to build Spark before running this program.
>>>>>>
>>>>>> Did the same for 1.2.0 and it worked fine.
>>>>>>
>>>>>> Nick
>>>>>>
>>>>>>
>>>>>
>>>>>
>>
Re: 1.2.1 start-all.sh broken?
Posted by Nicholas Chammas <ni...@gmail.com>.
lol yeah, I changed the path for the email... turned out to be the issue
itself.
On Wed Feb 11 2015 at 2:43:09 PM Ted Yu <yu...@gmail.com> wrote:
> I see.
> '/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-)
>
> On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Found it:
>>
>>
>> https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-73058f8e51951ec0b4cb3d48ade91a1fR73
>>
>> GRRR BASH WORD SPLITTING
>>
>> My path has a space in it...
>>
>> Nick
>>
>> On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> This is what get:
>>>
>>> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
>>> datanucleus-api-jdo-3.2.6.jar
>>> datanucleus-core-3.2.10.jar
>>> datanucleus-rdbms-3.2.9.jar
>>> spark-1.2.1-yarn-shuffle.jar
>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>
>>> So that looks correct… Hmm.
>>>
>>> Nick
>>>
>>>
>>> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> I downloaded 1.2.1 tar ball for hadoop 2.4
>>>> I got:
>>>>
>>>> ls lib/
>>>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
>>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
>>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>>
>>>> FYI
>>>>
>>>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
>>>> nicholas.chammas@gmail.com> wrote:
>>>>
>>>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>>>>> sbin/start-all.sh
>>>>> on my OS X.
>>>>>
>>>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-
>>>>> hadoop2.4/lib
>>>>> You need to build Spark before running this program.
>>>>>
>>>>> Did the same for 1.2.0 and it worked fine.
>>>>>
>>>>> Nick
>>>>>
>>>>>
>>>>
>>>>
>
Re: 1.2.1 start-all.sh broken?
Posted by Ted Yu <yu...@gmail.com>.
I see.
'/path/to/spark-1.2.1-bin-hadoop2.4' didn't contain space :-)
On Wed, Feb 11, 2015 at 2:41 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> Found it:
>
>
> https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-73058f8e51951ec0b4cb3d48ade91a1fR73
>
> GRRR BASH WORD SPLITTING
>
> My path has a space in it...
>
> Nick
>
> On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> This is what get:
>>
>> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
>> datanucleus-api-jdo-3.2.6.jar
>> datanucleus-core-3.2.10.jar
>> datanucleus-rdbms-3.2.9.jar
>> spark-1.2.1-yarn-shuffle.jar
>> spark-assembly-1.2.1-hadoop2.4.0.jar
>> spark-examples-1.2.1-hadoop2.4.0.jar
>>
>> So that looks correct… Hmm.
>>
>> Nick
>>
>>
>> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
>>
>>> I downloaded 1.2.1 tar ball for hadoop 2.4
>>> I got:
>>>
>>> ls lib/
>>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
>>> spark-assembly-1.2.1-hadoop2.4.0.jar
>>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
>>> spark-examples-1.2.1-hadoop2.4.0.jar
>>>
>>> FYI
>>>
>>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>>>> sbin/start-all.sh
>>>> on my OS X.
>>>>
>>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib
>>>> You need to build Spark before running this program.
>>>>
>>>> Did the same for 1.2.0 and it worked fine.
>>>>
>>>> Nick
>>>>
>>>>
>>>
>>>
Re: 1.2.1 start-all.sh broken?
Posted by Nicholas Chammas <ni...@gmail.com>.
Found it:
https://github.com/apache/spark/compare/v1.2.0...v1.2.1#diff-73058f8e51951ec0b4cb3d48ade91a1fR73
GRRR BASH WORD SPLITTING
My path has a space in it...
Nick
On Wed Feb 11 2015 at 2:37:39 PM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> This is what get:
>
> spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
> datanucleus-api-jdo-3.2.6.jar
> datanucleus-core-3.2.10.jar
> datanucleus-rdbms-3.2.9.jar
> spark-1.2.1-yarn-shuffle.jar
> spark-assembly-1.2.1-hadoop2.4.0.jar
> spark-examples-1.2.1-hadoop2.4.0.jar
>
> So that looks correct… Hmm.
>
> Nick
>
>
> On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
>
>> I downloaded 1.2.1 tar ball for hadoop 2.4
>> I got:
>>
>> ls lib/
>> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
>> spark-assembly-1.2.1-hadoop2.4.0.jar
>> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
>> spark-examples-1.2.1-hadoop2.4.0.jar
>>
>> FYI
>>
>> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>>> sbin/start-all.sh
>>> on my OS X.
>>>
>>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib
>>> You need to build Spark before running this program.
>>>
>>> Did the same for 1.2.0 and it worked fine.
>>>
>>> Nick
>>>
>>>
>>
>>
Re: 1.2.1 start-all.sh broken?
Posted by Nicholas Chammas <ni...@gmail.com>.
This is what get:
spark-1.2.1-bin-hadoop2.4$ ls -1 lib/
datanucleus-api-jdo-3.2.6.jar
datanucleus-core-3.2.10.jar
datanucleus-rdbms-3.2.9.jar
spark-1.2.1-yarn-shuffle.jar
spark-assembly-1.2.1-hadoop2.4.0.jar
spark-examples-1.2.1-hadoop2.4.0.jar
So that looks correct… Hmm.
Nick
On Wed Feb 11 2015 at 2:34:51 PM Ted Yu <yu...@gmail.com> wrote:
> I downloaded 1.2.1 tar ball for hadoop 2.4
> I got:
>
> ls lib/
> datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
> spark-assembly-1.2.1-hadoop2.4.0.jar
> datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
> spark-examples-1.2.1-hadoop2.4.0.jar
>
> FYI
>
> On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran
>> sbin/start-all.sh
>> on my OS X.
>>
>> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib
>> You need to build Spark before running this program.
>>
>> Did the same for 1.2.0 and it worked fine.
>>
>> Nick
>>
>>
>
>
Re: 1.2.1 start-all.sh broken?
Posted by Ted Yu <yu...@gmail.com>.
I downloaded 1.2.1 tar ball for hadoop 2.4
I got:
ls lib/
datanucleus-api-jdo-3.2.6.jar datanucleus-rdbms-3.2.9.jar
spark-assembly-1.2.1-hadoop2.4.0.jar
datanucleus-core-3.2.10.jar spark-1.2.1-yarn-shuffle.jar
spark-examples-1.2.1-hadoop2.4.0.jar
FYI
On Wed, Feb 11, 2015 at 2:27 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I just downloaded 1.2.1 pre-built for Hadoop 2.4+ and ran sbin/start-all.sh
> on my OS X.
>
> Failed to find Spark assembly in /path/to/spark-1.2.1-bin-hadoop2.4/lib
> You need to build Spark before running this program.
>
> Did the same for 1.2.0 and it worked fine.
>
> Nick
>
>