You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by "Periya.Data" <pe...@gmail.com> on 2011/12/04 20:44:41 UTC
Java and Hadoop on Ubuntu 11.10 AMI
Hi,
I am trying Whirr to spin-up EC2 clusters. Here are the specs:
- Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
- EC2 Whirr config :
-
- whirr.hadoop.install-function=install_cdh_hadoop
- whirr.hadoop.configure-function=configure_cdh_hadoop
- whirr.hadoop.version=0.20.2
- whirr.hadoop.tarball.url=
http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
- whirr.hardware-id=t1.micro
# Ubuntu 11.10 Oneiric EBS boot from alestic
- whirr.image-id=us-east-1/ami-a7f539ce
- whirr.location-id=us-east-1
*Problem:* The cluster seems to be up and running. But, cannot see Java and
Hadoop on the instance.
*Question:* Is this a problem or should I clearly mention in my Whirr
config/properties file what version of JDK to be installed and its download
URL?
*Some logs:* (I ssh-ed into one of the nodes and tried to see the logs)
sri@domU-12-31-39-02-21-20:/tmp$ ls -al
total 40
drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
-rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
-rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
drwxr-xr-x 2 root root 4096 2011-12-04 05:46 jclouds-script-1322977554131
drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
-rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
-rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
sri@domU-12-31-39-02-21-20:/tmp$
sri@domU-12-31-39-02-21-20:/tmp$
sri@domU-12-31-39-02-21-20:/tmp$
sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
total 12
-rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
-rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
E: Package 'sun-java6-jdk' has no installation candidate
/tmp/setup-sri.sh: line 161: java: command not found
sri@domU-12-31-39-02-21-20:/tmp/logs$
----------
Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
i386 Packages [57.2 kB]
Get:37 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
i386 Packages [4396 B]
Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
TranslationIndex [73 B]
Get:39 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
TranslationIndex [72 B
]
Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
TranslationIndex [73 B]
Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main Translation-en
[701 kB]
Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
Translation-en
Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
Translation-en [3165 kB]
Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
Translation-en [99.2 kB]
Get:44 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
Translation-en [2419 B
]
Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
Translation-en [35.4 kB]
Fetched 16.2 MB in 12s (1290 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Package sun-java6-jdk is not available, but is referred to by another
package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
sri@domU-12-31-39-02-21-20:/tmp/logs$
Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
i386 Packages [57.2 kB]
Get:37 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
i386 Packages [4396 B]
Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
TranslationIndex [73 B]
Get:39 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
TranslationIndex [72 B]
Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
TranslationIndex [73 B]
Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main Translation-en
[701 kB]
Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
Translation-en
Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
Translation-en [3165 kB]
Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
Translation-en [99.2 kB]
Get:44 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
Translation-en [2419 B]
Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
Translation-en [35.4 kB]
Fetched 16.2 MB in 12s (1290 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Package sun-java6-jdk is not available, but is referred to by another
package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
sri@domU-12-31-39-02-21-20:/tmp/logs$
*[On a different cluster spun-up at a later time]*
sri@domU-12-31-39-10-6D-38:~$ java
The program 'java' can be found in the following packages:
* gcj-4.4-jre-headless
* gcj-4.6-jre-headless
* openjdk-6-jre-headless
* gcj-4.5-jre-headless
* openjdk-7-jre-headless
Ask your administrator to install one of them
sri@domU-12-31-39-10-6D-38:~$ java -version
The program 'java' can be found in the following packages:
* gcj-4.4-jre-headless
* gcj-4.6-jre-headless
* openjdk-6-jre-headless
* gcj-4.5-jre-headless
* openjdk-7-jre-headless
Ask your administrator to install one of them
sri@domU-12-31-39-10-6D-38:~$
sri@domU-12-31-39-10-6D-38:~$ hadoop version
hadoop: command not found
sri@domU-12-31-39-10-6D-38:~$
===========================================
Finally, a quick Googling pointed to a JIRA issue:
https://issues.apache.org/jira/browse/WHIRR-435
I am seeing the same stuff. Any updates? If this is a real issues, I do not
have any problems in using Ubuntu 11.04 AMI for now.
Your help is very much appreciated.
Thanks,
PD.
Re: Java and Hadoop on Ubuntu 11.10 AMI
Posted by Andrei Savu <sa...@gmail.com>.
I am happy to hear that!! Ping the email list any time you have questions.
On Sun, Dec 4, 2011 at 11:14 PM, Periya.Data <pe...@gmail.com> wrote:
> Hi Andrei,
> Many thanks again. Will see those links and try to create a cluster
> with the tools I need.
>
> Will be happy to contribute in any way I can. Will see the Wiki on how to
> contribute and will proceed from there.
>
> /PD.
>
>
> On Sun, Dec 4, 2011 at 1:10 PM, Andrei Savu <sa...@gmail.com> wrote:
>
>> The best way would be to define a new service for each one with new
>> install / configure scripts.
>>
>> See this presentation about creating a new Whirr service:
>> http://www.oscon.com/oscon2011/public/schedule/detail/19214
>>
>> For Flume the following repo contains some working code:
>> https://github.com/tomwhite/whirr-service-example
>>
>> For Hive we are tracking progress here:
>> https://issues.apache.org/jira/browse/WHIRR-343
>>
>> Currently there is no open issue for adding Sqoop as a service.
>>
>> I strongly recommend you to submit patches / pull requests as early as
>> possible and I will give as
>> much assistance as needed. See the following wiki page to learn more
>> about how to contribute:
>> https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute
>>
>> On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <pe...@gmail.com>wrote:
>>
>>> Thanks much. Will follow your suggestions. A quick question:
>>>
>>> If I have to install Hive, Sqoop and Flume on the my cluster nodes, what
>>> is the suggested way?
>>> --> Log into each cluster node and install them manually?
>>> --> Maybe the I have to hack the install script
>>> (/whirr/services/cdh/src/main/resources/functions) or create a new one for
>>> each of the tools?
>>>
>>> Thanks,
>>> PD.
>>>
>>>
>>> On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <sa...@gmail.com>wrote:
>>>
>>>> Here are a few advices:
>>>>
>>>> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS
>>>> version we are testing against before making a release
>>>> * whirr.hadoop.tarball.url is irrelevant if you specify
>>>> whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
>>>> * install_cdh_hadoop will install by default the latest release.
>>>> We've implemented a way of selecting the version in 0.7.0:
>>>> https://issues.apache.org/jira/browse/WHIRR-415
>>>> * use something bigger than t1.micro - this is just too small to run
>>>> Hadoop
>>>>
>>>> Let me know if you are still having problems.
>>>>
>>>> -- Andrei Savu / andreisavu.ro
>>>>
>>>>
>>>> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <pe...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>> I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>>>>>
>>>>>
>>>>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
>>>>> - EC2 Whirr config :
>>>>> -
>>>>> - whirr.hadoop.install-function=install_cdh_hadoop
>>>>> - whirr.hadoop.configure-function=configure_cdh_hadoop
>>>>> - whirr.hadoop.version=0.20.2
>>>>> - whirr.hadoop.tarball.url=
>>>>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
>>>>> - whirr.hardware-id=t1.micro
>>>>> # Ubuntu 11.10 Oneiric EBS boot from alestic
>>>>> - whirr.image-id=us-east-1/ami-a7f539ce
>>>>> - whirr.location-id=us-east-1
>>>>>
>>>>>
>>>>> *Problem:* The cluster seems to be up and running. But, cannot see
>>>>> Java and Hadoop on the instance.
>>>>>
>>>>> *Question:* Is this a problem or should I clearly mention in my Whirr
>>>>> config/properties file what version of JDK to be installed and its download
>>>>> URL?
>>>>>
>>>>>
>>>>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the
>>>>> logs)
>>>>>
>>>>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
>>>>> total 40
>>>>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
>>>>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
>>>>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
>>>>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
>>>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46
>>>>> jclouds-script-1322977554131
>>>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
>>>>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
>>>>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
>>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
>>>>> total 12
>>>>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
>>>>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
>>>>> E: Package 'sun-java6-jdk' has no installation candidate
>>>>> /tmp/setup-sri.sh: line 161: java: command not found
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>>
>>>>> ----------
>>>>>
>>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe i386 Packages [57.2 kB]
>>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> TranslationIndex [73 B]
>>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B
>>>>> ]
>>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe TranslationIndex [73 B]
>>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>>>> Translation-en [701 kB]
>>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>>>> Translation-en
>>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>>>> Translation-en [3165 kB]
>>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> Translation-en [99.2 kB]
>>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B
>>>>> ]
>>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe Translation-en [35.4 kB]
>>>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>>>> Reading package lists...
>>>>> Reading package lists...
>>>>> Building dependency tree...
>>>>> Reading state information...
>>>>> Package sun-java6-jdk is not available, but is referred to by another
>>>>> package.
>>>>> This may mean that the package is missing, has been obsoleted, or
>>>>> is only available from another source
>>>>>
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe i386 Packages [57.2 kB]
>>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> TranslationIndex [73 B]
>>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B]
>>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe TranslationIndex [73 B]
>>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>>>> Translation-en [701 kB]
>>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>>>> Translation-en
>>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>>>> Translation-en [3165 kB]
>>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>>> Translation-en [99.2 kB]
>>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B]
>>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe Translation-en [35.4 kB]
>>>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>>>> Reading package lists...
>>>>> Reading package lists...
>>>>> Building dependency tree...
>>>>> Reading state information...
>>>>> Package sun-java6-jdk is not available, but is referred to by another
>>>>> package.
>>>>> This may mean that the package is missing, has been obsoleted, or
>>>>> is only available from another source
>>>>>
>>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>>
>>>>> *[On a different cluster spun-up at a later time]*
>>>>>
>>>>> sri@domU-12-31-39-10-6D-38:~$ java
>>>>> The program 'java' can be found in the following packages:
>>>>> * gcj-4.4-jre-headless
>>>>> * gcj-4.6-jre-headless
>>>>> * openjdk-6-jre-headless
>>>>> * gcj-4.5-jre-headless
>>>>> * openjdk-7-jre-headless
>>>>> Ask your administrator to install one of them
>>>>> sri@domU-12-31-39-10-6D-38:~$ java -version
>>>>> The program 'java' can be found in the following packages:
>>>>> * gcj-4.4-jre-headless
>>>>> * gcj-4.6-jre-headless
>>>>> * openjdk-6-jre-headless
>>>>> * gcj-4.5-jre-headless
>>>>> * openjdk-7-jre-headless
>>>>> Ask your administrator to install one of them
>>>>> sri@domU-12-31-39-10-6D-38:~$
>>>>>
>>>>> sri@domU-12-31-39-10-6D-38:~$ hadoop version
>>>>> hadoop: command not found
>>>>> sri@domU-12-31-39-10-6D-38:~$
>>>>>
>>>>> ===========================================
>>>>>
>>>>> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>>>>>
>>>>>
>>>>> I am seeing the same stuff. Any updates? If this is a real issues, I
>>>>> do not have any problems in using Ubuntu 11.04 AMI for now.
>>>>>
>>>>>
>>>>> Your help is very much appreciated.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> PD.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Java and Hadoop on Ubuntu 11.10 AMI
Posted by "Periya.Data" <pe...@gmail.com>.
Hi Andrei,
Many thanks again. Will see those links and try to create a cluster with
the tools I need.
Will be happy to contribute in any way I can. Will see the Wiki on how to
contribute and will proceed from there.
/PD.
On Sun, Dec 4, 2011 at 1:10 PM, Andrei Savu <sa...@gmail.com> wrote:
> The best way would be to define a new service for each one with new
> install / configure scripts.
>
> See this presentation about creating a new Whirr service:
> http://www.oscon.com/oscon2011/public/schedule/detail/19214
>
> For Flume the following repo contains some working code:
> https://github.com/tomwhite/whirr-service-example
>
> For Hive we are tracking progress here:
> https://issues.apache.org/jira/browse/WHIRR-343
>
> Currently there is no open issue for adding Sqoop as a service.
>
> I strongly recommend you to submit patches / pull requests as early as
> possible and I will give as
> much assistance as needed. See the following wiki page to learn more about
> how to contribute:
> https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute
>
> On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <pe...@gmail.com>wrote:
>
>> Thanks much. Will follow your suggestions. A quick question:
>>
>> If I have to install Hive, Sqoop and Flume on the my cluster nodes, what
>> is the suggested way?
>> --> Log into each cluster node and install them manually?
>> --> Maybe the I have to hack the install script
>> (/whirr/services/cdh/src/main/resources/functions) or create a new one for
>> each of the tools?
>>
>> Thanks,
>> PD.
>>
>>
>> On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <sa...@gmail.com>wrote:
>>
>>> Here are a few advices:
>>>
>>> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS
>>> version we are testing against before making a release
>>> * whirr.hadoop.tarball.url is irrelevant if you specify
>>> whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
>>> * install_cdh_hadoop will install by default the latest release.
>>> We've implemented a way of selecting the version in 0.7.0:
>>> https://issues.apache.org/jira/browse/WHIRR-415
>>> * use something bigger than t1.micro - this is just too small to run
>>> Hadoop
>>>
>>> Let me know if you are still having problems.
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>>
>>> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <pe...@gmail.com>wrote:
>>>
>>>> Hi,
>>>> I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>>>>
>>>>
>>>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
>>>> - EC2 Whirr config :
>>>> -
>>>> - whirr.hadoop.install-function=install_cdh_hadoop
>>>> - whirr.hadoop.configure-function=configure_cdh_hadoop
>>>> - whirr.hadoop.version=0.20.2
>>>> - whirr.hadoop.tarball.url=
>>>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
>>>> - whirr.hardware-id=t1.micro
>>>> # Ubuntu 11.10 Oneiric EBS boot from alestic
>>>> - whirr.image-id=us-east-1/ami-a7f539ce
>>>> - whirr.location-id=us-east-1
>>>>
>>>>
>>>> *Problem:* The cluster seems to be up and running. But, cannot see
>>>> Java and Hadoop on the instance.
>>>>
>>>> *Question:* Is this a problem or should I clearly mention in my Whirr
>>>> config/properties file what version of JDK to be installed and its download
>>>> URL?
>>>>
>>>>
>>>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs)
>>>>
>>>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
>>>> total 40
>>>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
>>>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
>>>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
>>>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
>>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46
>>>> jclouds-script-1322977554131
>>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
>>>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
>>>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>> sri@domU-12-31-39-02-21-20:/tmp$
>>>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
>>>> total 12
>>>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
>>>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
>>>> E: Package 'sun-java6-jdk' has no installation candidate
>>>> /tmp/setup-sri.sh: line 161: java: command not found
>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>
>>>> ----------
>>>>
>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe i386 Packages [57.2 kB]
>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>> TranslationIndex [73 B]
>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B
>>>> ]
>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe TranslationIndex [73 B]
>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>>> Translation-en [701 kB]
>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>>> Translation-en
>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>>> Translation-en [3165 kB]
>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>> Translation-en [99.2 kB]
>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B
>>>> ]
>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe Translation-en [35.4 kB]
>>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>>> Reading package lists...
>>>> Reading package lists...
>>>> Building dependency tree...
>>>> Reading state information...
>>>> Package sun-java6-jdk is not available, but is referred to by another
>>>> package.
>>>> This may mean that the package is missing, has been obsoleted, or
>>>> is only available from another source
>>>>
>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe i386 Packages [57.2 kB]
>>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>> TranslationIndex [73 B]
>>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B]
>>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe TranslationIndex [73 B]
>>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>>> Translation-en [701 kB]
>>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>>> Translation-en
>>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>>> Translation-en [3165 kB]
>>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>>> Translation-en [99.2 kB]
>>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B]
>>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe Translation-en [35.4 kB]
>>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>>> Reading package lists...
>>>> Reading package lists...
>>>> Building dependency tree...
>>>> Reading state information...
>>>> Package sun-java6-jdk is not available, but is referred to by another
>>>> package.
>>>> This may mean that the package is missing, has been obsoleted, or
>>>> is only available from another source
>>>>
>>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>>
>>>> *[On a different cluster spun-up at a later time]*
>>>>
>>>> sri@domU-12-31-39-10-6D-38:~$ java
>>>> The program 'java' can be found in the following packages:
>>>> * gcj-4.4-jre-headless
>>>> * gcj-4.6-jre-headless
>>>> * openjdk-6-jre-headless
>>>> * gcj-4.5-jre-headless
>>>> * openjdk-7-jre-headless
>>>> Ask your administrator to install one of them
>>>> sri@domU-12-31-39-10-6D-38:~$ java -version
>>>> The program 'java' can be found in the following packages:
>>>> * gcj-4.4-jre-headless
>>>> * gcj-4.6-jre-headless
>>>> * openjdk-6-jre-headless
>>>> * gcj-4.5-jre-headless
>>>> * openjdk-7-jre-headless
>>>> Ask your administrator to install one of them
>>>> sri@domU-12-31-39-10-6D-38:~$
>>>>
>>>> sri@domU-12-31-39-10-6D-38:~$ hadoop version
>>>> hadoop: command not found
>>>> sri@domU-12-31-39-10-6D-38:~$
>>>>
>>>> ===========================================
>>>>
>>>> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>>>>
>>>>
>>>> I am seeing the same stuff. Any updates? If this is a real issues, I do
>>>> not have any problems in using Ubuntu 11.04 AMI for now.
>>>>
>>>>
>>>> Your help is very much appreciated.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> PD.
>>>>
>>>
>>>
>>
>
Re: Java and Hadoop on Ubuntu 11.10 AMI
Posted by Andrei Savu <sa...@gmail.com>.
The best way would be to define a new service for each one with new install
/ configure scripts.
See this presentation about creating a new Whirr service:
http://www.oscon.com/oscon2011/public/schedule/detail/19214
For Flume the following repo contains some working code:
https://github.com/tomwhite/whirr-service-example
For Hive we are tracking progress here:
https://issues.apache.org/jira/browse/WHIRR-343
Currently there is no open issue for adding Sqoop as a service.
I strongly recommend you to submit patches / pull requests as early as
possible and I will give as
much assistance as needed. See the following wiki page to learn more about
how to contribute:
https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute
On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <pe...@gmail.com> wrote:
> Thanks much. Will follow your suggestions. A quick question:
>
> If I have to install Hive, Sqoop and Flume on the my cluster nodes, what
> is the suggested way?
> --> Log into each cluster node and install them manually?
> --> Maybe the I have to hack the install script
> (/whirr/services/cdh/src/main/resources/functions) or create a new one for
> each of the tools?
>
> Thanks,
> PD.
>
>
> On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <sa...@gmail.com>wrote:
>
>> Here are a few advices:
>>
>> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS
>> version we are testing against before making a release
>> * whirr.hadoop.tarball.url is irrelevant if you specify
>> whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
>> * install_cdh_hadoop will install by default the latest release.
>> We've implemented a way of selecting the version in 0.7.0:
>> https://issues.apache.org/jira/browse/WHIRR-415
>> * use something bigger than t1.micro - this is just too small to run
>> Hadoop
>>
>> Let me know if you are still having problems.
>>
>> -- Andrei Savu / andreisavu.ro
>>
>>
>> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <pe...@gmail.com>wrote:
>>
>>> Hi,
>>> I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>>>
>>>
>>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
>>> - EC2 Whirr config :
>>> -
>>> - whirr.hadoop.install-function=install_cdh_hadoop
>>> - whirr.hadoop.configure-function=configure_cdh_hadoop
>>> - whirr.hadoop.version=0.20.2
>>> - whirr.hadoop.tarball.url=
>>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
>>> - whirr.hardware-id=t1.micro
>>> # Ubuntu 11.10 Oneiric EBS boot from alestic
>>> - whirr.image-id=us-east-1/ami-a7f539ce
>>> - whirr.location-id=us-east-1
>>>
>>>
>>> *Problem:* The cluster seems to be up and running. But, cannot see Java
>>> and Hadoop on the instance.
>>>
>>> *Question:* Is this a problem or should I clearly mention in my Whirr
>>> config/properties file what version of JDK to be installed and its download
>>> URL?
>>>
>>>
>>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs)
>>>
>>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
>>> total 40
>>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
>>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
>>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
>>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46
>>> jclouds-script-1322977554131
>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
>>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
>>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
>>> sri@domU-12-31-39-02-21-20:/tmp$
>>> sri@domU-12-31-39-02-21-20:/tmp$
>>> sri@domU-12-31-39-02-21-20:/tmp$
>>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
>>> total 12
>>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
>>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
>>> E: Package 'sun-java6-jdk' has no installation candidate
>>> /tmp/setup-sri.sh: line 161: java: command not found
>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>
>>> ----------
>>>
>>> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>>> i386 Packages [57.2 kB]
>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>> TranslationIndex [73 B]
>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B
>>> ]
>>> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>>> TranslationIndex [73 B]
>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>> Translation-en [701 kB]
>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>> Translation-en
>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>> Translation-en [3165 kB]
>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>> Translation-en [99.2 kB]
>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B
>>> ]
>>> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>>> Translation-en [35.4 kB]
>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>> Reading package lists...
>>> Reading package lists...
>>> Building dependency tree...
>>> Reading state information...
>>> Package sun-java6-jdk is not available, but is referred to by another
>>> package.
>>> This may mean that the package is missing, has been obsoleted, or
>>> is only available from another source
>>>
>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>>> i386 Packages [57.2 kB]
>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>> TranslationIndex [73 B]
>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B]
>>> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>>> TranslationIndex [73 B]
>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>>> Translation-en [701 kB]
>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>>> Translation-en
>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>>> Translation-en [3165 kB]
>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>>> Translation-en [99.2 kB]
>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B]
>>> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>>> Translation-en [35.4 kB]
>>> Fetched 16.2 MB in 12s (1290 kB/s)
>>> Reading package lists...
>>> Reading package lists...
>>> Building dependency tree...
>>> Reading state information...
>>> Package sun-java6-jdk is not available, but is referred to by another
>>> package.
>>> This may mean that the package is missing, has been obsoleted, or
>>> is only available from another source
>>>
>>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>>
>>> *[On a different cluster spun-up at a later time]*
>>>
>>> sri@domU-12-31-39-10-6D-38:~$ java
>>> The program 'java' can be found in the following packages:
>>> * gcj-4.4-jre-headless
>>> * gcj-4.6-jre-headless
>>> * openjdk-6-jre-headless
>>> * gcj-4.5-jre-headless
>>> * openjdk-7-jre-headless
>>> Ask your administrator to install one of them
>>> sri@domU-12-31-39-10-6D-38:~$ java -version
>>> The program 'java' can be found in the following packages:
>>> * gcj-4.4-jre-headless
>>> * gcj-4.6-jre-headless
>>> * openjdk-6-jre-headless
>>> * gcj-4.5-jre-headless
>>> * openjdk-7-jre-headless
>>> Ask your administrator to install one of them
>>> sri@domU-12-31-39-10-6D-38:~$
>>>
>>> sri@domU-12-31-39-10-6D-38:~$ hadoop version
>>> hadoop: command not found
>>> sri@domU-12-31-39-10-6D-38:~$
>>>
>>> ===========================================
>>>
>>> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>>>
>>>
>>> I am seeing the same stuff. Any updates? If this is a real issues, I do
>>> not have any problems in using Ubuntu 11.04 AMI for now.
>>>
>>>
>>> Your help is very much appreciated.
>>>
>>>
>>> Thanks,
>>>
>>> PD.
>>>
>>
>>
>
Re: Java and Hadoop on Ubuntu 11.10 AMI
Posted by "Periya.Data" <pe...@gmail.com>.
Thanks much. Will follow your suggestions. A quick question:
If I have to install Hive, Sqoop and Flume on the my cluster nodes, what is
the suggested way?
--> Log into each cluster node and install them manually?
--> Maybe the I have to hack the install script
(/whirr/services/cdh/src/main/resources/functions) or create a new one for
each of the tools?
Thanks,
PD.
On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <sa...@gmail.com> wrote:
> Here are a few advices:
>
> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS version
> we are testing against before making a release
> * whirr.hadoop.tarball.url is irrelevant if you specify
> whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
> * install_cdh_hadoop will install by default the latest release.
> We've implemented a way of selecting the version in 0.7.0:
> https://issues.apache.org/jira/browse/WHIRR-415
> * use something bigger than t1.micro - this is just too small to run
> Hadoop
>
> Let me know if you are still having problems.
>
> -- Andrei Savu / andreisavu.ro
>
>
> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <pe...@gmail.com> wrote:
>
>> Hi,
>> I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>>
>>
>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
>> - EC2 Whirr config :
>> -
>> - whirr.hadoop.install-function=install_cdh_hadoop
>> - whirr.hadoop.configure-function=configure_cdh_hadoop
>> - whirr.hadoop.version=0.20.2
>> - whirr.hadoop.tarball.url=
>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
>> - whirr.hardware-id=t1.micro
>> # Ubuntu 11.10 Oneiric EBS boot from alestic
>> - whirr.image-id=us-east-1/ami-a7f539ce
>> - whirr.location-id=us-east-1
>>
>>
>> *Problem:* The cluster seems to be up and running. But, cannot see Java
>> and Hadoop on the instance.
>>
>> *Question:* Is this a problem or should I clearly mention in my Whirr
>> config/properties file what version of JDK to be installed and its download
>> URL?
>>
>>
>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs)
>>
>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
>> total 40
>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46
>> jclouds-script-1322977554131
>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
>> sri@domU-12-31-39-02-21-20:/tmp$
>> sri@domU-12-31-39-02-21-20:/tmp$
>> sri@domU-12-31-39-02-21-20:/tmp$
>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
>> total 12
>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
>> E: Package 'sun-java6-jdk' has no installation candidate
>> /tmp/setup-sri.sh: line 161: java: command not found
>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>
>> ----------
>>
>> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>> i386 Packages [57.2 kB]
>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>> TranslationIndex [73 B]
>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B
>> ]
>> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>> TranslationIndex [73 B]
>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>> Translation-en [701 kB]
>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>> Translation-en
>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>> Translation-en [3165 kB]
>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>> Translation-en [99.2 kB]
>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B
>> ]
>> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>> Translation-en [35.4 kB]
>> Fetched 16.2 MB in 12s (1290 kB/s)
>> Reading package lists...
>> Reading package lists...
>> Building dependency tree...
>> Reading state information...
>> Package sun-java6-jdk is not available, but is referred to by another
>> package.
>> This may mean that the package is missing, has been obsoleted, or
>> is only available from another source
>>
>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>> i386 Packages [57.2 kB]
>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse i386 Packages [4396 B]
>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>> TranslationIndex [73 B]
>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse TranslationIndex [72 B]
>> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>> TranslationIndex [73 B]
>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
>> Translation-en [701 kB]
>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
>> Translation-en
>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
>> Translation-en [3165 kB]
>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
>> Translation-en [99.2 kB]
>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse Translation-en [2419 B]
>> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
>> Translation-en [35.4 kB]
>> Fetched 16.2 MB in 12s (1290 kB/s)
>> Reading package lists...
>> Reading package lists...
>> Building dependency tree...
>> Reading state information...
>> Package sun-java6-jdk is not available, but is referred to by another
>> package.
>> This may mean that the package is missing, has been obsoleted, or
>> is only available from another source
>>
>> sri@domU-12-31-39-02-21-20:/tmp/logs$
>>
>> *[On a different cluster spun-up at a later time]*
>>
>> sri@domU-12-31-39-10-6D-38:~$ java
>> The program 'java' can be found in the following packages:
>> * gcj-4.4-jre-headless
>> * gcj-4.6-jre-headless
>> * openjdk-6-jre-headless
>> * gcj-4.5-jre-headless
>> * openjdk-7-jre-headless
>> Ask your administrator to install one of them
>> sri@domU-12-31-39-10-6D-38:~$ java -version
>> The program 'java' can be found in the following packages:
>> * gcj-4.4-jre-headless
>> * gcj-4.6-jre-headless
>> * openjdk-6-jre-headless
>> * gcj-4.5-jre-headless
>> * openjdk-7-jre-headless
>> Ask your administrator to install one of them
>> sri@domU-12-31-39-10-6D-38:~$
>>
>> sri@domU-12-31-39-10-6D-38:~$ hadoop version
>> hadoop: command not found
>> sri@domU-12-31-39-10-6D-38:~$
>>
>> ===========================================
>>
>> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>>
>>
>> I am seeing the same stuff. Any updates? If this is a real issues, I do
>> not have any problems in using Ubuntu 11.04 AMI for now.
>>
>>
>> Your help is very much appreciated.
>>
>>
>> Thanks,
>>
>> PD.
>>
>
>
Re: Java and Hadoop on Ubuntu 11.10 AMI
Posted by Andrei Savu <sa...@gmail.com>.
Here are a few advices:
* use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS version
we are testing against before making a release
* whirr.hadoop.tarball.url is irrelevant if you specify
whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
* install_cdh_hadoop will install by default the latest release.
We've implemented a way of selecting the version in 0.7.0:
https://issues.apache.org/jira/browse/WHIRR-415
* use something bigger than t1.micro - this is just too small to run Hadoop
Let me know if you are still having problems.
-- Andrei Savu / andreisavu.ro
On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <pe...@gmail.com> wrote:
> Hi,
> I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>
>
> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
> - EC2 Whirr config :
> -
> - whirr.hadoop.install-function=install_cdh_hadoop
> - whirr.hadoop.configure-function=configure_cdh_hadoop
> - whirr.hadoop.version=0.20.2
> - whirr.hadoop.tarball.url=
> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
> - whirr.hardware-id=t1.micro
> # Ubuntu 11.10 Oneiric EBS boot from alestic
> - whirr.image-id=us-east-1/ami-a7f539ce
> - whirr.location-id=us-east-1
>
>
> *Problem:* The cluster seems to be up and running. But, cannot see Java
> and Hadoop on the instance.
>
> *Question:* Is this a problem or should I clearly mention in my Whirr
> config/properties file what version of JDK to be installed and its download
> URL?
>
>
> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs)
>
> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
> total 40
> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
> drwxr-xr-x 2 root root 4096 2011-12-04 05:46 jclouds-script-1322977554131
> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
> sri@domU-12-31-39-02-21-20:/tmp$
> sri@domU-12-31-39-02-21-20:/tmp$
> sri@domU-12-31-39-02-21-20:/tmp$
> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
> total 12
> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
> E: Package 'sun-java6-jdk' has no installation candidate
> /tmp/setup-sri.sh: line 161: java: command not found
> sri@domU-12-31-39-02-21-20:/tmp/logs$
>
> ----------
>
> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> i386 Packages [57.2 kB]
> Get:37 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> i386 Packages [4396 B]
> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> TranslationIndex [73 B]
> Get:39 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> TranslationIndex [72 B
> ]
> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> TranslationIndex [73 B]
> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
> Translation-en [701 kB]
> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
> Translation-en
> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
> Translation-en [3165 kB]
> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> Translation-en [99.2 kB]
> Get:44 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> Translation-en [2419 B
> ]
> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> Translation-en [35.4 kB]
> Fetched 16.2 MB in 12s (1290 kB/s)
> Reading package lists...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Package sun-java6-jdk is not available, but is referred to by another
> package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
>
> sri@domU-12-31-39-02-21-20:/tmp/logs$
> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> i386 Packages [57.2 kB]
> Get:37 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> i386 Packages [4396 B]
> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> TranslationIndex [73 B]
> Get:39 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> TranslationIndex [72 B]
> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> TranslationIndex [73 B]
> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
> Translation-en [701 kB]
> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
> Translation-en
> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
> Translation-en [3165 kB]
> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> Translation-en [99.2 kB]
> Get:44 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> Translation-en [2419 B]
> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> Translation-en [35.4 kB]
> Fetched 16.2 MB in 12s (1290 kB/s)
> Reading package lists...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Package sun-java6-jdk is not available, but is referred to by another
> package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
>
> sri@domU-12-31-39-02-21-20:/tmp/logs$
>
> *[On a different cluster spun-up at a later time]*
>
> sri@domU-12-31-39-10-6D-38:~$ java
> The program 'java' can be found in the following packages:
> * gcj-4.4-jre-headless
> * gcj-4.6-jre-headless
> * openjdk-6-jre-headless
> * gcj-4.5-jre-headless
> * openjdk-7-jre-headless
> Ask your administrator to install one of them
> sri@domU-12-31-39-10-6D-38:~$ java -version
> The program 'java' can be found in the following packages:
> * gcj-4.4-jre-headless
> * gcj-4.6-jre-headless
> * openjdk-6-jre-headless
> * gcj-4.5-jre-headless
> * openjdk-7-jre-headless
> Ask your administrator to install one of them
> sri@domU-12-31-39-10-6D-38:~$
>
> sri@domU-12-31-39-10-6D-38:~$ hadoop version
> hadoop: command not found
> sri@domU-12-31-39-10-6D-38:~$
>
> ===========================================
>
> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>
>
> I am seeing the same stuff. Any updates? If this is a real issues, I do
> not have any problems in using Ubuntu 11.04 AMI for now.
>
>
> Your help is very much appreciated.
>
>
> Thanks,
>
> PD.
>
Re: Java and Hadoop on Ubuntu 11.10 AMI
Posted by Andrei Savu <sa...@gmail.com>.
Here are a few advices:
* use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS version
we are testing against before making a release
* whirr.hadoop.tarball.url is irrelevant if you specify
whirr.hadoop.install-function. Same is true for whirr.hadoop.version.
* install_cdh_hadoop will install by default the latest release.
We've implemented a way of selecting the version in 0.7.0:
https://issues.apache.org/jira/browse/WHIRR-415
* use something bigger than t1.micro - this is just too small to run Hadoop
Let me know if you are still having problems.
-- Andrei Savu / andreisavu.ro
On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <pe...@gmail.com> wrote:
> Hi,
> I am trying Whirr to spin-up EC2 clusters. Here are the specs:
>
>
> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2
> - EC2 Whirr config :
> -
> - whirr.hadoop.install-function=install_cdh_hadoop
> - whirr.hadoop.configure-function=configure_cdh_hadoop
> - whirr.hadoop.version=0.20.2
> - whirr.hadoop.tarball.url=
> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz
> - whirr.hardware-id=t1.micro
> # Ubuntu 11.10 Oneiric EBS boot from alestic
> - whirr.image-id=us-east-1/ami-a7f539ce
> - whirr.location-id=us-east-1
>
>
> *Problem:* The cluster seems to be up and running. But, cannot see Java
> and Hadoop on the instance.
>
> *Question:* Is this a problem or should I clearly mention in my Whirr
> config/properties file what version of JDK to be installed and its download
> URL?
>
>
> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs)
>
> sri@domU-12-31-39-02-21-20:/tmp$ ls -al
> total 40
> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 .
> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 ..
> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml
> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml
> drwxr-xr-x 2 root root 4096 2011-12-04 05:46 jclouds-script-1322977554131
> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs
> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml
> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh
> sri@domU-12-31-39-02-21-20:/tmp$
> sri@domU-12-31-39-02-21-20:/tmp$
> sri@domU-12-31-39-02-21-20:/tmp$
> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/
> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l
> total 12
> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log
> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log
> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log
> E: Package 'sun-java6-jdk' has no installation candidate
> /tmp/setup-sri.sh: line 161: java: command not found
> sri@domU-12-31-39-02-21-20:/tmp/logs$
>
> ----------
>
> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> i386 Packages [57.2 kB]
> Get:37 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> i386 Packages [4396 B]
> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> TranslationIndex [73 B]
> Get:39 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> TranslationIndex [72 B
> ]
> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> TranslationIndex [73 B]
> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
> Translation-en [701 kB]
> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
> Translation-en
> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
> Translation-en [3165 kB]
> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> Translation-en [99.2 kB]
> Get:44 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> Translation-en [2419 B
> ]
> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> Translation-en [35.4 kB]
> Fetched 16.2 MB in 12s (1290 kB/s)
> Reading package lists...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Package sun-java6-jdk is not available, but is referred to by another
> package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
>
> sri@domU-12-31-39-02-21-20:/tmp/logs$
> Get:36 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> i386 Packages [57.2 kB]
> Get:37 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> i386 Packages [4396 B]
> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> TranslationIndex [73 B]
> Get:39 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> TranslationIndex [72 B]
> Get:40 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> TranslationIndex [73 B]
> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main
> Translation-en [701 kB]
> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse
> Translation-en
> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe
> Translation-en [3165 kB]
> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main
> Translation-en [99.2 kB]
> Get:44 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/multiverse
> Translation-en [2419 B]
> Get:45 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/universe
> Translation-en [35.4 kB]
> Fetched 16.2 MB in 12s (1290 kB/s)
> Reading package lists...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Package sun-java6-jdk is not available, but is referred to by another
> package.
> This may mean that the package is missing, has been obsoleted, or
> is only available from another source
>
> sri@domU-12-31-39-02-21-20:/tmp/logs$
>
> *[On a different cluster spun-up at a later time]*
>
> sri@domU-12-31-39-10-6D-38:~$ java
> The program 'java' can be found in the following packages:
> * gcj-4.4-jre-headless
> * gcj-4.6-jre-headless
> * openjdk-6-jre-headless
> * gcj-4.5-jre-headless
> * openjdk-7-jre-headless
> Ask your administrator to install one of them
> sri@domU-12-31-39-10-6D-38:~$ java -version
> The program 'java' can be found in the following packages:
> * gcj-4.4-jre-headless
> * gcj-4.6-jre-headless
> * openjdk-6-jre-headless
> * gcj-4.5-jre-headless
> * openjdk-7-jre-headless
> Ask your administrator to install one of them
> sri@domU-12-31-39-10-6D-38:~$
>
> sri@domU-12-31-39-10-6D-38:~$ hadoop version
> hadoop: command not found
> sri@domU-12-31-39-10-6D-38:~$
>
> ===========================================
>
> Finally, a quick Googling pointed to a JIRA issue: https://issues.apache.org/jira/browse/WHIRR-435
>
>
> I am seeing the same stuff. Any updates? If this is a real issues, I do
> not have any problems in using Ubuntu 11.04 AMI for now.
>
>
> Your help is very much appreciated.
>
>
> Thanks,
>
> PD.
>