You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Demai Ni <ni...@gmail.com> on 2015/07/15 00:18:00 UTC
Error: CSV::IllegalFormatError
Dear BigTop friends,
I am following instruction here:
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Bigtop+0.8.0+Hadoop+on+CentOS+6+with+Puppet
to setup a 2-node clusters. But failed at very early stage: puppet apply.
Well, quite possible that I made a small mistake somewhere, but I have no
clue.
the error location *site.pp:21 *points to jdk property. I do have a jdk1.7
on my system, and was hoping puppet will automatically install 1.6 as
required by bigtop.
Below are the error log and the site.cv files. Hope to get some suggestion
from the email list.
Many thanks
Demai
----------------------------------------------------------------
error log
[ptadm@hdfs003 puppet]$ puppet apply -d --modulepath=modules --confdir=.
manifests/site.pp
/usr/lib/ruby/site_ruby/1.8/puppet/defaults.rb:214: warning: Insecure world
writable dir /home in PATH, mode 040777
Debug: Runtime environment: run_mode=user, ruby_version=1.8.7,
puppet_version=3.8.1
Info: Loading facts
Debug: Loading facts from
/opt/bigtop/bigtop-deploy/puppet/modules/kerberos/lib/facter/kadm_keytab.rb
Info: Loading facts
Debug: Loading facts from
/opt/bigtop/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_dirs.rb
Debug: Loading facts from
/opt/bigtop/bigtop-deploy/puppet/modules/hadoop/lib/facter/hadoop_storage_locations.rb
Warning: The use of 'import' is deprecated at
/opt/bigtop/bigtop-deploy/puppet/manifests/site.pp:46. See
http://links.puppetlabs.com/puppet-import-deprecation
(at /usr/lib/ruby/site_ruby/1.8/puppet/parser/parser_support.rb:110:in
`import')
Debug: importing '/opt/bigtop/bigtop-deploy/puppet/manifests/cluster.pp' in
environment production
Debug: importing
'/opt/bigtop/bigtop-deploy/puppet/modules/bigtop_util/manifests/init.pp' in
environment production
Debug: Automatically imported bigtop_util from bigtop_util into production
*Error: CSV::IllegalFormatError at
/opt/bigtop/bigtop-deploy/puppet/manifests/site.pp:21 on
node hdfs003.demai.com <http://hdfs003.demai.com/>*
*Error: CSV::IllegalFormatError at
/opt/bigtop/bigtop-deploy/puppet/manifests/site.pp:21 on
node hdfs003.demai.com <http://hdfs003.demai.com/>*
site.csv
### WARNING:
### actual site.csv file shouldn't contain lines starting with '#'
### It will cause the parse to choke.
### End of WARNING
### This file needs to be customized to reflect the configuration of your
cluster
### Store it as $BIGTOP_DEPLOY_PATH/config/site.csv
### use --confdir=$BIGTOP_DEPLOY_PATH (see README for more info)
# FQDN of Namenode
hadoop_head_node,hadoopmaster.example.com
# FQDN of standby node (for HA)
#standby_head_node,standbyNN.example.com <http://standbynn.example.com/>
# FQDN of gateway node (if separate from NN)
#standby_head_node,gateway.example.com
# Storage directories (will be created if doesn't exist)
hadoop_storage_dirs,/data/1,/data/2,/data/3,/data/4
bigtop_yumrepo_uri,http://mirror.example.com/path/to/mirror/
jdk_package_name,java-1.6.0-openjdk
# A list of stack' components to be deployed can be specified via special
# "$components" list. If $components isn't set then everything in the stack
will
# be installed as usual. Otherwise only a specified list will be set
# Possible elements:
#
hadoop,yarn,hbase,mapred-app,flume,solrcloud,spark,oozie,hcat,sqoop,httpfs,
# hue,mahout,giraph,crunch,pig,hive,zookeeper
# Example (to deploy only HDFS and YARN server and gateway parts)
components,hadoop,yarn,hbase
Re: Error: CSV::IllegalFormatError
Posted by Demai Ni <ni...@gmail.com>.
Martin,
many thanks for your response. Finally I got a chance to try out your
suggestion and exam the java environment of my node.
about the site.csv, here is my current configuration. Hopefully, the only
troubling one is the jdk package. for that value, I tried several different
one: *java-1.6.0-openjdk,
java-1.7.0-openjdk,java-1.6.0-openjdk.x86_64,java-1.7.0-openjdk.x86_64*.
But keeping get the same error.
The following is my csv and the jdk package info from yum. I must
misunderstand some points here.
/* site.csv */
"
hadoop_head_node,hdfs003.demai.com
hadoop_storage_dirs,/data/1,/data/2,/data/3,/data/4
bigtop_yumrepo_uri,
http://bigtop.s3.amazonaws.com/releases/0.8.0/redhat/6/x86_64
jdk_package_name,*java-1.6.0-openjdk*
components,hadoop,yarn,hbase
"
/* jdk packages */
[ptadm@hdfs003 pluginconf.d]$ yum list *OpenJDK*
base | 3.7
kB 00:00
bigtop | 2.9
kB 00:00
epel | 4.4
kB 00:00
epel/primary_db | 6.7
MB 00:12
extras | 3.4
kB 00:00
puppetlabs-deps | 2.5
kB 00:00
puppetlabs-products | 2.5
kB 00:00
remi | 2.9
kB 00:00
remi/primary_db | 1.1
MB 00:01
updates | 3.4
kB 00:00
updates/primary_db | 4.4
MB 00:00
virtualbox | 951
B 00:00
virtualbox
26/26
Installed Packages
java-1.6.0-openjdk.x86_64 1:1.6.0.0-6.1.13.4.el6_5
@updates
java-1.6.0-openjdk-devel.x86_64 1:1.6.0.0-6.1.13.4.el6_5
@updates
java-1.6.0-openjdk-javadoc.x86_64 1:1.6.0.0-6.1.13.4.el6_5
@updates
java-1.7.0-openjdk.x86_64 1:1.7.0.65-2.5.1.2.el6_5
@updates
java-1.7.0-openjdk-devel.x86_64 1:1.7.0.65-2.5.1.2.el6_5
@updates
Available Packages
java-1.6.0-openjdk.x86_64 1:1.6.0.35-1.13.7.1.el6_6
updates
java-1.6.0-openjdk-demo.x86_64 1:1.6.0.35-1.13.7.1.el6_6
updates
java-1.6.0-openjdk-devel.x86_64 1:1.6.0.35-1.13.7.1.el6_6
updates
java-1.6.0-openjdk-javadoc.x86_64 1:1.6.0.35-1.13.7.1.el6_6
updates
java-1.6.0-openjdk-src.x86_64 1:1.6.0.35-1.13.7.1.el6_6
updates
java-1.7.0-openjdk.x86_64 1:1.7.0.85-2.6.1.3.el6_6
updates
java-1.7.0-openjdk-demo.x86_64 1:1.7.0.85-2.6.1.3.el6_6
updates
java-1.7.0-openjdk-devel.x86_64 1:1.7.0.85-2.6.1.3.el6_6
updates
java-1.7.0-openjdk-javadoc.noarch 1:1.7.0.85-2.6.1.3.el6_6
updates
java-1.7.0-openjdk-src.x86_64 1:1.7.0.85-2.6.1.3.el6_6
updates
java-1.8.0-openjdk.x86_64 1:1.8.0.51-0.b16.el6_6
updates
java-1.8.0-openjdk-demo.x86_64 1:1.8.0.51-0.b16.el6_6
updates
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.51-0.b16.el6_6
updates
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.51-0.b16.el6_6
updates
java-1.8.0-openjdk-javadoc.noarch 1:1.8.0.51-0.b16.el6_6
updates
java-1.8.0-openjdk-src.x86_64 1:1.8.0.51-0.b16.el6_6
updates
On Wed, Jul 15, 2015 at 1:37 AM, Martin Bukatovic <mb...@redhat.com>
wrote:
> Hi Demai,
>
> see my reply inline:
>
> On 07/15/2015 12:18 AM, Demai Ni wrote:
> > I am following instruction here:
> >
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Bigtop+0.8.0+Hadoop+on+CentOS+6+with+Puppet
> >
> >
> > to setup a 2-node clusters. But failed at very early stage: puppet
> > apply. Well, quite possible that I made a small mistake somewhere, but I
> > have no clue.
> >
> >
> > the error location *site.pp:21 *points to jdk property. I do have a
> > jdk1.7 on my system, and was hoping puppet will automatically install
> > 1.6 as required by bigtop.
>
> You are right that puppet will install the jdk package based on value
> of this property, but you need to make this package available in the
> repositories of nodes you are deploying bigtop on first.
>
> That is because this jdk property is expected to be configured by
> an admin deploying bigtop based on java available on theirs machines.
> It doesn't magically try to download java package from somewhere
> behind the scene.
>
> So in your case, when you already have java 1.7 installed on your
> machines, you can
>
> * either check proper name of package for java 1.6 available in
> repositories of your distribution and edit the property
> accordingly
> * or use java 1.7 you already have installed on your system
> (but this may not work as official requirement of 0.8.0
> release is java 1.6)
>
> But looking at your example, you seem not to configure your
> site.csv file at all. You need to properly fill out all
> required values as described in the mentioned page, otherwise
> you can't expect it to work.
>
> Hope it helps a bit.
>
> --
> Martin Bukatovic
>
Re: Error: CSV::IllegalFormatError
Posted by Martin Bukatovic <mb...@redhat.com>.
Hi Demai,
see my reply inline:
On 07/15/2015 12:18 AM, Demai Ni wrote:
> I am following instruction here:
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Bigtop+0.8.0+Hadoop+on+CentOS+6+with+Puppet
>
>
> to setup a 2-node clusters. But failed at very early stage: puppet
> apply. Well, quite possible that I made a small mistake somewhere, but I
> have no clue.
>
>
> the error location *site.pp:21 *points to jdk property. I do have a
> jdk1.7 on my system, and was hoping puppet will automatically install
> 1.6 as required by bigtop.
You are right that puppet will install the jdk package based on value
of this property, but you need to make this package available in the
repositories of nodes you are deploying bigtop on first.
That is because this jdk property is expected to be configured by
an admin deploying bigtop based on java available on theirs machines.
It doesn't magically try to download java package from somewhere
behind the scene.
So in your case, when you already have java 1.7 installed on your
machines, you can
* either check proper name of package for java 1.6 available in
repositories of your distribution and edit the property
accordingly
* or use java 1.7 you already have installed on your system
(but this may not work as official requirement of 0.8.0
release is java 1.6)
But looking at your example, you seem not to configure your
site.csv file at all. You need to properly fill out all
required values as described in the mentioned page, otherwise
you can't expect it to work.
Hope it helps a bit.
--
Martin Bukatovic