You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by YouPeng Yang <yy...@gmail.com> on 2013/11/01 02:20:36 UTC
Any reference for upgrade hadoop from 1.x to 2.2
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x
to 2.2.
Regards
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Not that I'm aware of.
-Sandy
On Thu, Dec 5, 2013 at 10:11 PM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Thanks Sandy for the useful info.
>
>
>
> Is there any open JIRA issue for that?
>
>
>
> -Nirmal
>
>
>
> *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com]
> *Sent:* Thursday, December 05, 2013 10:38 PM
>
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Unfortunately there is no way to see MR1 jobs in the MR2 job history.
>
>
>
> -Sandy
>
>
>
> On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Not that I'm aware of.
-Sandy
On Thu, Dec 5, 2013 at 10:11 PM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Thanks Sandy for the useful info.
>
>
>
> Is there any open JIRA issue for that?
>
>
>
> -Nirmal
>
>
>
> *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com]
> *Sent:* Thursday, December 05, 2013 10:38 PM
>
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Unfortunately there is no way to see MR1 jobs in the MR2 job history.
>
>
>
> -Sandy
>
>
>
> On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Not that I'm aware of.
-Sandy
On Thu, Dec 5, 2013 at 10:11 PM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Thanks Sandy for the useful info.
>
>
>
> Is there any open JIRA issue for that?
>
>
>
> -Nirmal
>
>
>
> *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com]
> *Sent:* Thursday, December 05, 2013 10:38 PM
>
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Unfortunately there is no way to see MR1 jobs in the MR2 job history.
>
>
>
> -Sandy
>
>
>
> On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Not that I'm aware of.
-Sandy
On Thu, Dec 5, 2013 at 10:11 PM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Thanks Sandy for the useful info.
>
>
>
> Is there any open JIRA issue for that?
>
>
>
> -Nirmal
>
>
>
> *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com]
> *Sent:* Thursday, December 05, 2013 10:38 PM
>
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Unfortunately there is no way to see MR1 jobs in the MR2 job history.
>
>
>
> -Sandy
>
>
>
> On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Sandy for the useful info.
Is there any open JIRA issue for that?
-Nirmal
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, December 05, 2013 10:38 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310<http://127.0.0.1:54310>
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Sandy for the useful info.
Is there any open JIRA issue for that?
-Nirmal
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, December 05, 2013 10:38 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310<http://127.0.0.1:54310>
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Sandy for the useful info.
Is there any open JIRA issue for that?
-Nirmal
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, December 05, 2013 10:38 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310<http://127.0.0.1:54310>
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Sandy for the useful info.
Is there any open JIRA issue for that?
-Nirmal
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, December 05, 2013 10:38 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310<http://127.0.0.1:54310>
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
Unfortunately there is no way to see MR1 jobs in the MR2 job history.
-Sandy
On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1. Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2. HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> old LV = -41; old CTime = 0.
>
> new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
> cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
> However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <ka...@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <ka...@gmail.com>
>
> @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Adam,
Apache Hadoop-2.0.6-alpha has the following issue.
This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
1. Hadoop HDFS<https://issues.apache.org/jira/browse/HDFS>
2. HDFS-4917<https://issues.apache.org/jira/browse/HDFS-4917>
Start-dfs.sh cannot pass the parameters correctly
https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
I setup Apache Hadoop 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031> and then were able to run the commands :
./hadoop-daemon.sh start namenode -upgrade
./hdfs dfsadmin -finalizeUpgrade
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 21:16:44,412 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 21:16:44,426 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory = 889 MB
2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries
2013-12-05 21:16:44,923 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 21:16:44,930 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 21:16:44,931 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled
2013-12-05 21:16:44,932 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing capacity for map Namenode Retry Cache
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type = 32-bit
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory = 889 MB
2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries
2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 8695@Impetus-942.impetus.co.in
2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage: Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
2013-12-05 21:16:45,145 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering unfinalized segments in /home/cloud/hadoop_migration/hadoop-data/name/current
2013-12-05 21:16:45,166 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential block IDs. Generation stamp for new blocks set to 1099511628823
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no compression
2013-12-05 21:16:45,169 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
2013-12-05 21:16:45,203 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under construction = 0
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975 bytes loaded in 0 seconds.
2013-12-05 21:16:45,204 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0 from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Reading /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start txid #1
2013-12-05 21:16:45,211 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits
2013-12-05 21:16:45,232 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits # 0 loaded in 0 seconds
2013-12-05 21:16:45,233 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image directory /home/cloud/hadoop_migration/hadoop-data/name.
old LV = -41; old CTime = 0.
new LV = -47; new CTime = 1386258405233
2013-12-05 21:16:45,241 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 using no compression
2013-12-05 21:16:45,321 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000 of size 4923 bytes saved in 0 seconds.
2013-12-05 21:16:45,365 INFO org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector: No version file in /home/cloud/hadoop_migration/hadoop-data/name
2013-12-05 21:16:45,421 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of /home/cloud/hadoop_migration/hadoop-data/name is complete.
2013-12-05 21:16:45,422 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
2013-12-05 21:16:45,741 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 791 msecs
2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 54310
2013-12-05 21:16:46,113 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemState MBean
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,126 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under construction: 0
2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 0 needs additional 15 blocks to reach the threshold 0.9990 of total blocks 15.
Safe mode will be turned off automatically
2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:54310
2013-12-05 21:16:46,177 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for storage directory /home/cloud/hadoop_migration/hadoop-data/name.
cur LV = -47; cur CTime = 1386258405233
2013-12-05 21:23:08,461 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for /home/cloud/hadoop_migration/hadoop-data/name is complete.
I can now see the existing files in the HDFS that were used by earlier Map Reduce jobs(input\output files) using Apache Hadoop-1.2.0
However, I cannot see the history of those Map Reduce jobs through MapReduce JobHistory Server.
Is there some way in which I can see the history of those Map Reduce jobs as well?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, December 04, 2013 7:41 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Cc: rdyer@iastate.edu<ma...@iastate.edu>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Thanks Adam,
I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*
I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode -upgrade
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 31742@Impetus-942.impetus.co.in
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$
I have the following environment variables set :
YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
Regards,
-Nirmal
From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>>
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>>
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in<ma...@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>; rdyer@iastate.edu<ma...@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>
> @Nirmal,
>
> You need to run NameNode with upgrade option e.g.
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
>> Hi All,
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>>
>>
>> I did the following:
>>
>> · Installed Apache Hadoop-1.2.0
>>
>> · Ran word count sample MR jobs. The jobs executed successfully.
>>
>> · I stop all the services in Apache Hadoop-1.2.0 and then was
>> able to start all services again.
>>
>> · The previous submitted jobs are visible after the stop/start
>> in the job tracker url.
>>
>>
>>
>> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>>
>> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
>> in the configuration files namely:
>>
>> core-site.xml
>>
>> ----------------
>>
>> $hadoop.tmp.dir
>> /home/cloud/hadoop_migration/hadoop-data/tempdir
>>
>>
>>
>> hdfs-site.xml
>>
>> -----------------
>>
>> $dfs.data.dir
>> /home/cloud/hadoop_migration/hadoop-data/data
>>
>> $dfs.name.dir
>> /home/cloud/hadoop_migration/hadoop-data/name
>>
>>
>>
>> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
>> installation I am getting the error:
>>
>>
>>
>> 2013-12-03 18:28:23,941 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>>
>> 2013-12-03 18:28:24,080 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>>
>> 2013-12-03 18:28:24,081 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>>
>> 2013-12-03 18:28:24,749 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication = 512
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication = 1
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> cloud (auth:SIMPLE)
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> supergroup
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>
>> 2013-12-03 18:28:24,776 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>
>> 2013-12-03 18:28:25,230 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>>
>> 2013-12-03 18:28:25,243 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 30000
>>
>> 2013-12-03 18:28:25,288 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by
>> nodename 21371@Impetus-942.impetus.co.in
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>>
>> 2013-12-03 18:28:25,473 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>>
>> 2013-12-03 18:28:25,474 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>
>> org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
>> Unexpected version of storage directory
>> /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting =
>> -40.
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>>
>> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>>
>> 2013-12-03 18:28:25,481 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>
>> /************************************************************
>>
>> SHUTDOWN_MSG: Shutting down NameNode at
>> Impetus-942.impetus.co.in/192.168.41.106
>>
>> ************************************************************/
>>
>>
>>
>> Independently both the installations(Apache Hadoop-1.2.0 and Apache
>> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
>> both the installations independently though.
>>
>> But I aim to migrate the data and jobs submitted from Apache
>> Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
>>
>>
>>
>> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha?
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Nirmal Kumar
>> *Sent:* Wednesday, November 27, 2013 2:56 PM
>> *To:* user@hadoop.apache.org; rdyer@iastate.edu
>> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Hello Sandy,
>>
>>
>>
>> The post was useful and gave an insight of the migration.
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>> I am having the Apache Hadoop-1.2.0 up and running.
>>
>>
>>
>> Can you please let me know the steps that one should follow for the
>> migration?
>>
>> I am thinking of doing something like:
>>
>> · Install Apache Hadoop-2.0.6-alpha alongside the existing
>> Apache Hadoop-1.2.0
>>
>> · Use the same HDFS locations
>>
>> · Change the various required configuration files
>>
>> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>>
>> · Verify all the services are running
>>
>> · Test via mapreduce (test MRv1 and MRv2 examples)
>>
>> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>>
>>
>>
>> These above steps needs to be performed on all the nodes in a cluster
>> environment.
>>
>>
>>
>> The translation table mapping old configuration to new would be
>> definitely *very* useful.
>>
>>
>>
>> Also the existing Hadoop ecosystem components needs to be considered:
>>
>> · Hive Scripts
>>
>> · Pig Scripts
>>
>> · Oozie Workflows
>>
>> Their compatibility and version support would need to be checked.
>>
>>
>>
>> Also thinking of any risks like Data Loss, others that one should keep in
>> mind.
>>
>>
>>
>> Also I found:
>> http://strataconf.com/strata2014/public/schedule/detail/32247
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
>> *Sent:* Friday, November 22, 2013 9:08 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Thanks Sandy! These seem helpful!
>>
>>
>>
>> "MapReduce cluster configuration options have been split into YARN
>> configuration options, which go in yarn-site.xml; and MapReduce
>> configuration options, which go in mapred-site.xml. Many have been given
>> new names to reflect the shift. ... *We’ll follow up with a full
>> translation table in a future post.*"
>>
>>
>> This type of translation table mapping old configuration to new would be
>> *very* useful!
>>
>>
>>
>> - Robert
>>
>> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
>> wrote:
>>
>> For MapReduce and YARN, we recently published a couple blog posts on
>> migrating:
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>>
>>
>>
>> hope that helps,
>>
>> Sandy
>>
>>
>>
>> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>
> @Nirmal,
>
> You need to run NameNode with upgrade option e.g.
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
>> Hi All,
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>>
>>
>> I did the following:
>>
>> · Installed Apache Hadoop-1.2.0
>>
>> · Ran word count sample MR jobs. The jobs executed successfully.
>>
>> · I stop all the services in Apache Hadoop-1.2.0 and then was
>> able to start all services again.
>>
>> · The previous submitted jobs are visible after the stop/start
>> in the job tracker url.
>>
>>
>>
>> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>>
>> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
>> in the configuration files namely:
>>
>> core-site.xml
>>
>> ----------------
>>
>> $hadoop.tmp.dir
>> /home/cloud/hadoop_migration/hadoop-data/tempdir
>>
>>
>>
>> hdfs-site.xml
>>
>> -----------------
>>
>> $dfs.data.dir
>> /home/cloud/hadoop_migration/hadoop-data/data
>>
>> $dfs.name.dir
>> /home/cloud/hadoop_migration/hadoop-data/name
>>
>>
>>
>> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
>> installation I am getting the error:
>>
>>
>>
>> 2013-12-03 18:28:23,941 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>>
>> 2013-12-03 18:28:24,080 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>>
>> 2013-12-03 18:28:24,081 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>>
>> 2013-12-03 18:28:24,749 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication = 512
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication = 1
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> cloud (auth:SIMPLE)
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> supergroup
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>
>> 2013-12-03 18:28:24,776 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>
>> 2013-12-03 18:28:25,230 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>>
>> 2013-12-03 18:28:25,243 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 30000
>>
>> 2013-12-03 18:28:25,288 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by
>> nodename 21371@Impetus-942.impetus.co.in
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>>
>> 2013-12-03 18:28:25,473 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>>
>> 2013-12-03 18:28:25,474 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>
>> org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
>> Unexpected version of storage directory
>> /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting =
>> -40.
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>>
>> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>>
>> 2013-12-03 18:28:25,481 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>
>> /************************************************************
>>
>> SHUTDOWN_MSG: Shutting down NameNode at
>> Impetus-942.impetus.co.in/192.168.41.106
>>
>> ************************************************************/
>>
>>
>>
>> Independently both the installations(Apache Hadoop-1.2.0 and Apache
>> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
>> both the installations independently though.
>>
>> But I aim to migrate the data and jobs submitted from Apache
>> Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
>>
>>
>>
>> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha?
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Nirmal Kumar
>> *Sent:* Wednesday, November 27, 2013 2:56 PM
>> *To:* user@hadoop.apache.org; rdyer@iastate.edu
>> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Hello Sandy,
>>
>>
>>
>> The post was useful and gave an insight of the migration.
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>> I am having the Apache Hadoop-1.2.0 up and running.
>>
>>
>>
>> Can you please let me know the steps that one should follow for the
>> migration?
>>
>> I am thinking of doing something like:
>>
>> · Install Apache Hadoop-2.0.6-alpha alongside the existing
>> Apache Hadoop-1.2.0
>>
>> · Use the same HDFS locations
>>
>> · Change the various required configuration files
>>
>> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>>
>> · Verify all the services are running
>>
>> · Test via mapreduce (test MRv1 and MRv2 examples)
>>
>> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>>
>>
>>
>> These above steps needs to be performed on all the nodes in a cluster
>> environment.
>>
>>
>>
>> The translation table mapping old configuration to new would be
>> definitely *very* useful.
>>
>>
>>
>> Also the existing Hadoop ecosystem components needs to be considered:
>>
>> · Hive Scripts
>>
>> · Pig Scripts
>>
>> · Oozie Workflows
>>
>> Their compatibility and version support would need to be checked.
>>
>>
>>
>> Also thinking of any risks like Data Loss, others that one should keep in
>> mind.
>>
>>
>>
>> Also I found:
>> http://strataconf.com/strata2014/public/schedule/detail/32247
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
>> *Sent:* Friday, November 22, 2013 9:08 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Thanks Sandy! These seem helpful!
>>
>>
>>
>> "MapReduce cluster configuration options have been split into YARN
>> configuration options, which go in yarn-site.xml; and MapReduce
>> configuration options, which go in mapred-site.xml. Many have been given
>> new names to reflect the shift. ... *We’ll follow up with a full
>> translation table in a future post.*"
>>
>>
>> This type of translation table mapping old configuration to new would be
>> *very* useful!
>>
>>
>>
>> - Robert
>>
>> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
>> wrote:
>>
>> For MapReduce and YARN, we recently published a couple blog posts on
>> migrating:
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>>
>>
>>
>> hope that helps,
>>
>> Sandy
>>
>>
>>
>> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>
> @Nirmal,
>
> You need to run NameNode with upgrade option e.g.
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
>> Hi All,
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>>
>>
>> I did the following:
>>
>> · Installed Apache Hadoop-1.2.0
>>
>> · Ran word count sample MR jobs. The jobs executed successfully.
>>
>> · I stop all the services in Apache Hadoop-1.2.0 and then was
>> able to start all services again.
>>
>> · The previous submitted jobs are visible after the stop/start
>> in the job tracker url.
>>
>>
>>
>> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>>
>> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
>> in the configuration files namely:
>>
>> core-site.xml
>>
>> ----------------
>>
>> $hadoop.tmp.dir
>> /home/cloud/hadoop_migration/hadoop-data/tempdir
>>
>>
>>
>> hdfs-site.xml
>>
>> -----------------
>>
>> $dfs.data.dir
>> /home/cloud/hadoop_migration/hadoop-data/data
>>
>> $dfs.name.dir
>> /home/cloud/hadoop_migration/hadoop-data/name
>>
>>
>>
>> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
>> installation I am getting the error:
>>
>>
>>
>> 2013-12-03 18:28:23,941 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>>
>> 2013-12-03 18:28:24,080 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>>
>> 2013-12-03 18:28:24,081 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>>
>> 2013-12-03 18:28:24,749 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication = 512
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication = 1
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> cloud (auth:SIMPLE)
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> supergroup
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>
>> 2013-12-03 18:28:24,776 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>
>> 2013-12-03 18:28:25,230 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>>
>> 2013-12-03 18:28:25,243 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 30000
>>
>> 2013-12-03 18:28:25,288 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by
>> nodename 21371@Impetus-942.impetus.co.in
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>>
>> 2013-12-03 18:28:25,473 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>>
>> 2013-12-03 18:28:25,474 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>
>> org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
>> Unexpected version of storage directory
>> /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting =
>> -40.
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>>
>> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>>
>> 2013-12-03 18:28:25,481 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>
>> /************************************************************
>>
>> SHUTDOWN_MSG: Shutting down NameNode at
>> Impetus-942.impetus.co.in/192.168.41.106
>>
>> ************************************************************/
>>
>>
>>
>> Independently both the installations(Apache Hadoop-1.2.0 and Apache
>> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
>> both the installations independently though.
>>
>> But I aim to migrate the data and jobs submitted from Apache
>> Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
>>
>>
>>
>> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha?
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Nirmal Kumar
>> *Sent:* Wednesday, November 27, 2013 2:56 PM
>> *To:* user@hadoop.apache.org; rdyer@iastate.edu
>> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Hello Sandy,
>>
>>
>>
>> The post was useful and gave an insight of the migration.
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>> I am having the Apache Hadoop-1.2.0 up and running.
>>
>>
>>
>> Can you please let me know the steps that one should follow for the
>> migration?
>>
>> I am thinking of doing something like:
>>
>> · Install Apache Hadoop-2.0.6-alpha alongside the existing
>> Apache Hadoop-1.2.0
>>
>> · Use the same HDFS locations
>>
>> · Change the various required configuration files
>>
>> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>>
>> · Verify all the services are running
>>
>> · Test via mapreduce (test MRv1 and MRv2 examples)
>>
>> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>>
>>
>>
>> These above steps needs to be performed on all the nodes in a cluster
>> environment.
>>
>>
>>
>> The translation table mapping old configuration to new would be
>> definitely *very* useful.
>>
>>
>>
>> Also the existing Hadoop ecosystem components needs to be considered:
>>
>> · Hive Scripts
>>
>> · Pig Scripts
>>
>> · Oozie Workflows
>>
>> Their compatibility and version support would need to be checked.
>>
>>
>>
>> Also thinking of any risks like Data Loss, others that one should keep in
>> mind.
>>
>>
>>
>> Also I found:
>> http://strataconf.com/strata2014/public/schedule/detail/32247
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
>> *Sent:* Friday, November 22, 2013 9:08 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Thanks Sandy! These seem helpful!
>>
>>
>>
>> "MapReduce cluster configuration options have been split into YARN
>> configuration options, which go in yarn-site.xml; and MapReduce
>> configuration options, which go in mapred-site.xml. Many have been given
>> new names to reflect the shift. ... *We’ll follow up with a full
>> translation table in a future post.*"
>>
>>
>> This type of translation table mapping old configuration to new would be
>> *very* useful!
>>
>>
>>
>> - Robert
>>
>> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
>> wrote:
>>
>> For MapReduce and YARN, we recently published a couple blog posts on
>> migrating:
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>>
>>
>>
>> hope that helps,
>>
>> Sandy
>>
>>
>>
>> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
And later, you need to make a decision to finalize the upgrade or rollback.
2013/12/3 Adam Kawa <ka...@gmail.com>
> @Nirmal,
>
> You need to run NameNode with upgrade option e.g.
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
> 2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
>
>> Hi All,
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>>
>>
>> I did the following:
>>
>> · Installed Apache Hadoop-1.2.0
>>
>> · Ran word count sample MR jobs. The jobs executed successfully.
>>
>> · I stop all the services in Apache Hadoop-1.2.0 and then was
>> able to start all services again.
>>
>> · The previous submitted jobs are visible after the stop/start
>> in the job tracker url.
>>
>>
>>
>> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>>
>> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
>> in the configuration files namely:
>>
>> core-site.xml
>>
>> ----------------
>>
>> $hadoop.tmp.dir
>> /home/cloud/hadoop_migration/hadoop-data/tempdir
>>
>>
>>
>> hdfs-site.xml
>>
>> -----------------
>>
>> $dfs.data.dir
>> /home/cloud/hadoop_migration/hadoop-data/data
>>
>> $dfs.name.dir
>> /home/cloud/hadoop_migration/hadoop-data/name
>>
>>
>>
>> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
>> installation I am getting the error:
>>
>>
>>
>> 2013-12-03 18:28:23,941 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>>
>> 2013-12-03 18:28:24,080 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>>
>> 2013-12-03 18:28:24,081 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
>> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
>> lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,576 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
>> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
>> dataloss due to lack of redundant storage directories!
>>
>> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
>> Refreshing hosts (include/exclude) list
>>
>> 2013-12-03 18:28:24,749 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
>> dfs.block.invalidate.limit=1000
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> dfs.block.access.token.enable=false
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> defaultReplication = 1
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplication = 512
>>
>> 2013-12-03 18:28:24,762 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> minReplication = 1
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> maxReplicationStreams = 2
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> shouldCheckForEnoughRacks = false
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> replicationRecheckInterval = 3000
>>
>> 2013-12-03 18:28:24,763 INFO
>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
>> encryptDataTransfer = false
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
>> cloud (auth:SIMPLE)
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
>> supergroup
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
>> true
>>
>> 2013-12-03 18:28:24,771 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>>
>> 2013-12-03 18:28:24,776 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>>
>> 2013-12-03 18:28:25,230 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>>
>> 2013-12-03 18:28:25,243 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.min.datanodes = 0
>>
>> 2013-12-03 18:28:25,244 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.namenode.safemode.extension = 30000
>>
>> 2013-12-03 18:28:25,288 INFO
>> org.apache.hadoop.hdfs.server.common.Storage: Lock on
>> /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by
>> nodename 21371@Impetus-942.impetus.co.in
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
>> metrics system...
>>
>> 2013-12-03 18:28:25,462 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> stopped.
>>
>> 2013-12-03 18:28:25,473 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> shutdown complete.
>>
>> 2013-12-03 18:28:25,474 FATAL
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>>
>> org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
>> Unexpected version of storage directory
>> /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting =
>> -40.
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>>
>> at
>> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>>
>> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting
>> with status 1
>>
>> 2013-12-03 18:28:25,481 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>
>> /************************************************************
>>
>> SHUTDOWN_MSG: Shutting down NameNode at
>> Impetus-942.impetus.co.in/192.168.41.106
>>
>> ************************************************************/
>>
>>
>>
>> Independently both the installations(Apache Hadoop-1.2.0 and Apache
>> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
>> both the installations independently though.
>>
>> But I aim to migrate the data and jobs submitted from Apache
>> Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
>>
>>
>>
>> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha?
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Nirmal Kumar
>> *Sent:* Wednesday, November 27, 2013 2:56 PM
>> *To:* user@hadoop.apache.org; rdyer@iastate.edu
>> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Hello Sandy,
>>
>>
>>
>> The post was useful and gave an insight of the migration.
>>
>>
>>
>> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
>> Hadoop-2.0.6-alpha on a single node environment.
>>
>> I am having the Apache Hadoop-1.2.0 up and running.
>>
>>
>>
>> Can you please let me know the steps that one should follow for the
>> migration?
>>
>> I am thinking of doing something like:
>>
>> · Install Apache Hadoop-2.0.6-alpha alongside the existing
>> Apache Hadoop-1.2.0
>>
>> · Use the same HDFS locations
>>
>> · Change the various required configuration files
>>
>> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>>
>> · Verify all the services are running
>>
>> · Test via mapreduce (test MRv1 and MRv2 examples)
>>
>> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>>
>>
>>
>> These above steps needs to be performed on all the nodes in a cluster
>> environment.
>>
>>
>>
>> The translation table mapping old configuration to new would be
>> definitely *very* useful.
>>
>>
>>
>> Also the existing Hadoop ecosystem components needs to be considered:
>>
>> · Hive Scripts
>>
>> · Pig Scripts
>>
>> · Oozie Workflows
>>
>> Their compatibility and version support would need to be checked.
>>
>>
>>
>> Also thinking of any risks like Data Loss, others that one should keep in
>> mind.
>>
>>
>>
>> Also I found:
>> http://strataconf.com/strata2014/public/schedule/detail/32247
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
>> *Sent:* Friday, November 22, 2013 9:08 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> Thanks Sandy! These seem helpful!
>>
>>
>>
>> "MapReduce cluster configuration options have been split into YARN
>> configuration options, which go in yarn-site.xml; and MapReduce
>> configuration options, which go in mapred-site.xml. Many have been given
>> new names to reflect the shift. ... *We’ll follow up with a full
>> translation table in a future post.*"
>>
>>
>> This type of translation table mapping old configuration to new would be
>> *very* useful!
>>
>>
>>
>> - Robert
>>
>> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
>> wrote:
>>
>> For MapReduce and YARN, we recently published a couple blog posts on
>> migrating:
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>>
>>
>> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>>
>>
>>
>> hope that helps,
>>
>> Sandy
>>
>>
>>
>> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Adam Kawa <ka...@gmail.com>.
@Nirmal,
You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
2013/12/3 Nirmal Kumar <ni...@impetus.co.in>
> Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> · Installed Apache Hadoop-1.2.0
>
> · Ran word count sample MR jobs. The jobs executed successfully.
>
> · I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> · The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
> at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> · Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> · Use the same HDFS locations
>
> · Change the various required configuration files
>
> · Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
>
> · Verify all the services are running
>
> · Test via mapreduce (test MRv1 and MRv2 examples)
>
> · Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> · Hive Scripts
>
> · Pig Scripts
>
> · Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <ps...@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>
> wrote:
>
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
> ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org; rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org; rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org; rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I did the following:
* Installed Apache Hadoop-1.2.0
* Ran word count sample MR jobs. The jobs executed successfully.
* I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services again.
* The previous submitted jobs are visible after the stop/start in the job tracker url.
Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration files namely:
core-site.xml
----------------
$hadoop.tmp.dir /home/cloud/hadoop_migration/hadoop-data/tempdir
hdfs-site.xml
-----------------
$dfs.data.dir /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir /home/cloud/hadoop_migration/hadoop-data/name
I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting the error:
2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval = 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired by nodename 21371@Impetus-942.impetus.co.in
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/
Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.
Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?
Thanks,
-Nirmal
From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org; rdyer@iastate.edu
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hello Sandy,
The post was useful and gave an insight of the migration.
I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single node environment.
I am having the Apache Hadoop-1.2.0 up and running.
Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:
* Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0
* Use the same HDFS locations
* Change the various required configuration files
* Stop Apache Hadoop-1.2.0 and start Apache Hadoop-2.0.6-alpha
* Verify all the services are running
* Test via mapreduce (test MRv1 and MRv2 examples)
* Check Web UI Console and verify the MRv1 and MRv2 jobs
These above steps needs to be performed on all the nodes in a cluster environment.
The translation table mapping old configuration to new would be definitely *very* useful.
Also the existing Hadoop ecosystem components needs to be considered:
* Hive Scripts
* Pig Scripts
* Oozie Workflows
Their compatibility and version support would need to be checked.
Also thinking of any risks like Data Loss, others that one should keep in mind.
Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247
Thanks,
-Nirmal
From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN configuration options, which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml. Many have been given new names to reflect the shift. ... We'll follow up with a full translation table in a future post."
This type of translation table mapping old configuration to new would be *very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com>> wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com<ma...@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Robert Dyer <ps...@gmail.com>.
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN
configuration options, which go in yarn-site.xml; and MapReduce
configuration options, which go in mapred-site.xml. Many have been given
new names to reflect the shift. ... *We’ll follow up with a full
translation table in a future post.*"
This type of translation table mapping old configuration to new would be
*very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
> hope that helps,
> Sandy
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Robert Dyer <ps...@gmail.com>.
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN
configuration options, which go in yarn-site.xml; and MapReduce
configuration options, which go in mapred-site.xml. Many have been given
new names to reflect the shift. ... *We’ll follow up with a full
translation table in a future post.*"
This type of translation table mapping old configuration to new would be
*very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
> hope that helps,
> Sandy
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Robert Dyer <ps...@gmail.com>.
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN
configuration options, which go in yarn-site.xml; and MapReduce
configuration options, which go in mapred-site.xml. Many have been given
new names to reflect the shift. ... *We’ll follow up with a full
translation table in a future post.*"
This type of translation table mapping old configuration to new would be
*very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
> hope that helps,
> Sandy
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Robert Dyer <ps...@gmail.com>.
Thanks Sandy! These seem helpful!
"MapReduce cluster configuration options have been split into YARN
configuration options, which go in yarn-site.xml; and MapReduce
configuration options, which go in mapred-site.xml. Many have been given
new names to reflect the shift. ... *We’ll follow up with a full
translation table in a future post.*"
This type of translation table mapping old configuration to new would be
*very* useful!
- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
> hope that helps,
> Sandy
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
>
>> Hi All,
>>
>>
>>
>> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
>> Apache Hadoop 2.x.
>>
>> I didn’t find any doc\guide\blogs for the same.
>>
>> Although there are guides\docs for the CDH and HDP migration\upgradation
>> from Hadoop 1.x to Hadoop 2.x
>>
>> Would referring those be of some use?
>>
>>
>>
>> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
>> Hadoop 2.x.
>>
>>
>>
>> I found something on slideshare though. Not sure how much useful that is
>> going to be. I still need to verify that.
>>
>> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>>
>>
>>
>> Any suggestions\comments will be of great help.
>>
>>
>>
>> Thanks,
>>
>> -Nirmal
>>
>>
>>
>> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
>> *Sent:* Friday, November 08, 2013 9:13 PM
>> *To:* user@hadoop.apache.org
>> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>>
>>
>>
>> I am looking for the same thing if anyone can point us to a good
>> direction please.
>>
>> Thank you.
>>
>> (Currently running Hadoop 1.2.1)
>>
>>
>>
>> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>>
>> Hi users
>>
>> Are there any reference docs to introduce how to upgrade hadoop from
>> 1.x to 2.2.
>>
>>
>>
>> Regards
>>
>>
>>
>> ------------------------------
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited when
>> received in error. Impetus does not represent, warrant and/or guarantee,
>> that the integrity of this communication has been maintained nor that the
>> communication is free of errors, virus, interception or interference.
>>
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
For MapReduce and YARN, we recently published a couple blog posts on
migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
For MapReduce and YARN, we recently published a couple blog posts on
migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
For MapReduce and YARN, we recently published a couple blog posts on
migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Sandy Ryza <sa...@cloudera.com>.
For MapReduce and YARN, we recently published a couple blog posts on
migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
hope that helps,
Sandy
On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yy...@gmail.com>
>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn’t find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn’t find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn’t find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi All,
I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn’t find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to Hadoop 2.x
Would referring those be of some use?
I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.
I found something on slideshare though. Not sure how much useful that is going to be. I still need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
Any suggestions\comments will be of great help.
Thanks,
-Nirmal
From: Jilal Oussama [mailto:jilal.oussama@gmail.com]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2
I am looking for the same thing if anyone can point us to a good direction please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>>
Hi users
Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.
Regards
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Jilal Oussama <ji...@gmail.com>.
I am looking for the same thing if anyone can point us to a good direction
please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Jilal Oussama <ji...@gmail.com>.
I am looking for the same thing if anyone can point us to a good direction
please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Jilal Oussama <ji...@gmail.com>.
I am looking for the same thing if anyone can point us to a good direction
please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
Re: Any reference for upgrade hadoop from 1.x to 2.2
Posted by Jilal Oussama <ji...@gmail.com>.
I am looking for the same thing if anyone can point us to a good direction
please.
Thank you.
(Currently running Hadoop 1.2.1)
2013/11/1 YouPeng Yang <yy...@gmail.com>
> Hi users
>
> Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>