You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Alok Kumar <al...@gmail.com> on 2012/07/23 10:26:21 UTC
problem configuring hadoop with s3 bucket
Hello Group,
I've hadoop setup locally running.
Now I want to use Amazon s3://<mybucket> as my data store,
so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
hdfs-site.xml, Is it the correct way?
I'm getting error :
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
2012-07-23 13:15:06,260 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
dfs.data.dir are invalid.
and
when i changed like " dfs.data.dir=s3://<mybucket>/ "
I got error :
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Also,
When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not coming
up with error : ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
java.net.BindException: (Any way I want to run namenode locally, so I
reverted it back to hdfs://localhost:9000 )
Your help is highly appreciated!
Thanks
--
Alok Kumar
Re: problem configuring hadoop with s3 bucket
Posted by Yanbo Liang <ya...@gmail.com>.
>From the log, I guess that you have made wrong configurations or operations.
The S3 storage layer is at the same level with HDFS.
So if you configure to use S3, it's irrational to occur the HDFS related
logs such as "org.apache.hadoop.hdfs.server.namenode.FSNamesystem".
The HDFS related logs only occur at the condition that you use HDFS as the
underlying storage layer.
I infer that you should not format the file system, because S3 is already
service delivery and the Hadoop framework just use it as the underlying
file system.
Don't execute " bin/hadoop namenode -format ", it is used only when your
underlying file system is HDFS.
Yanbo
2012/7/25 Alok Kumar <al...@gmail.com>
> Hi Yanbo,
>
> I'm using hadoop-1.0.1 on local machine (not EC2)
>
> conf/master : localhost
> conf/slave : localhost
> conf/core-site.xml :
> <property>
> <name>fs.default.name</name>
> <value>s3://bucketname</value>
> </property>
>
> <property>
> <name>fs.s3.awsAccessKeyId</name>
> <value>***********</value>
> </property>
>
> <property>
> <name>fs.s3.awsSecretAccessKey</name>
> <value>**************</value>
> </property>
>
> /etc/hosts :
> 192.168.2.24 DRPM4 # Added by NetworkManager
> 127.0.0.1 localhost.localdomain localhost
>
>
> $ bin/hadoop namenode -format : successful
>
> $ bin/start-dfs.sh
>
> Namenode log :
>
> 2012-07-25 13:19:32,896 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = DRPM4/192.168.2.24
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.1
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012
> ************************************************************/
> 2012-07-25 13:19:33,037 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-07-25 13:19:33,049 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-07-25 13:19:33,050 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-07-25 13:19:33,050 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2012-07-25 13:19:33,889 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-07-25 13:19:33,896 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2012-07-25 13:19:33,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2012-07-25 13:19:33,906 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> NameNode registered.
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: VM
> type = 32-bit
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
> memory = 17.77875 MB
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet:
> capacity = 2^22 = 4194304 entries
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet:
> recommended=4194304, actual=4194304
> 2012-07-25 13:19:33,986 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=alok
> 2012-07-25 13:19:33,986 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
> 2012-07-25 13:19:33,986 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2012-07-25 13:19:33,995 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 2012-07-25 13:19:33,995 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
> accessTokenLifetime=0 min(s)
> 2012-07-25 13:19:34,396 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStateMBean and NameNodeMXBean
> 2012-07-25 13:19:34,416 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2012-07-25 13:19:34,442 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Number of files = 1
> 2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Number of files under construction = 0
> 2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 110 loaded in 0 seconds.
> 2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Edits file /home/alok/work/s3s/hadoop/namenode/current/edits of size 4
> edits # 0 loaded in 0 seconds.
> 2012-07-25 13:19:34,451 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 110 saved in 0 seconds.
> 2012-07-25 13:19:35,161 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 110 saved in 0 seconds.
> 2012-07-25 13:19:35,873 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
> 2012-07-25 13:19:35,873 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 1900 msecs
> 2012-07-25 13:19:35,888 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks
> = 0
> 2012-07-25 13:19:35,888 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid
> blocks = 0
> 2012-07-25 13:19:35,888 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> under-replicated blocks = 0
> 2012-07-25 13:19:35,888 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> over-replicated blocks = 0
> 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode termination scan for invalid, over- and under-replicated blocks
> completed in 15 msec
> 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Leaving safe mode after 1 secs.
> 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Network topology has 0 racks and 0 datanodes
> 2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> UnderReplicatedBlocks has 0 blocks
> 2012-07-25 13:19:35,897 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2012-07-25 13:19:35,905 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue
> QueueProcessingStatistics: First cycle completed 0 blocks in 7 msec
> 2012-07-25 13:19:35,905 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue
> QueueProcessingStatistics: Queue flush completed 0 blocks in 7 msec
> processing time, 7 msec clock time, 1 cycles
> 2012-07-25 13:19:35,905 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> FSNamesystemMetrics registered.
> 2012-07-25 13:19:35,906 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue
> QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec
> 2012-07-25 13:19:35,906 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue
> QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec
> processing time, 1 msec clock time, 1 cycles
> 2012-07-25 13:19:35,916 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> transactions: 0 Total time for transactions(ms): 0Number of transactions
> batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
> 2012-07-25 13:19:35,916 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicationMonitor
> thread received InterruptedException.java.lang.InterruptedException: sleep
> interrupted
> 2012-07-25 13:19:35,917 INFO
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager: Interrupted
> Monitor
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-25 13:19:35,993 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
> Problem binding to *bucket*/67.215.65.132:8020 : Cannot assign requested
> address
> at org.apache.hadoop.ipc.Server.bind(Server.java:227)
> at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
> at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
> at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
> Caused by: java.net.BindException: Cannot assign requested address
> at sun.nio.ch.Net.bind(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at org.apache.hadoop.ipc.Server.bind(Server.java:225)
> ... 8 more
>
> 2012-07-25 13:19:35,994 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at DRPM4/192.168.2.24
> ************************************************************/
>
> Thanks,
>
>
>
> On Wed, Jul 25, 2012 at 12:05 PM, Yanbo Liang <ya...@gmail.com>wrote:
>
>> Could you provide your execute environment, operation procedure and the
>> detail log information?
>>
>>
>> 2012/7/24 Alok Kumar <al...@gmail.com>
>>
>>> Hi Yanbo,
>>>
>>> Thank you for your reply..
>>>
>>> Now I've made changes exactly , present in this link
>>> http://wiki.apache.org/hadoop/AmazonS3
>>>
>>> But my namenode is not coming up with exception (I tried it both locally
>>> and inside EC2)
>>>
>>> ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
>>> java.net.BindException: Problem binding to <bucket-name>.
>>> s3.amazonaws.com/207.171.163.14:8020 : Cannot assign requested address
>>>
>>> fs.default.name = s3://<mybucket> // Error : UnknownHost Exception
>>> in Namenode!
>>> or
>>> fs.default.name = s3://<mybucket>.s3.amazonaws.com // Error
>>> BindException in Namenode log!
>>>
>>> Also I can't see any dir created inside my bucket. ( I'm able to execute
>>> command $ bin/hadoop dfs -ls s3://<bucket>/ )
>>>
>>> "bin/hadoop namenode -format " is saying succesfully formatted namenode
>>> dir S3://bucket/hadoop/namenode , when it is not even existing there!
>>>
>>>
>>> any suggestion?
>>>
>>> Thanks again.
>>>
>>>
>>> On Tue, Jul 24, 2012 at 4:11 PM, Yanbo Liang <ya...@gmail.com>wrote:
>>>
>>>> I think you have made confusion about the integration of hadoop and S3.
>>>> 1) If you set "dfs.data.dir=s3://******", it means that you set S3 as
>>>> the DataNode local storage.
>>>> You are still using HDFS as the under storage layer.
>>>> As far as I know, it is not supported at present.
>>>> 2) The right way of using S3 integrated with Hadoop is to replace the
>>>> HDFS with S3 just like you have tried.
>>>> But I think you have missed some configuration parameters.
>>>> " fs.default.name=s3://<<mybucket> " is the most important parameter
>>>> when you are using S3 to replace HDFS, but it is not enough. The detail
>>>> configuration can be obtained from here
>>>> http://wiki.apache.org/hadoop/AmazonS3
>>>>
>>>> Yanbo
>>>>
>>>>
>>>> 2012/7/23 Alok Kumar <al...@gmail.com>
>>>>
>>>>> Hello Group,
>>>>>
>>>>> I've hadoop setup locally running.
>>>>>
>>>>> Now I want to use Amazon s3://<mybucket> as my data store,
>>>>> so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
>>>>> hdfs-site.xml, Is it the correct way?
>>>>> I'm getting error :
>>>>>
>>>>> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid
>>>>> directory in dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
>>>>> 2012-07-23 13:15:06,260 ERROR
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
>>>>> dfs.data.dir are invalid.
>>>>>
>>>>> and
>>>>> when i changed like " dfs.data.dir=s3://<mybucket>/ "
>>>>> I got error :
>>>>> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>>> java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
>>>>> file:///
>>>>> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
>>>>> at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>>>>> at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
>>>>> at
>>>>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>>>>> at
>>>>> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
>>>>> at
>>>>> org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>>>
>>>>> Also,
>>>>> When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not
>>>>> coming up with error : ERROR
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
>>>>> (Any way I want to run namenode locally, so I reverted it back to
>>>>> hdfs://localhost:9000 )
>>>>>
>>>>> Your help is highly appreciated!
>>>>> Thanks
>>>>> --
>>>>> Alok Kumar
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Alok Kumar
>>>
>>>
>>>
>>
>
>
> --
> Alok Kumar
>
>
>
Re: problem configuring hadoop with s3 bucket
Posted by Alok Kumar <al...@gmail.com>.
Hi Yanbo,
I'm using hadoop-1.0.1 on local machine (not EC2)
conf/master : localhost
conf/slave : localhost
conf/core-site.xml :
<property>
<name>fs.default.name</name>
<value>s3://bucketname</value>
</property>
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>***********</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>**************</value>
</property>
/etc/hosts :
192.168.2.24 DRPM4 # Added by NetworkManager
127.0.0.1 localhost.localdomain localhost
$ bin/hadoop namenode -format : successful
$ bin/start-dfs.sh
Namenode log :
2012-07-25 13:19:32,896 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = DRPM4/192.168.2.24
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.1
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012
************************************************************/
2012-07-25 13:19:33,037 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2012-07-25 13:19:33,049 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2012-07-25 13:19:33,050 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2012-07-25 13:19:33,050 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started
2012-07-25 13:19:33,889 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
registered.
2012-07-25 13:19:33,896 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2012-07-25 13:19:33,904 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
registered.
2012-07-25 13:19:33,906 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
NameNode registered.
2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: VM
type = 32-bit
2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
memory = 17.77875 MB
2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet:
capacity = 2^22 = 4194304 entries
2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet:
recommended=4194304, actual=4194304
2012-07-25 13:19:33,986 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=alok
2012-07-25 13:19:33,986 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2012-07-25 13:19:33,986 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=true
2012-07-25 13:19:33,995 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.block.invalidate.limit=100
2012-07-25 13:19:33,995 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
accessTokenLifetime=0 min(s)
2012-07-25 13:19:34,396 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStateMBean and NameNodeMXBean
2012-07-25 13:19:34,416 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2012-07-25 13:19:34,442 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files = 1
2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
Number of files under construction = 0
2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 110 loaded in 0 seconds.
2012-07-25 13:19:34,449 INFO org.apache.hadoop.hdfs.server.common.Storage:
Edits file /home/alok/work/s3s/hadoop/namenode/current/edits of size 4
edits # 0 loaded in 0 seconds.
2012-07-25 13:19:34,451 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 110 saved in 0 seconds.
2012-07-25 13:19:35,161 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 110 saved in 0 seconds.
2012-07-25 13:19:35,873 INFO
org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
entries 0 lookups
2012-07-25 13:19:35,873 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
FSImage in 1900 msecs
2012-07-25 13:19:35,888 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks
= 0
2012-07-25 13:19:35,888 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid
blocks = 0
2012-07-25 13:19:35,888 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
under-replicated blocks = 0
2012-07-25 13:19:35,888 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
over-replicated blocks = 0
2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Safe mode termination scan for invalid, over- and under-replicated blocks
completed in 15 msec
2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Leaving safe mode after 1 secs.
2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Network topology has 0 racks and 0 datanodes
2012-07-25 13:19:35,889 INFO org.apache.hadoop.hdfs.StateChange: STATE*
UnderReplicatedBlocks has 0 blocks
2012-07-25 13:19:35,897 INFO org.apache.hadoop.util.HostsFileReader:
Refreshing hosts (include/exclude) list
2012-07-25 13:19:35,905 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue
QueueProcessingStatistics: First cycle completed 0 blocks in 7 msec
2012-07-25 13:19:35,905 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicateQueue
QueueProcessingStatistics: Queue flush completed 0 blocks in 7 msec
processing time, 7 msec clock time, 1 cycles
2012-07-25 13:19:35,905 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
FSNamesystemMetrics registered.
2012-07-25 13:19:35,906 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue
QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec
2012-07-25 13:19:35,906 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: InvalidateQueue
QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec
processing time, 1 msec clock time, 1 cycles
2012-07-25 13:19:35,916 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
transactions: 0 Total time for transactions(ms): 0Number of transactions
batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
2012-07-25 13:19:35,916 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicationMonitor
thread received InterruptedException.java.lang.InterruptedException: sleep
interrupted
2012-07-25 13:19:35,917 INFO
org.apache.hadoop.hdfs.server.namenode.DecommissionManager: Interrupted
Monitor
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
at java.lang.Thread.run(Thread.java:662)
2012-07-25 13:19:35,993 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
Problem binding to *bucket*/67.215.65.132:8020 : Cannot assign requested
address
at org.apache.hadoop.ipc.Server.bind(Server.java:227)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:301)
at org.apache.hadoop.ipc.Server.<init>(Server.java:1483)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:545)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:506)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:294)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at org.apache.hadoop.ipc.Server.bind(Server.java:225)
... 8 more
2012-07-25 13:19:35,994 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at DRPM4/192.168.2.24
************************************************************/
Thanks,
On Wed, Jul 25, 2012 at 12:05 PM, Yanbo Liang <ya...@gmail.com> wrote:
> Could you provide your execute environment, operation procedure and the
> detail log information?
>
>
> 2012/7/24 Alok Kumar <al...@gmail.com>
>
>> Hi Yanbo,
>>
>> Thank you for your reply..
>>
>> Now I've made changes exactly , present in this link
>> http://wiki.apache.org/hadoop/AmazonS3
>>
>> But my namenode is not coming up with exception (I tried it both locally
>> and inside EC2)
>>
>> ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
>> java.net.BindException: Problem binding to <bucket-name>.
>> s3.amazonaws.com/207.171.163.14:8020 : Cannot assign requested address
>>
>> fs.default.name = s3://<mybucket> // Error : UnknownHost Exception in
>> Namenode!
>> or
>> fs.default.name = s3://<mybucket>.s3.amazonaws.com // Error
>> BindException in Namenode log!
>>
>> Also I can't see any dir created inside my bucket. ( I'm able to execute
>> command $ bin/hadoop dfs -ls s3://<bucket>/ )
>>
>> "bin/hadoop namenode -format " is saying succesfully formatted namenode
>> dir S3://bucket/hadoop/namenode , when it is not even existing there!
>>
>>
>> any suggestion?
>>
>> Thanks again.
>>
>>
>> On Tue, Jul 24, 2012 at 4:11 PM, Yanbo Liang <ya...@gmail.com>wrote:
>>
>>> I think you have made confusion about the integration of hadoop and S3.
>>> 1) If you set "dfs.data.dir=s3://******", it means that you set S3 as
>>> the DataNode local storage.
>>> You are still using HDFS as the under storage layer.
>>> As far as I know, it is not supported at present.
>>> 2) The right way of using S3 integrated with Hadoop is to replace the
>>> HDFS with S3 just like you have tried.
>>> But I think you have missed some configuration parameters.
>>> " fs.default.name=s3://<<mybucket> " is the most important parameter
>>> when you are using S3 to replace HDFS, but it is not enough. The detail
>>> configuration can be obtained from here
>>> http://wiki.apache.org/hadoop/AmazonS3
>>>
>>> Yanbo
>>>
>>>
>>> 2012/7/23 Alok Kumar <al...@gmail.com>
>>>
>>>> Hello Group,
>>>>
>>>> I've hadoop setup locally running.
>>>>
>>>> Now I want to use Amazon s3://<mybucket> as my data store,
>>>> so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
>>>> hdfs-site.xml, Is it the correct way?
>>>> I'm getting error :
>>>>
>>>> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory
>>>> in dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
>>>> 2012-07-23 13:15:06,260 ERROR
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
>>>> dfs.data.dir are invalid.
>>>>
>>>> and
>>>> when i changed like " dfs.data.dir=s3://<mybucket>/ "
>>>> I got error :
>>>> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>> java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
>>>> file:///
>>>> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
>>>> at
>>>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>>>> at
>>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
>>>> at
>>>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>>>> at
>>>> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
>>>> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>>
>>>> Also,
>>>> When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not
>>>> coming up with error : ERROR
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
>>>> (Any way I want to run namenode locally, so I reverted it back to
>>>> hdfs://localhost:9000 )
>>>>
>>>> Your help is highly appreciated!
>>>> Thanks
>>>> --
>>>> Alok Kumar
>>>>
>>>
>>>
>>
>>
>> --
>> Alok Kumar
>>
>>
>>
>
--
Alok Kumar
Re: problem configuring hadoop with s3 bucket
Posted by Yanbo Liang <ya...@gmail.com>.
Could you provide your execute environment, operation procedure and the
detail log information?
2012/7/24 Alok Kumar <al...@gmail.com>
> Hi Yanbo,
>
> Thank you for your reply..
>
> Now I've made changes exactly , present in this link
> http://wiki.apache.org/hadoop/AmazonS3
>
> But my namenode is not coming up with exception (I tried it both locally
> and inside EC2)
>
> ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.net.BindException: Problem binding to <bucket-name>.
> s3.amazonaws.com/207.171.163.14:8020 : Cannot assign requested address
>
> fs.default.name = s3://<mybucket> // Error : UnknownHost Exception in
> Namenode!
> or
> fs.default.name = s3://<mybucket>.s3.amazonaws.com // Error BindException
> in Namenode log!
>
> Also I can't see any dir created inside my bucket. ( I'm able to execute
> command $ bin/hadoop dfs -ls s3://<bucket>/ )
>
> "bin/hadoop namenode -format " is saying succesfully formatted namenode
> dir S3://bucket/hadoop/namenode , when it is not even existing there!
>
>
> any suggestion?
>
> Thanks again.
>
>
> On Tue, Jul 24, 2012 at 4:11 PM, Yanbo Liang <ya...@gmail.com> wrote:
>
>> I think you have made confusion about the integration of hadoop and S3.
>> 1) If you set "dfs.data.dir=s3://******", it means that you set S3 as the
>> DataNode local storage.
>> You are still using HDFS as the under storage layer.
>> As far as I know, it is not supported at present.
>> 2) The right way of using S3 integrated with Hadoop is to replace the
>> HDFS with S3 just like you have tried.
>> But I think you have missed some configuration parameters.
>> " fs.default.name=s3://<<mybucket> " is the most important parameter
>> when you are using S3 to replace HDFS, but it is not enough. The detail
>> configuration can be obtained from here
>> http://wiki.apache.org/hadoop/AmazonS3
>>
>> Yanbo
>>
>>
>> 2012/7/23 Alok Kumar <al...@gmail.com>
>>
>>> Hello Group,
>>>
>>> I've hadoop setup locally running.
>>>
>>> Now I want to use Amazon s3://<mybucket> as my data store,
>>> so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
>>> hdfs-site.xml, Is it the correct way?
>>> I'm getting error :
>>>
>>> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory
>>> in dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
>>> 2012-07-23 13:15:06,260 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
>>> dfs.data.dir are invalid.
>>>
>>> and
>>> when i changed like " dfs.data.dir=s3://<mybucket>/ "
>>> I got error :
>>> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
>>> file:///
>>> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
>>> at
>>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>>> at
>>> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
>>> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> Also,
>>> When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not
>>> coming up with error : ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
>>> (Any way I want to run namenode locally, so I reverted it back to
>>> hdfs://localhost:9000 )
>>>
>>> Your help is highly appreciated!
>>> Thanks
>>> --
>>> Alok Kumar
>>>
>>
>>
>
>
> --
> Alok Kumar
>
>
>
Re: problem configuring hadoop with s3 bucket
Posted by Alok Kumar <al...@gmail.com>.
Hi Yanbo,
Thank you for your reply..
Now I've made changes exactly , present in this link
http://wiki.apache.org/hadoop/AmazonS3
But my namenode is not coming up with exception (I tried it both locally
and inside EC2)
ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
java.net.BindException: Problem binding to <bucket-name>.
s3.amazonaws.com/207.171.163.14:8020 : Cannot assign requested address
fs.default.name = s3://<mybucket> // Error : UnknownHost Exception in
Namenode!
or
fs.default.name = s3://<mybucket>.s3.amazonaws.com // Error BindException
in Namenode log!
Also I can't see any dir created inside my bucket. ( I'm able to execute
command $ bin/hadoop dfs -ls s3://<bucket>/ )
"bin/hadoop namenode -format " is saying succesfully formatted namenode dir
S3://bucket/hadoop/namenode , when it is not even existing there!
any suggestion?
Thanks again.
On Tue, Jul 24, 2012 at 4:11 PM, Yanbo Liang <ya...@gmail.com> wrote:
> I think you have made confusion about the integration of hadoop and S3.
> 1) If you set "dfs.data.dir=s3://******", it means that you set S3 as the
> DataNode local storage.
> You are still using HDFS as the under storage layer.
> As far as I know, it is not supported at present.
> 2) The right way of using S3 integrated with Hadoop is to replace the
> HDFS with S3 just like you have tried.
> But I think you have missed some configuration parameters.
> " fs.default.name=s3://<<mybucket> " is the most important parameter
> when you are using S3 to replace HDFS, but it is not enough. The detail
> configuration can be obtained from here
> http://wiki.apache.org/hadoop/AmazonS3
>
> Yanbo
>
>
> 2012/7/23 Alok Kumar <al...@gmail.com>
>
>> Hello Group,
>>
>> I've hadoop setup locally running.
>>
>> Now I want to use Amazon s3://<mybucket> as my data store,
>> so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
>> hdfs-site.xml, Is it the correct way?
>> I'm getting error :
>>
>> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory
>> in dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
>> 2012-07-23 13:15:06,260 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
>> dfs.data.dir are invalid.
>>
>> and
>> when i changed like " dfs.data.dir=s3://<mybucket>/ "
>> I got error :
>> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
>> java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
>> file:///
>> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
>> at
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> at
>> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
>> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> Also,
>> When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not
>> coming up with error : ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
>> (Any way I want to run namenode locally, so I reverted it back to
>> hdfs://localhost:9000 )
>>
>> Your help is highly appreciated!
>> Thanks
>> --
>> Alok Kumar
>>
>
>
--
Alok Kumar
Re: problem configuring hadoop with s3 bucket
Posted by Yanbo Liang <ya...@gmail.com>.
I think you have made confusion about the integration of hadoop and S3.
1) If you set "dfs.data.dir=s3://******", it means that you set S3 as the
DataNode local storage.
You are still using HDFS as the under storage layer.
As far as I know, it is not supported at present.
2) The right way of using S3 integrated with Hadoop is to replace the HDFS
with S3 just like you have tried.
But I think you have missed some configuration parameters.
" fs.default.name=s3://<<mybucket> " is the most important parameter when
you are using S3 to replace HDFS, but it is not enough. The detail
configuration can be obtained from here
http://wiki.apache.org/hadoop/AmazonS3
Yanbo
2012/7/23 Alok Kumar <al...@gmail.com>
> Hello Group,
>
> I've hadoop setup locally running.
>
> Now I want to use Amazon s3://<mybucket> as my data store,
> so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
> hdfs-site.xml, Is it the correct way?
> I'm getting error :
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
> dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
> 2012-07-23 13:15:06,260 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
> dfs.data.dir are invalid.
>
> and
> when i changed like " dfs.data.dir=s3://<mybucket>/ "
> I got error :
> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
> file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
> at
> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> Also,
> When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not
> coming up with error : ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.net.BindException:
> (Any way I want to run namenode locally, so I reverted it back to
> hdfs://localhost:9000 )
>
> Your help is highly appreciated!
> Thanks
> --
> Alok Kumar
>