You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Amandeep Khurana <am...@gmail.com> on 2009/02/15 20:57:26 UTC
HDFS architecture based on GFS?
Hi
Is the HDFS architecture completely based on the Google Filesystem? If it
isnt, what are the differences between the two?
Secondly, is the coupling between Hadoop and HDFS same as how it is between
the Google's version of Map Reduce and GFS?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
As far as I know, datanodes just know the block ID, and the namenode knows
which file this belongs to.
On Mon, Feb 16, 2009 at 4:54 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Ok. Thanks..
>
> Another question now. Do the datanodes have any way of linking a particular
> block of data to a global file identifier?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 9:37 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > In general, yeah, the scripts can access any resource they want (within
> the
> > permissions of the user that the task runs as). It's also possible to
> > access
> > HDFS from scripts because HDFS provides a FUSE interface that can make it
> > look like a regular file system on the machine. (The FUSE module in turn
> > talks to the namenode as a regular HDFS client.)
> >
> > On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > I dont know much about Hadoop streaming and have a quick question here.
> > >
> > > The snippets of code/programs that you attach into the map reduce job
> > might
> > > want to access outside resources (like you mentioned). Now these might
> > not
> > > need to go to the namenode right? For example a python script. How
> would
> > it
> > > access the data? Would it ask the parent java process (in the
> > tasktracker)
> > > to get the data or would it go and do stuff on its own?
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > > > Nope, typically the JobTracker just starts the process, and the
> > > tasktracker
> > > > talks directly to the namenode to get a pointer to the datanode, and
> > then
> > > > directly to the datanode.
> > > >
> > > > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Alright.. Got it.
> > > > >
> > > > > Now, do the task trackers talk to the namenode and the data node
> > > directly
> > > > > or
> > > > > do they go through the job tracker for it? So, if my code is such
> > that
> > > I
> > > > > need to access more files from the hdfs, would the job tracker get
> > > > involved
> > > > > or not?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <matei@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > > > Normally, HDFS files are accessed through the namenode. If there
> > was
> > > a
> > > > > > malicious process though, then I imagine it could talk to a
> > datanode
> > > > > > directly and request a specific block.
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Ok. Got it.
> > > > > > >
> > > > > > > Now, when my job needs to access another file, does it go to
> the
> > > > > Namenode
> > > > > > > to
> > > > > > > get the block ids? How does the java process know where the
> files
> > > are
> > > > > and
> > > > > > > how to access them?
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <
> > matei@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > I mentioned this case because even jobs written in Java can
> use
> > > the
> > > > > > HDFS
> > > > > > > > API
> > > > > > > > to talk to the NameNode and access the filesystem. People
> often
> > > do
> > > > > this
> > > > > > > > because their job needs to read a config file, some small
> data
> > > > table,
> > > > > > etc
> > > > > > > > and use this information in its map or reduce functions. In
> > this
> > > > > case,
> > > > > > > you
> > > > > > > > open the second file separately in your mapper's init
> function
> > > and
> > > > > read
> > > > > > > > whatever you need from it. In general I wanted to point out
> > that
> > > > you
> > > > > > > can't
> > > > > > > > know which files a job will access unless you look at its
> > source
> > > > code
> > > > > > or
> > > > > > > > monitor the calls it makes; the input file(s) you provide in
> > the
> > > > job
> > > > > > > > description are a hint to the MapReduce framework to place
> your
> > > job
> > > > > on
> > > > > > > > certain nodes, but it's reasonable for the job to access
> other
> > > > files
> > > > > as
> > > > > > > > well.
> > > > > > > >
> > > > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> > > > amansk@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Another question that I have here - When the jobs run
> > arbitrary
> > > > > code
> > > > > > > and
> > > > > > > > > access data from the HDFS, do they go to the namenode to
> get
> > > the
> > > > > > block
> > > > > > > > > information?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Amandeep Khurana
> > > > > > > > > Computer Science Graduate Student
> > > > > > > > > University of California, Santa Cruz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > > > > amansk@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Assuming that the job is purely in Java and not involving
> > > > > streaming
> > > > > > > or
> > > > > > > > > > pipes, wouldnt the resources (files) required by the job
> as
> > > > > inputs
> > > > > > be
> > > > > > > > > known
> > > > > > > > > > beforehand? So, if the map task is accessing a second
> file,
> > > how
> > > > > > does
> > > > > > > it
> > > > > > > > > make
> > > > > > > > > > it different except that there are multiple files. The
> > > > JobTracker
> > > > > > > would
> > > > > > > > > know
> > > > > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > > > > >
> > > > > > > > > > I am slightly confused why you have mentioned this case
> > > > > > separately...
> > > > > > > > Can
> > > > > > > > > > you elaborate on it a little bit?
> > > > > > > > > >
> > > > > > > > > > Amandeep
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Amandeep Khurana
> > > > > > > > > > Computer Science Graduate Student
> > > > > > > > > > University of California, Santa Cruz
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > > > > matei@cloudera.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Typically the data flow is like this:1) Client submits a
> > job
> > > > > > > > description
> > > > > > > > > >> to
> > > > > > > > > >> the JobTracker.
> > > > > > > > > >> 2) JobTracker figures out block locations for the input
> > > > file(s)
> > > > > by
> > > > > > > > > talking
> > > > > > > > > >> to HDFS NameNode.
> > > > > > > > > >> 3) JobTracker creates a job description file in HDFS
> which
> > > > will
> > > > > be
> > > > > > > > read
> > > > > > > > > by
> > > > > > > > > >> the nodes to copy over the job's code etc.
> > > > > > > > > >> 4) JobTracker starts map tasks on the slaves
> > (TaskTrackers)
> > > > with
> > > > > > the
> > > > > > > > > >> appropriate data blocks.
> > > > > > > > > >> 5) After running, maps create intermediate output files
> on
> > > > those
> > > > > > > > slaves.
> > > > > > > > > >> These are not in HDFS, they're in some temporary storage
> > > used
> > > > by
> > > > > > > > > >> MapReduce.
> > > > > > > > > >> 6) JobTracker starts reduces on a series of slaves,
> which
> > > copy
> > > > > > over
> > > > > > > > the
> > > > > > > > > >> appropriate map outputs, apply the reduce function, and
> > > write
> > > > > the
> > > > > > > > > outputs
> > > > > > > > > >> to
> > > > > > > > > >> HDFS (one output file per reducer).
> > > > > > > > > >> 7) Some logs for the job may also be put into HDFS by
> the
> > > > > > > JobTracker.
> > > > > > > > > >>
> > > > > > > > > >> However, there is a big caveat, which is that the map
> and
> > > > reduce
> > > > > > > tasks
> > > > > > > > > run
> > > > > > > > > >> arbitrary code. It is not unusual to have a map that
> opens
> > a
> > > > > > second
> > > > > > > > HDFS
> > > > > > > > > >> file to read some information (e.g. for doing a join of
> a
> > > > small
> > > > > > > table
> > > > > > > > > >> against a big file). If you use Hadoop Streaming or
> Pipes
> > to
> > > > > write
> > > > > > a
> > > > > > > > job
> > > > > > > > > >> in
> > > > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> > > > processes
> > > > > > > which
> > > > > > > > > may
> > > > > > > > > >> also access external resources in this manner. Some
> people
> > > > also
> > > > > > > > > read/write
> > > > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> > > security
> > > > > > > > solution
> > > > > > > > > >> would ideally deal with these cases too.
> > > > > > > > > >>
> > > > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > > > > amansk@gmail.com
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > A quick question here. How does a typical hadoop job
> > work
> > > at
> > > > > the
> > > > > > > > > system
> > > > > > > > > >> > level? What are the various interactions and how does
> > the
> > > > data
> > > > > > > flow?
> > > > > > > > > >> >
> > > > > > > > > >> > Amandeep
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Amandeep Khurana
> > > > > > > > > >> > Computer Science Graduate Student
> > > > > > > > > >> > University of California, Santa Cruz
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > > > > amansk@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Thanks Matei. If the basic architecture is similar
> to
> > > the
> > > > > > Google
> > > > > > > > > >> stuff, I
> > > > > > > > > >> > > can safely just work on the project using the
> > > information
> > > > > from
> > > > > > > the
> > > > > > > > > >> > papers.
> > > > > > > > > >> > >
> > > > > > > > > >> > > I am aware of the 4487 jira and the current status
> of
> > > the
> > > > > > > > > permissions
> > > > > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Cheers
> > > > > > > > > >> > > Amandeep
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > Amandeep Khurana
> > > > > > > > > >> > > Computer Science Graduate Student
> > > > > > > > > >> > > University of California, Santa Cruz
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > > > > matei@cloudera.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > >> Forgot to add, this JIRA details the latest
> security
> > > > > features
> > > > > > > > that
> > > > > > > > > >> are
> > > > > > > > > >> > >> being
> > > > > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > > > > >> > >> This document describes the current status and
> > > > limitations
> > > > > of
> > > > > > > the
> > > > > > > > > >> > >> permissions mechanism:
> > > > > > > > > >> > >>
> > > > > > > > > >>
> > > > > > > >
> > > > >
> > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > > > > .
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > > > > matei@cloudera.com
> > > > > > > > > >
> > > > > > > > > >> > >> wrote:
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> > I think it's safe to assume that Hadoop works
> like
> > > > > > > > MapReduce/GFS
> > > > > > > > > at
> > > > > > > > > >> > the
> > > > > > > > > >> > >> > level described in those papers. In particular,
> in
> > > > HDFS,
> > > > > > > there
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > >> > >> master
> > > > > > > > > >> > >> > node containing metadata and a number of slave
> > nodes
> > > > > > > > (datanodes)
> > > > > > > > > >> > >> containing
> > > > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to
> the
> > > > master
> > > > > > to
> > > > > > > > list
> > > > > > > > > >> > >> > directories, etc. When they want to read a region
> > of
> > > > some
> > > > > > > file,
> > > > > > > > > >> they
> > > > > > > > > >> > >> tell
> > > > > > > > > >> > >> > the master the filename and offset, and they
> > receive
> > > a
> > > > > list
> > > > > > > of
> > > > > > > > > >> block
> > > > > > > > > >> > >> > locations (datanodes). They then contact the
> > > individual
> > > > > > > > datanodes
> > > > > > > > > >> to
> > > > > > > > > >> > >> read
> > > > > > > > > >> > >> > the blocks. When clients write a file, they first
> > > > obtain
> > > > > a
> > > > > > > new
> > > > > > > > > >> block
> > > > > > > > > >> > ID
> > > > > > > > > >> > >> and
> > > > > > > > > >> > >> > list of nodes to write it to from the master,
> then
> > > > > contact
> > > > > > > the
> > > > > > > > > >> > datanodes
> > > > > > > > > >> > >> to
> > > > > > > > > >> > >> > write it (actually, the datanodes pipeline the
> > write
> > > as
> > > > > in
> > > > > > > GFS)
> > > > > > > > > and
> > > > > > > > > >> > >> report
> > > > > > > > > >> > >> > when the write is complete. HDFS actually has
> some
> > > > > security
> > > > > > > > > >> mechanisms
> > > > > > > > > >> > >> built
> > > > > > > > > >> > >> > in, authenticating users based on their Unix ID
> and
> > > > > > providing
> > > > > > > > > >> > Unix-like
> > > > > > > > > >> > >> file
> > > > > > > > > >> > >> > permissions. I don't know much about how these
> are
> > > > > > > implemented,
> > > > > > > > > but
> > > > > > > > > >> > they
> > > > > > > > > >> > >> > would be a good place to start looking.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana
> <
> > > > > > > > > >> amansk@gmail.com
> > > > > > > > > >> > >> >wrote:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >> Thanks Matie
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> I had gone through the architecture document
> > online.
> > > I
> > > > > am
> > > > > > > > > >> currently
> > > > > > > > > >> > >> >> working
> > > > > > > > > >> > >> >> on a project towards Security in Hadoop. I do
> know
> > > how
> > > > > the
> > > > > > > > data
> > > > > > > > > >> moves
> > > > > > > > > >> > >> >> around
> > > > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does
> > HDFS
> > > > > > follow
> > > > > > > > and
> > > > > > > > > >> how
> > > > > > > > > >> > >> >> different it is from GFS. Can you throw some
> light
> > > on
> > > > > > that?
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > > > > following
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > >> > >> >> protocols. Thats why the question about how does
> > the
> > > > > > Hadoop
> > > > > > > > > >> framework
> > > > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> > > from
> > > > > Map
> > > > > > > > Reduce
> > > > > > > > > >> and
> > > > > > > > > >> > >> GFS.
> > > > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> > > information
> > > > on
> > > > > > how
> > > > > > > > > those
> > > > > > > > > >> > >> systems
> > > > > > > > > >> > >> >> are designed but there is nothing that concrete
> > for
> > > > > Hadoop
> > > > > > > > that
> > > > > > > > > I
> > > > > > > > > >> > have
> > > > > > > > > >> > >> >> been
> > > > > > > > > >> > >> >> able to find.
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> Amandeep
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> Amandeep Khurana
> > > > > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia
> <
> > > > > > > > > >> matei@cloudera.com>
> > > > > > > > > >> > >> >> wrote:
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> > Hi Amandeep,
> > > > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS
> > and
> > > > > aims
> > > > > > to
> > > > > > > > > >> provide
> > > > > > > > > >> > >> those
> > > > > > > > > >> > >> >> > capabilities as an open-source project. HDFS
> is
> > > > > similar
> > > > > > to
> > > > > > > > GFS
> > > > > > > > > >> > (large
> > > > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> > > > missing
> > > > > > are
> > > > > > > > > >> > read-write
> > > > > > > > > >> > >> >> > support in the middle of a file (unlikely to
> be
> > > > > provided
> > > > > > > > > because
> > > > > > > > > >> > few
> > > > > > > > > >> > >> >> Hadoop
> > > > > > > > > >> > >> >> > applications require it) and multiple
> appenders
> > > (the
> > > > > > > record
> > > > > > > > > >> append
> > > > > > > > > >> > >> >> > operation). You can read about HDFS
> architecture
> > > at
> > > > > > > > > >> > >> >> >
> > > > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > > > > .
> > > > > > > > > >> The
> > > > > > > > > >> > >> >> MapReduce
> > > > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same
> > way
> > > > > that
> > > > > > > > > Google's
> > > > > > > > > >> > >> >> MapReduce
> > > > > > > > > >> > >> >> > interacts with GFS (shipping computation to
> the
> > > > data),
> > > > > > > > > although
> > > > > > > > > >> > >> Hadoop
> > > > > > > > > >> > >> >> > MapReduce also supports running over other
> > > > distributed
> > > > > > > > > >> filesystems.
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >> > Matei
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep
> > Khurana
> > > <
> > > > > > > > > >> > amansk@gmail.com
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >> > wrote:
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >> > > Hi
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Is the HDFS architecture completely based on
> > the
> > > > > > Google
> > > > > > > > > >> > Filesystem?
> > > > > > > > > >> > >> If
> > > > > > > > > >> > >> >> it
> > > > > > > > > >> > >> >> > > isnt, what are the differences between the
> > two?
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> > > HDFS
> > > > > same
> > > > > > > as
> > > > > > > > > how
> > > > > > > > > >> it
> > > > > > > > > >> > is
> > > > > > > > > >> > >> >> > between
> > > > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Amandeep
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >>
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
As far as I know, datanodes just know the block ID, and the namenode knows
which file this belongs to.
On Mon, Feb 16, 2009 at 4:54 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Ok. Thanks..
>
> Another question now. Do the datanodes have any way of linking a particular
> block of data to a global file identifier?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 9:37 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > In general, yeah, the scripts can access any resource they want (within
> the
> > permissions of the user that the task runs as). It's also possible to
> > access
> > HDFS from scripts because HDFS provides a FUSE interface that can make it
> > look like a regular file system on the machine. (The FUSE module in turn
> > talks to the namenode as a regular HDFS client.)
> >
> > On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > I dont know much about Hadoop streaming and have a quick question here.
> > >
> > > The snippets of code/programs that you attach into the map reduce job
> > might
> > > want to access outside resources (like you mentioned). Now these might
> > not
> > > need to go to the namenode right? For example a python script. How
> would
> > it
> > > access the data? Would it ask the parent java process (in the
> > tasktracker)
> > > to get the data or would it go and do stuff on its own?
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > > > Nope, typically the JobTracker just starts the process, and the
> > > tasktracker
> > > > talks directly to the namenode to get a pointer to the datanode, and
> > then
> > > > directly to the datanode.
> > > >
> > > > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Alright.. Got it.
> > > > >
> > > > > Now, do the task trackers talk to the namenode and the data node
> > > directly
> > > > > or
> > > > > do they go through the job tracker for it? So, if my code is such
> > that
> > > I
> > > > > need to access more files from the hdfs, would the job tracker get
> > > > involved
> > > > > or not?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <matei@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > > > Normally, HDFS files are accessed through the namenode. If there
> > was
> > > a
> > > > > > malicious process though, then I imagine it could talk to a
> > datanode
> > > > > > directly and request a specific block.
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Ok. Got it.
> > > > > > >
> > > > > > > Now, when my job needs to access another file, does it go to
> the
> > > > > Namenode
> > > > > > > to
> > > > > > > get the block ids? How does the java process know where the
> files
> > > are
> > > > > and
> > > > > > > how to access them?
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <
> > matei@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > I mentioned this case because even jobs written in Java can
> use
> > > the
> > > > > > HDFS
> > > > > > > > API
> > > > > > > > to talk to the NameNode and access the filesystem. People
> often
> > > do
> > > > > this
> > > > > > > > because their job needs to read a config file, some small
> data
> > > > table,
> > > > > > etc
> > > > > > > > and use this information in its map or reduce functions. In
> > this
> > > > > case,
> > > > > > > you
> > > > > > > > open the second file separately in your mapper's init
> function
> > > and
> > > > > read
> > > > > > > > whatever you need from it. In general I wanted to point out
> > that
> > > > you
> > > > > > > can't
> > > > > > > > know which files a job will access unless you look at its
> > source
> > > > code
> > > > > > or
> > > > > > > > monitor the calls it makes; the input file(s) you provide in
> > the
> > > > job
> > > > > > > > description are a hint to the MapReduce framework to place
> your
> > > job
> > > > > on
> > > > > > > > certain nodes, but it's reasonable for the job to access
> other
> > > > files
> > > > > as
> > > > > > > > well.
> > > > > > > >
> > > > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> > > > amansk@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Another question that I have here - When the jobs run
> > arbitrary
> > > > > code
> > > > > > > and
> > > > > > > > > access data from the HDFS, do they go to the namenode to
> get
> > > the
> > > > > > block
> > > > > > > > > information?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Amandeep Khurana
> > > > > > > > > Computer Science Graduate Student
> > > > > > > > > University of California, Santa Cruz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > > > > amansk@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Assuming that the job is purely in Java and not involving
> > > > > streaming
> > > > > > > or
> > > > > > > > > > pipes, wouldnt the resources (files) required by the job
> as
> > > > > inputs
> > > > > > be
> > > > > > > > > known
> > > > > > > > > > beforehand? So, if the map task is accessing a second
> file,
> > > how
> > > > > > does
> > > > > > > it
> > > > > > > > > make
> > > > > > > > > > it different except that there are multiple files. The
> > > > JobTracker
> > > > > > > would
> > > > > > > > > know
> > > > > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > > > > >
> > > > > > > > > > I am slightly confused why you have mentioned this case
> > > > > > separately...
> > > > > > > > Can
> > > > > > > > > > you elaborate on it a little bit?
> > > > > > > > > >
> > > > > > > > > > Amandeep
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Amandeep Khurana
> > > > > > > > > > Computer Science Graduate Student
> > > > > > > > > > University of California, Santa Cruz
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > > > > matei@cloudera.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Typically the data flow is like this:1) Client submits a
> > job
> > > > > > > > description
> > > > > > > > > >> to
> > > > > > > > > >> the JobTracker.
> > > > > > > > > >> 2) JobTracker figures out block locations for the input
> > > > file(s)
> > > > > by
> > > > > > > > > talking
> > > > > > > > > >> to HDFS NameNode.
> > > > > > > > > >> 3) JobTracker creates a job description file in HDFS
> which
> > > > will
> > > > > be
> > > > > > > > read
> > > > > > > > > by
> > > > > > > > > >> the nodes to copy over the job's code etc.
> > > > > > > > > >> 4) JobTracker starts map tasks on the slaves
> > (TaskTrackers)
> > > > with
> > > > > > the
> > > > > > > > > >> appropriate data blocks.
> > > > > > > > > >> 5) After running, maps create intermediate output files
> on
> > > > those
> > > > > > > > slaves.
> > > > > > > > > >> These are not in HDFS, they're in some temporary storage
> > > used
> > > > by
> > > > > > > > > >> MapReduce.
> > > > > > > > > >> 6) JobTracker starts reduces on a series of slaves,
> which
> > > copy
> > > > > > over
> > > > > > > > the
> > > > > > > > > >> appropriate map outputs, apply the reduce function, and
> > > write
> > > > > the
> > > > > > > > > outputs
> > > > > > > > > >> to
> > > > > > > > > >> HDFS (one output file per reducer).
> > > > > > > > > >> 7) Some logs for the job may also be put into HDFS by
> the
> > > > > > > JobTracker.
> > > > > > > > > >>
> > > > > > > > > >> However, there is a big caveat, which is that the map
> and
> > > > reduce
> > > > > > > tasks
> > > > > > > > > run
> > > > > > > > > >> arbitrary code. It is not unusual to have a map that
> opens
> > a
> > > > > > second
> > > > > > > > HDFS
> > > > > > > > > >> file to read some information (e.g. for doing a join of
> a
> > > > small
> > > > > > > table
> > > > > > > > > >> against a big file). If you use Hadoop Streaming or
> Pipes
> > to
> > > > > write
> > > > > > a
> > > > > > > > job
> > > > > > > > > >> in
> > > > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> > > > processes
> > > > > > > which
> > > > > > > > > may
> > > > > > > > > >> also access external resources in this manner. Some
> people
> > > > also
> > > > > > > > > read/write
> > > > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> > > security
> > > > > > > > solution
> > > > > > > > > >> would ideally deal with these cases too.
> > > > > > > > > >>
> > > > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > > > > amansk@gmail.com
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > A quick question here. How does a typical hadoop job
> > work
> > > at
> > > > > the
> > > > > > > > > system
> > > > > > > > > >> > level? What are the various interactions and how does
> > the
> > > > data
> > > > > > > flow?
> > > > > > > > > >> >
> > > > > > > > > >> > Amandeep
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Amandeep Khurana
> > > > > > > > > >> > Computer Science Graduate Student
> > > > > > > > > >> > University of California, Santa Cruz
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > > > > amansk@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Thanks Matei. If the basic architecture is similar
> to
> > > the
> > > > > > Google
> > > > > > > > > >> stuff, I
> > > > > > > > > >> > > can safely just work on the project using the
> > > information
> > > > > from
> > > > > > > the
> > > > > > > > > >> > papers.
> > > > > > > > > >> > >
> > > > > > > > > >> > > I am aware of the 4487 jira and the current status
> of
> > > the
> > > > > > > > > permissions
> > > > > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Cheers
> > > > > > > > > >> > > Amandeep
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > Amandeep Khurana
> > > > > > > > > >> > > Computer Science Graduate Student
> > > > > > > > > >> > > University of California, Santa Cruz
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > > > > matei@cloudera.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > >> Forgot to add, this JIRA details the latest
> security
> > > > > features
> > > > > > > > that
> > > > > > > > > >> are
> > > > > > > > > >> > >> being
> > > > > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > > > > >> > >> This document describes the current status and
> > > > limitations
> > > > > of
> > > > > > > the
> > > > > > > > > >> > >> permissions mechanism:
> > > > > > > > > >> > >>
> > > > > > > > > >>
> > > > > > > >
> > > > >
> > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > > > > .
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > > > > matei@cloudera.com
> > > > > > > > > >
> > > > > > > > > >> > >> wrote:
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> > I think it's safe to assume that Hadoop works
> like
> > > > > > > > MapReduce/GFS
> > > > > > > > > at
> > > > > > > > > >> > the
> > > > > > > > > >> > >> > level described in those papers. In particular,
> in
> > > > HDFS,
> > > > > > > there
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > >> > >> master
> > > > > > > > > >> > >> > node containing metadata and a number of slave
> > nodes
> > > > > > > > (datanodes)
> > > > > > > > > >> > >> containing
> > > > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to
> the
> > > > master
> > > > > > to
> > > > > > > > list
> > > > > > > > > >> > >> > directories, etc. When they want to read a region
> > of
> > > > some
> > > > > > > file,
> > > > > > > > > >> they
> > > > > > > > > >> > >> tell
> > > > > > > > > >> > >> > the master the filename and offset, and they
> > receive
> > > a
> > > > > list
> > > > > > > of
> > > > > > > > > >> block
> > > > > > > > > >> > >> > locations (datanodes). They then contact the
> > > individual
> > > > > > > > datanodes
> > > > > > > > > >> to
> > > > > > > > > >> > >> read
> > > > > > > > > >> > >> > the blocks. When clients write a file, they first
> > > > obtain
> > > > > a
> > > > > > > new
> > > > > > > > > >> block
> > > > > > > > > >> > ID
> > > > > > > > > >> > >> and
> > > > > > > > > >> > >> > list of nodes to write it to from the master,
> then
> > > > > contact
> > > > > > > the
> > > > > > > > > >> > datanodes
> > > > > > > > > >> > >> to
> > > > > > > > > >> > >> > write it (actually, the datanodes pipeline the
> > write
> > > as
> > > > > in
> > > > > > > GFS)
> > > > > > > > > and
> > > > > > > > > >> > >> report
> > > > > > > > > >> > >> > when the write is complete. HDFS actually has
> some
> > > > > security
> > > > > > > > > >> mechanisms
> > > > > > > > > >> > >> built
> > > > > > > > > >> > >> > in, authenticating users based on their Unix ID
> and
> > > > > > providing
> > > > > > > > > >> > Unix-like
> > > > > > > > > >> > >> file
> > > > > > > > > >> > >> > permissions. I don't know much about how these
> are
> > > > > > > implemented,
> > > > > > > > > but
> > > > > > > > > >> > they
> > > > > > > > > >> > >> > would be a good place to start looking.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana
> <
> > > > > > > > > >> amansk@gmail.com
> > > > > > > > > >> > >> >wrote:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >> Thanks Matie
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> I had gone through the architecture document
> > online.
> > > I
> > > > > am
> > > > > > > > > >> currently
> > > > > > > > > >> > >> >> working
> > > > > > > > > >> > >> >> on a project towards Security in Hadoop. I do
> know
> > > how
> > > > > the
> > > > > > > > data
> > > > > > > > > >> moves
> > > > > > > > > >> > >> >> around
> > > > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does
> > HDFS
> > > > > > follow
> > > > > > > > and
> > > > > > > > > >> how
> > > > > > > > > >> > >> >> different it is from GFS. Can you throw some
> light
> > > on
> > > > > > that?
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > > > > following
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > >> > >> >> protocols. Thats why the question about how does
> > the
> > > > > > Hadoop
> > > > > > > > > >> framework
> > > > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> > > from
> > > > > Map
> > > > > > > > Reduce
> > > > > > > > > >> and
> > > > > > > > > >> > >> GFS.
> > > > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> > > information
> > > > on
> > > > > > how
> > > > > > > > > those
> > > > > > > > > >> > >> systems
> > > > > > > > > >> > >> >> are designed but there is nothing that concrete
> > for
> > > > > Hadoop
> > > > > > > > that
> > > > > > > > > I
> > > > > > > > > >> > have
> > > > > > > > > >> > >> >> been
> > > > > > > > > >> > >> >> able to find.
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> Amandeep
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> Amandeep Khurana
> > > > > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia
> <
> > > > > > > > > >> matei@cloudera.com>
> > > > > > > > > >> > >> >> wrote:
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >> > Hi Amandeep,
> > > > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS
> > and
> > > > > aims
> > > > > > to
> > > > > > > > > >> provide
> > > > > > > > > >> > >> those
> > > > > > > > > >> > >> >> > capabilities as an open-source project. HDFS
> is
> > > > > similar
> > > > > > to
> > > > > > > > GFS
> > > > > > > > > >> > (large
> > > > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> > > > missing
> > > > > > are
> > > > > > > > > >> > read-write
> > > > > > > > > >> > >> >> > support in the middle of a file (unlikely to
> be
> > > > > provided
> > > > > > > > > because
> > > > > > > > > >> > few
> > > > > > > > > >> > >> >> Hadoop
> > > > > > > > > >> > >> >> > applications require it) and multiple
> appenders
> > > (the
> > > > > > > record
> > > > > > > > > >> append
> > > > > > > > > >> > >> >> > operation). You can read about HDFS
> architecture
> > > at
> > > > > > > > > >> > >> >> >
> > > > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > > > > .
> > > > > > > > > >> The
> > > > > > > > > >> > >> >> MapReduce
> > > > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same
> > way
> > > > > that
> > > > > > > > > Google's
> > > > > > > > > >> > >> >> MapReduce
> > > > > > > > > >> > >> >> > interacts with GFS (shipping computation to
> the
> > > > data),
> > > > > > > > > although
> > > > > > > > > >> > >> Hadoop
> > > > > > > > > >> > >> >> > MapReduce also supports running over other
> > > > distributed
> > > > > > > > > >> filesystems.
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >> > Matei
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep
> > Khurana
> > > <
> > > > > > > > > >> > amansk@gmail.com
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >> > wrote:
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >> > > Hi
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Is the HDFS architecture completely based on
> > the
> > > > > > Google
> > > > > > > > > >> > Filesystem?
> > > > > > > > > >> > >> If
> > > > > > > > > >> > >> >> it
> > > > > > > > > >> > >> >> > > isnt, what are the differences between the
> > two?
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> > > HDFS
> > > > > same
> > > > > > > as
> > > > > > > > > how
> > > > > > > > > >> it
> > > > > > > > > >> > is
> > > > > > > > > >> > >> >> > between
> > > > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Amandeep
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > > > > >> > >> >> > >
> > > > > > > > > >> > >> >> >
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >>
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Ok. Thanks..
Another question now. Do the datanodes have any way of linking a particular
block of data to a global file identifier?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 9:37 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> In general, yeah, the scripts can access any resource they want (within the
> permissions of the user that the task runs as). It's also possible to
> access
> HDFS from scripts because HDFS provides a FUSE interface that can make it
> look like a regular file system on the machine. (The FUSE module in turn
> talks to the namenode as a regular HDFS client.)
>
> On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > I dont know much about Hadoop streaming and have a quick question here.
> >
> > The snippets of code/programs that you attach into the map reduce job
> might
> > want to access outside resources (like you mentioned). Now these might
> not
> > need to go to the namenode right? For example a python script. How would
> it
> > access the data? Would it ask the parent java process (in the
> tasktracker)
> > to get the data or would it go and do stuff on its own?
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> > > Nope, typically the JobTracker just starts the process, and the
> > tasktracker
> > > talks directly to the namenode to get a pointer to the datanode, and
> then
> > > directly to the datanode.
> > >
> > > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Alright.. Got it.
> > > >
> > > > Now, do the task trackers talk to the namenode and the data node
> > directly
> > > > or
> > > > do they go through the job tracker for it? So, if my code is such
> that
> > I
> > > > need to access more files from the hdfs, would the job tracker get
> > > involved
> > > > or not?
> > > >
> > > >
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> > > wrote:
> > > >
> > > > > Normally, HDFS files are accessed through the namenode. If there
> was
> > a
> > > > > malicious process though, then I imagine it could talk to a
> datanode
> > > > > directly and request a specific block.
> > > > >
> > > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Ok. Got it.
> > > > > >
> > > > > > Now, when my job needs to access another file, does it go to the
> > > > Namenode
> > > > > > to
> > > > > > get the block ids? How does the java process know where the files
> > are
> > > > and
> > > > > > how to access them?
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <
> matei@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I mentioned this case because even jobs written in Java can use
> > the
> > > > > HDFS
> > > > > > > API
> > > > > > > to talk to the NameNode and access the filesystem. People often
> > do
> > > > this
> > > > > > > because their job needs to read a config file, some small data
> > > table,
> > > > > etc
> > > > > > > and use this information in its map or reduce functions. In
> this
> > > > case,
> > > > > > you
> > > > > > > open the second file separately in your mapper's init function
> > and
> > > > read
> > > > > > > whatever you need from it. In general I wanted to point out
> that
> > > you
> > > > > > can't
> > > > > > > know which files a job will access unless you look at its
> source
> > > code
> > > > > or
> > > > > > > monitor the calls it makes; the input file(s) you provide in
> the
> > > job
> > > > > > > description are a hint to the MapReduce framework to place your
> > job
> > > > on
> > > > > > > certain nodes, but it's reasonable for the job to access other
> > > files
> > > > as
> > > > > > > well.
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> > > amansk@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Another question that I have here - When the jobs run
> arbitrary
> > > > code
> > > > > > and
> > > > > > > > access data from the HDFS, do they go to the namenode to get
> > the
> > > > > block
> > > > > > > > information?
> > > > > > > >
> > > > > > > >
> > > > > > > > Amandeep Khurana
> > > > > > > > Computer Science Graduate Student
> > > > > > > > University of California, Santa Cruz
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > > > amansk@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Assuming that the job is purely in Java and not involving
> > > > streaming
> > > > > > or
> > > > > > > > > pipes, wouldnt the resources (files) required by the job as
> > > > inputs
> > > > > be
> > > > > > > > known
> > > > > > > > > beforehand? So, if the map task is accessing a second file,
> > how
> > > > > does
> > > > > > it
> > > > > > > > make
> > > > > > > > > it different except that there are multiple files. The
> > > JobTracker
> > > > > > would
> > > > > > > > know
> > > > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > > > >
> > > > > > > > > I am slightly confused why you have mentioned this case
> > > > > separately...
> > > > > > > Can
> > > > > > > > > you elaborate on it a little bit?
> > > > > > > > >
> > > > > > > > > Amandeep
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Amandeep Khurana
> > > > > > > > > Computer Science Graduate Student
> > > > > > > > > University of California, Santa Cruz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > > > matei@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Typically the data flow is like this:1) Client submits a
> job
> > > > > > > description
> > > > > > > > >> to
> > > > > > > > >> the JobTracker.
> > > > > > > > >> 2) JobTracker figures out block locations for the input
> > > file(s)
> > > > by
> > > > > > > > talking
> > > > > > > > >> to HDFS NameNode.
> > > > > > > > >> 3) JobTracker creates a job description file in HDFS which
> > > will
> > > > be
> > > > > > > read
> > > > > > > > by
> > > > > > > > >> the nodes to copy over the job's code etc.
> > > > > > > > >> 4) JobTracker starts map tasks on the slaves
> (TaskTrackers)
> > > with
> > > > > the
> > > > > > > > >> appropriate data blocks.
> > > > > > > > >> 5) After running, maps create intermediate output files on
> > > those
> > > > > > > slaves.
> > > > > > > > >> These are not in HDFS, they're in some temporary storage
> > used
> > > by
> > > > > > > > >> MapReduce.
> > > > > > > > >> 6) JobTracker starts reduces on a series of slaves, which
> > copy
> > > > > over
> > > > > > > the
> > > > > > > > >> appropriate map outputs, apply the reduce function, and
> > write
> > > > the
> > > > > > > > outputs
> > > > > > > > >> to
> > > > > > > > >> HDFS (one output file per reducer).
> > > > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > > > > JobTracker.
> > > > > > > > >>
> > > > > > > > >> However, there is a big caveat, which is that the map and
> > > reduce
> > > > > > tasks
> > > > > > > > run
> > > > > > > > >> arbitrary code. It is not unusual to have a map that opens
> a
> > > > > second
> > > > > > > HDFS
> > > > > > > > >> file to read some information (e.g. for doing a join of a
> > > small
> > > > > > table
> > > > > > > > >> against a big file). If you use Hadoop Streaming or Pipes
> to
> > > > write
> > > > > a
> > > > > > > job
> > > > > > > > >> in
> > > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> > > processes
> > > > > > which
> > > > > > > > may
> > > > > > > > >> also access external resources in this manner. Some people
> > > also
> > > > > > > > read/write
> > > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> > security
> > > > > > > solution
> > > > > > > > >> would ideally deal with these cases too.
> > > > > > > > >>
> > > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > > > amansk@gmail.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > A quick question here. How does a typical hadoop job
> work
> > at
> > > > the
> > > > > > > > system
> > > > > > > > >> > level? What are the various interactions and how does
> the
> > > data
> > > > > > flow?
> > > > > > > > >> >
> > > > > > > > >> > Amandeep
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Amandeep Khurana
> > > > > > > > >> > Computer Science Graduate Student
> > > > > > > > >> > University of California, Santa Cruz
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > > > amansk@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Thanks Matei. If the basic architecture is similar to
> > the
> > > > > Google
> > > > > > > > >> stuff, I
> > > > > > > > >> > > can safely just work on the project using the
> > information
> > > > from
> > > > > > the
> > > > > > > > >> > papers.
> > > > > > > > >> > >
> > > > > > > > >> > > I am aware of the 4487 jira and the current status of
> > the
> > > > > > > > permissions
> > > > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > > > >> > >
> > > > > > > > >> > > Cheers
> > > > > > > > >> > > Amandeep
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Amandeep Khurana
> > > > > > > > >> > > Computer Science Graduate Student
> > > > > > > > >> > > University of California, Santa Cruz
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > > > matei@cloudera.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >> Forgot to add, this JIRA details the latest security
> > > > features
> > > > > > > that
> > > > > > > > >> are
> > > > > > > > >> > >> being
> > > > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > > > >> > >> This document describes the current status and
> > > limitations
> > > > of
> > > > > > the
> > > > > > > > >> > >> permissions mechanism:
> > > > > > > > >> > >>
> > > > > > > > >>
> > > > > > >
> > > >
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > > > .
> > > > > > > > >> > >>
> > > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > > > matei@cloudera.com
> > > > > > > > >
> > > > > > > > >> > >> wrote:
> > > > > > > > >> > >>
> > > > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > > > > MapReduce/GFS
> > > > > > > > at
> > > > > > > > >> > the
> > > > > > > > >> > >> > level described in those papers. In particular, in
> > > HDFS,
> > > > > > there
> > > > > > > is
> > > > > > > > a
> > > > > > > > >> > >> master
> > > > > > > > >> > >> > node containing metadata and a number of slave
> nodes
> > > > > > > (datanodes)
> > > > > > > > >> > >> containing
> > > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> > > master
> > > > > to
> > > > > > > list
> > > > > > > > >> > >> > directories, etc. When they want to read a region
> of
> > > some
> > > > > > file,
> > > > > > > > >> they
> > > > > > > > >> > >> tell
> > > > > > > > >> > >> > the master the filename and offset, and they
> receive
> > a
> > > > list
> > > > > > of
> > > > > > > > >> block
> > > > > > > > >> > >> > locations (datanodes). They then contact the
> > individual
> > > > > > > datanodes
> > > > > > > > >> to
> > > > > > > > >> > >> read
> > > > > > > > >> > >> > the blocks. When clients write a file, they first
> > > obtain
> > > > a
> > > > > > new
> > > > > > > > >> block
> > > > > > > > >> > ID
> > > > > > > > >> > >> and
> > > > > > > > >> > >> > list of nodes to write it to from the master, then
> > > > contact
> > > > > > the
> > > > > > > > >> > datanodes
> > > > > > > > >> > >> to
> > > > > > > > >> > >> > write it (actually, the datanodes pipeline the
> write
> > as
> > > > in
> > > > > > GFS)
> > > > > > > > and
> > > > > > > > >> > >> report
> > > > > > > > >> > >> > when the write is complete. HDFS actually has some
> > > > security
> > > > > > > > >> mechanisms
> > > > > > > > >> > >> built
> > > > > > > > >> > >> > in, authenticating users based on their Unix ID and
> > > > > providing
> > > > > > > > >> > Unix-like
> > > > > > > > >> > >> file
> > > > > > > > >> > >> > permissions. I don't know much about how these are
> > > > > > implemented,
> > > > > > > > but
> > > > > > > > >> > they
> > > > > > > > >> > >> > would be a good place to start looking.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > > > > >> amansk@gmail.com
> > > > > > > > >> > >> >wrote:
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >> Thanks Matie
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> I had gone through the architecture document
> online.
> > I
> > > > am
> > > > > > > > >> currently
> > > > > > > > >> > >> >> working
> > > > > > > > >> > >> >> on a project towards Security in Hadoop. I do know
> > how
> > > > the
> > > > > > > data
> > > > > > > > >> moves
> > > > > > > > >> > >> >> around
> > > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does
> HDFS
> > > > > follow
> > > > > > > and
> > > > > > > > >> how
> > > > > > > > >> > >> >> different it is from GFS. Can you throw some light
> > on
> > > > > that?
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > > > following
> > > > > > the
> > > > > > > > same
> > > > > > > > >> > >> >> protocols. Thats why the question about how does
> the
> > > > > Hadoop
> > > > > > > > >> framework
> > > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> > from
> > > > Map
> > > > > > > Reduce
> > > > > > > > >> and
> > > > > > > > >> > >> GFS.
> > > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> > information
> > > on
> > > > > how
> > > > > > > > those
> > > > > > > > >> > >> systems
> > > > > > > > >> > >> >> are designed but there is nothing that concrete
> for
> > > > Hadoop
> > > > > > > that
> > > > > > > > I
> > > > > > > > >> > have
> > > > > > > > >> > >> >> been
> > > > > > > > >> > >> >> able to find.
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> Amandeep
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> Amandeep Khurana
> > > > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > > > > >> matei@cloudera.com>
> > > > > > > > >> > >> >> wrote:
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> > Hi Amandeep,
> > > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS
> and
> > > > aims
> > > > > to
> > > > > > > > >> provide
> > > > > > > > >> > >> those
> > > > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> > > > similar
> > > > > to
> > > > > > > GFS
> > > > > > > > >> > (large
> > > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> > > missing
> > > > > are
> > > > > > > > >> > read-write
> > > > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> > > > provided
> > > > > > > > because
> > > > > > > > >> > few
> > > > > > > > >> > >> >> Hadoop
> > > > > > > > >> > >> >> > applications require it) and multiple appenders
> > (the
> > > > > > record
> > > > > > > > >> append
> > > > > > > > >> > >> >> > operation). You can read about HDFS architecture
> > at
> > > > > > > > >> > >> >> >
> > > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > > > .
> > > > > > > > >> The
> > > > > > > > >> > >> >> MapReduce
> > > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same
> way
> > > > that
> > > > > > > > Google's
> > > > > > > > >> > >> >> MapReduce
> > > > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> > > data),
> > > > > > > > although
> > > > > > > > >> > >> Hadoop
> > > > > > > > >> > >> >> > MapReduce also supports running over other
> > > distributed
> > > > > > > > >> filesystems.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > Matei
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep
> Khurana
> > <
> > > > > > > > >> > amansk@gmail.com
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >> > wrote:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > > Hi
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Is the HDFS architecture completely based on
> the
> > > > > Google
> > > > > > > > >> > Filesystem?
> > > > > > > > >> > >> If
> > > > > > > > >> > >> >> it
> > > > > > > > >> > >> >> > > isnt, what are the differences between the
> two?
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> > HDFS
> > > > same
> > > > > > as
> > > > > > > > how
> > > > > > > > >> it
> > > > > > > > >> > is
> > > > > > > > >> > >> >> > between
> > > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Amandeep
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >>
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Ok. Thanks..
Another question now. Do the datanodes have any way of linking a particular
block of data to a global file identifier?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 9:37 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> In general, yeah, the scripts can access any resource they want (within the
> permissions of the user that the task runs as). It's also possible to
> access
> HDFS from scripts because HDFS provides a FUSE interface that can make it
> look like a regular file system on the machine. (The FUSE module in turn
> talks to the namenode as a regular HDFS client.)
>
> On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > I dont know much about Hadoop streaming and have a quick question here.
> >
> > The snippets of code/programs that you attach into the map reduce job
> might
> > want to access outside resources (like you mentioned). Now these might
> not
> > need to go to the namenode right? For example a python script. How would
> it
> > access the data? Would it ask the parent java process (in the
> tasktracker)
> > to get the data or would it go and do stuff on its own?
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> > > Nope, typically the JobTracker just starts the process, and the
> > tasktracker
> > > talks directly to the namenode to get a pointer to the datanode, and
> then
> > > directly to the datanode.
> > >
> > > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Alright.. Got it.
> > > >
> > > > Now, do the task trackers talk to the namenode and the data node
> > directly
> > > > or
> > > > do they go through the job tracker for it? So, if my code is such
> that
> > I
> > > > need to access more files from the hdfs, would the job tracker get
> > > involved
> > > > or not?
> > > >
> > > >
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> > > wrote:
> > > >
> > > > > Normally, HDFS files are accessed through the namenode. If there
> was
> > a
> > > > > malicious process though, then I imagine it could talk to a
> datanode
> > > > > directly and request a specific block.
> > > > >
> > > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Ok. Got it.
> > > > > >
> > > > > > Now, when my job needs to access another file, does it go to the
> > > > Namenode
> > > > > > to
> > > > > > get the block ids? How does the java process know where the files
> > are
> > > > and
> > > > > > how to access them?
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <
> matei@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I mentioned this case because even jobs written in Java can use
> > the
> > > > > HDFS
> > > > > > > API
> > > > > > > to talk to the NameNode and access the filesystem. People often
> > do
> > > > this
> > > > > > > because their job needs to read a config file, some small data
> > > table,
> > > > > etc
> > > > > > > and use this information in its map or reduce functions. In
> this
> > > > case,
> > > > > > you
> > > > > > > open the second file separately in your mapper's init function
> > and
> > > > read
> > > > > > > whatever you need from it. In general I wanted to point out
> that
> > > you
> > > > > > can't
> > > > > > > know which files a job will access unless you look at its
> source
> > > code
> > > > > or
> > > > > > > monitor the calls it makes; the input file(s) you provide in
> the
> > > job
> > > > > > > description are a hint to the MapReduce framework to place your
> > job
> > > > on
> > > > > > > certain nodes, but it's reasonable for the job to access other
> > > files
> > > > as
> > > > > > > well.
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> > > amansk@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Another question that I have here - When the jobs run
> arbitrary
> > > > code
> > > > > > and
> > > > > > > > access data from the HDFS, do they go to the namenode to get
> > the
> > > > > block
> > > > > > > > information?
> > > > > > > >
> > > > > > > >
> > > > > > > > Amandeep Khurana
> > > > > > > > Computer Science Graduate Student
> > > > > > > > University of California, Santa Cruz
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > > > amansk@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Assuming that the job is purely in Java and not involving
> > > > streaming
> > > > > > or
> > > > > > > > > pipes, wouldnt the resources (files) required by the job as
> > > > inputs
> > > > > be
> > > > > > > > known
> > > > > > > > > beforehand? So, if the map task is accessing a second file,
> > how
> > > > > does
> > > > > > it
> > > > > > > > make
> > > > > > > > > it different except that there are multiple files. The
> > > JobTracker
> > > > > > would
> > > > > > > > know
> > > > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > > > >
> > > > > > > > > I am slightly confused why you have mentioned this case
> > > > > separately...
> > > > > > > Can
> > > > > > > > > you elaborate on it a little bit?
> > > > > > > > >
> > > > > > > > > Amandeep
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Amandeep Khurana
> > > > > > > > > Computer Science Graduate Student
> > > > > > > > > University of California, Santa Cruz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > > > matei@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Typically the data flow is like this:1) Client submits a
> job
> > > > > > > description
> > > > > > > > >> to
> > > > > > > > >> the JobTracker.
> > > > > > > > >> 2) JobTracker figures out block locations for the input
> > > file(s)
> > > > by
> > > > > > > > talking
> > > > > > > > >> to HDFS NameNode.
> > > > > > > > >> 3) JobTracker creates a job description file in HDFS which
> > > will
> > > > be
> > > > > > > read
> > > > > > > > by
> > > > > > > > >> the nodes to copy over the job's code etc.
> > > > > > > > >> 4) JobTracker starts map tasks on the slaves
> (TaskTrackers)
> > > with
> > > > > the
> > > > > > > > >> appropriate data blocks.
> > > > > > > > >> 5) After running, maps create intermediate output files on
> > > those
> > > > > > > slaves.
> > > > > > > > >> These are not in HDFS, they're in some temporary storage
> > used
> > > by
> > > > > > > > >> MapReduce.
> > > > > > > > >> 6) JobTracker starts reduces on a series of slaves, which
> > copy
> > > > > over
> > > > > > > the
> > > > > > > > >> appropriate map outputs, apply the reduce function, and
> > write
> > > > the
> > > > > > > > outputs
> > > > > > > > >> to
> > > > > > > > >> HDFS (one output file per reducer).
> > > > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > > > > JobTracker.
> > > > > > > > >>
> > > > > > > > >> However, there is a big caveat, which is that the map and
> > > reduce
> > > > > > tasks
> > > > > > > > run
> > > > > > > > >> arbitrary code. It is not unusual to have a map that opens
> a
> > > > > second
> > > > > > > HDFS
> > > > > > > > >> file to read some information (e.g. for doing a join of a
> > > small
> > > > > > table
> > > > > > > > >> against a big file). If you use Hadoop Streaming or Pipes
> to
> > > > write
> > > > > a
> > > > > > > job
> > > > > > > > >> in
> > > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> > > processes
> > > > > > which
> > > > > > > > may
> > > > > > > > >> also access external resources in this manner. Some people
> > > also
> > > > > > > > read/write
> > > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> > security
> > > > > > > solution
> > > > > > > > >> would ideally deal with these cases too.
> > > > > > > > >>
> > > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > > > amansk@gmail.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > A quick question here. How does a typical hadoop job
> work
> > at
> > > > the
> > > > > > > > system
> > > > > > > > >> > level? What are the various interactions and how does
> the
> > > data
> > > > > > flow?
> > > > > > > > >> >
> > > > > > > > >> > Amandeep
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Amandeep Khurana
> > > > > > > > >> > Computer Science Graduate Student
> > > > > > > > >> > University of California, Santa Cruz
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > > > amansk@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Thanks Matei. If the basic architecture is similar to
> > the
> > > > > Google
> > > > > > > > >> stuff, I
> > > > > > > > >> > > can safely just work on the project using the
> > information
> > > > from
> > > > > > the
> > > > > > > > >> > papers.
> > > > > > > > >> > >
> > > > > > > > >> > > I am aware of the 4487 jira and the current status of
> > the
> > > > > > > > permissions
> > > > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > > > >> > >
> > > > > > > > >> > > Cheers
> > > > > > > > >> > > Amandeep
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Amandeep Khurana
> > > > > > > > >> > > Computer Science Graduate Student
> > > > > > > > >> > > University of California, Santa Cruz
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > > > matei@cloudera.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >> Forgot to add, this JIRA details the latest security
> > > > features
> > > > > > > that
> > > > > > > > >> are
> > > > > > > > >> > >> being
> > > > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > > > >> > >> This document describes the current status and
> > > limitations
> > > > of
> > > > > > the
> > > > > > > > >> > >> permissions mechanism:
> > > > > > > > >> > >>
> > > > > > > > >>
> > > > > > >
> > > >
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > > > .
> > > > > > > > >> > >>
> > > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > > > matei@cloudera.com
> > > > > > > > >
> > > > > > > > >> > >> wrote:
> > > > > > > > >> > >>
> > > > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > > > > MapReduce/GFS
> > > > > > > > at
> > > > > > > > >> > the
> > > > > > > > >> > >> > level described in those papers. In particular, in
> > > HDFS,
> > > > > > there
> > > > > > > is
> > > > > > > > a
> > > > > > > > >> > >> master
> > > > > > > > >> > >> > node containing metadata and a number of slave
> nodes
> > > > > > > (datanodes)
> > > > > > > > >> > >> containing
> > > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> > > master
> > > > > to
> > > > > > > list
> > > > > > > > >> > >> > directories, etc. When they want to read a region
> of
> > > some
> > > > > > file,
> > > > > > > > >> they
> > > > > > > > >> > >> tell
> > > > > > > > >> > >> > the master the filename and offset, and they
> receive
> > a
> > > > list
> > > > > > of
> > > > > > > > >> block
> > > > > > > > >> > >> > locations (datanodes). They then contact the
> > individual
> > > > > > > datanodes
> > > > > > > > >> to
> > > > > > > > >> > >> read
> > > > > > > > >> > >> > the blocks. When clients write a file, they first
> > > obtain
> > > > a
> > > > > > new
> > > > > > > > >> block
> > > > > > > > >> > ID
> > > > > > > > >> > >> and
> > > > > > > > >> > >> > list of nodes to write it to from the master, then
> > > > contact
> > > > > > the
> > > > > > > > >> > datanodes
> > > > > > > > >> > >> to
> > > > > > > > >> > >> > write it (actually, the datanodes pipeline the
> write
> > as
> > > > in
> > > > > > GFS)
> > > > > > > > and
> > > > > > > > >> > >> report
> > > > > > > > >> > >> > when the write is complete. HDFS actually has some
> > > > security
> > > > > > > > >> mechanisms
> > > > > > > > >> > >> built
> > > > > > > > >> > >> > in, authenticating users based on their Unix ID and
> > > > > providing
> > > > > > > > >> > Unix-like
> > > > > > > > >> > >> file
> > > > > > > > >> > >> > permissions. I don't know much about how these are
> > > > > > implemented,
> > > > > > > > but
> > > > > > > > >> > they
> > > > > > > > >> > >> > would be a good place to start looking.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > > > > >> amansk@gmail.com
> > > > > > > > >> > >> >wrote:
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >> Thanks Matie
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> I had gone through the architecture document
> online.
> > I
> > > > am
> > > > > > > > >> currently
> > > > > > > > >> > >> >> working
> > > > > > > > >> > >> >> on a project towards Security in Hadoop. I do know
> > how
> > > > the
> > > > > > > data
> > > > > > > > >> moves
> > > > > > > > >> > >> >> around
> > > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does
> HDFS
> > > > > follow
> > > > > > > and
> > > > > > > > >> how
> > > > > > > > >> > >> >> different it is from GFS. Can you throw some light
> > on
> > > > > that?
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > > > following
> > > > > > the
> > > > > > > > same
> > > > > > > > >> > >> >> protocols. Thats why the question about how does
> the
> > > > > Hadoop
> > > > > > > > >> framework
> > > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> > from
> > > > Map
> > > > > > > Reduce
> > > > > > > > >> and
> > > > > > > > >> > >> GFS.
> > > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> > information
> > > on
> > > > > how
> > > > > > > > those
> > > > > > > > >> > >> systems
> > > > > > > > >> > >> >> are designed but there is nothing that concrete
> for
> > > > Hadoop
> > > > > > > that
> > > > > > > > I
> > > > > > > > >> > have
> > > > > > > > >> > >> >> been
> > > > > > > > >> > >> >> able to find.
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> Amandeep
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> Amandeep Khurana
> > > > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > > > > >> matei@cloudera.com>
> > > > > > > > >> > >> >> wrote:
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> > Hi Amandeep,
> > > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS
> and
> > > > aims
> > > > > to
> > > > > > > > >> provide
> > > > > > > > >> > >> those
> > > > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> > > > similar
> > > > > to
> > > > > > > GFS
> > > > > > > > >> > (large
> > > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> > > missing
> > > > > are
> > > > > > > > >> > read-write
> > > > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> > > > provided
> > > > > > > > because
> > > > > > > > >> > few
> > > > > > > > >> > >> >> Hadoop
> > > > > > > > >> > >> >> > applications require it) and multiple appenders
> > (the
> > > > > > record
> > > > > > > > >> append
> > > > > > > > >> > >> >> > operation). You can read about HDFS architecture
> > at
> > > > > > > > >> > >> >> >
> > > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > > > .
> > > > > > > > >> The
> > > > > > > > >> > >> >> MapReduce
> > > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same
> way
> > > > that
> > > > > > > > Google's
> > > > > > > > >> > >> >> MapReduce
> > > > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> > > data),
> > > > > > > > although
> > > > > > > > >> > >> Hadoop
> > > > > > > > >> > >> >> > MapReduce also supports running over other
> > > distributed
> > > > > > > > >> filesystems.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > Matei
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep
> Khurana
> > <
> > > > > > > > >> > amansk@gmail.com
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >> > wrote:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> > > Hi
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Is the HDFS architecture completely based on
> the
> > > > > Google
> > > > > > > > >> > Filesystem?
> > > > > > > > >> > >> If
> > > > > > > > >> > >> >> it
> > > > > > > > >> > >> >> > > isnt, what are the differences between the
> two?
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> > HDFS
> > > > same
> > > > > > as
> > > > > > > > how
> > > > > > > > >> it
> > > > > > > > >> > is
> > > > > > > > >> > >> >> > between
> > > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Amandeep
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > > > >> > >> >> > >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >>
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Steve Loughran <st...@apache.org>.
Amr Awadallah wrote:
>> I didn't understand usage of "malicuous" here,
>> but any process using HDFS api should first ask NameNode where the
>
> Rasit,
>
> Matei is referring to fact that a malicious peace of code can bypass the
> Name Node and connect to any data node directly, or probe all data nodes for
> that matter. There is no strong authentication for RPC at this layer of
> HDFS, which is one of the current shortcomings that will be addressed in
> hadoop 1.0.
>
> -- amr
This shouldn't be a problem in a locked down datacentre. Where it is a
risk is when you host on EC2 or other Vm-hosting service that doesn't
set up private VPNs
Re: HDFS architecture based on GFS?
Posted by Amr Awadallah <aa...@cloudera.com>.
> I didn't understand usage of "malicuous" here,
> but any process using HDFS api should first ask NameNode where the
Rasit,
Matei is referring to fact that a malicious peace of code can bypass the
Name Node and connect to any data node directly, or probe all data nodes for
that matter. There is no strong authentication for RPC at this layer of
HDFS, which is one of the current shortcomings that will be addressed in
hadoop 1.0.
-- amr
On Sun, Feb 15, 2009 at 11:41 PM, Rasit OZDAS <ra...@gmail.com> wrote:
> "If there was a
> malicious process though, then I imagine it could talk to a datanode
> directly and request a specific block."
>
> I didn't understand usage of "malicuous" here,
> but any process using HDFS api should first ask NameNode where the
> file replications are.
> Then - I assume - namenode returns the IP of best DataNode (or all IPs),
> then call to specific DataNode is made.
> Please correct me if I'm wrong.
>
> Cheers,
> Rasit
>
> 2009/2/16 Matei Zaharia <ma...@cloudera.com>:
> > In general, yeah, the scripts can access any resource they want (within
> the
> > permissions of the user that the task runs as). It's also possible to
> access
> > HDFS from scripts because HDFS provides a FUSE interface that can make it
> > look like a regular file system on the machine. (The FUSE module in turn
> > talks to the namenode as a regular HDFS client.)
> >
> > On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
> >
> >> I dont know much about Hadoop streaming and have a quick question here.
> >>
> >> The snippets of code/programs that you attach into the map reduce job
> might
> >> want to access outside resources (like you mentioned). Now these might
> not
> >> need to go to the namenode right? For example a python script. How would
> it
> >> access the data? Would it ask the parent java process (in the
> tasktracker)
> >> to get the data or would it go and do stuff on its own?
> >>
> >>
> >> Amandeep Khurana
> >> Computer Science Graduate Student
> >> University of California, Santa Cruz
> >>
> >>
> >> On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >>
> >> > Nope, typically the JobTracker just starts the process, and the
> >> tasktracker
> >> > talks directly to the namenode to get a pointer to the datanode, and
> then
> >> > directly to the datanode.
> >> >
> >> > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> >> > wrote:
> >> >
> >> > > Alright.. Got it.
> >> > >
> >> > > Now, do the task trackers talk to the namenode and the data node
> >> directly
> >> > > or
> >> > > do they go through the job tracker for it? So, if my code is such
> that
> >> I
> >> > > need to access more files from the hdfs, would the job tracker get
> >> > involved
> >> > > or not?
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Amandeep Khurana
> >> > > Computer Science Graduate Student
> >> > > University of California, Santa Cruz
> >> > >
> >> > >
> >> > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> >> > wrote:
> >> > >
> >> > > > Normally, HDFS files are accessed through the namenode. If there
> was
> >> a
> >> > > > malicious process though, then I imagine it could talk to a
> datanode
> >> > > > directly and request a specific block.
> >> > > >
> >> > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <
> amansk@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Ok. Got it.
> >> > > > >
> >> > > > > Now, when my job needs to access another file, does it go to the
> >> > > Namenode
> >> > > > > to
> >> > > > > get the block ids? How does the java process know where the
> files
> >> are
> >> > > and
> >> > > > > how to access them?
> >> > > > >
> >> > > > >
> >> > > > > Amandeep Khurana
> >> > > > > Computer Science Graduate Student
> >> > > > > University of California, Santa Cruz
> >> > > > >
> >> > > > >
> >> > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <
> matei@cloudera.com
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > I mentioned this case because even jobs written in Java can
> use
> >> the
> >> > > > HDFS
> >> > > > > > API
> >> > > > > > to talk to the NameNode and access the filesystem. People
> often
> >> do
> >> > > this
> >> > > > > > because their job needs to read a config file, some small data
> >> > table,
> >> > > > etc
> >> > > > > > and use this information in its map or reduce functions. In
> this
> >> > > case,
> >> > > > > you
> >> > > > > > open the second file separately in your mapper's init function
> >> and
> >> > > read
> >> > > > > > whatever you need from it. In general I wanted to point out
> that
> >> > you
> >> > > > > can't
> >> > > > > > know which files a job will access unless you look at its
> source
> >> > code
> >> > > > or
> >> > > > > > monitor the calls it makes; the input file(s) you provide in
> the
> >> > job
> >> > > > > > description are a hint to the MapReduce framework to place
> your
> >> job
> >> > > on
> >> > > > > > certain nodes, but it's reasonable for the job to access other
> >> > files
> >> > > as
> >> > > > > > well.
> >> > > > > >
> >> > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> >> > amansk@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Another question that I have here - When the jobs run
> arbitrary
> >> > > code
> >> > > > > and
> >> > > > > > > access data from the HDFS, do they go to the namenode to get
> >> the
> >> > > > block
> >> > > > > > > information?
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Amandeep Khurana
> >> > > > > > > Computer Science Graduate Student
> >> > > > > > > University of California, Santa Cruz
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> >> > > amansk@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Assuming that the job is purely in Java and not involving
> >> > > streaming
> >> > > > > or
> >> > > > > > > > pipes, wouldnt the resources (files) required by the job
> as
> >> > > inputs
> >> > > > be
> >> > > > > > > known
> >> > > > > > > > beforehand? So, if the map task is accessing a second
> file,
> >> how
> >> > > > does
> >> > > > > it
> >> > > > > > > make
> >> > > > > > > > it different except that there are multiple files. The
> >> > JobTracker
> >> > > > > would
> >> > > > > > > know
> >> > > > > > > > beforehand that multiple files would be accessed. Right?
> >> > > > > > > >
> >> > > > > > > > I am slightly confused why you have mentioned this case
> >> > > > separately...
> >> > > > > > Can
> >> > > > > > > > you elaborate on it a little bit?
> >> > > > > > > >
> >> > > > > > > > Amandeep
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Amandeep Khurana
> >> > > > > > > > Computer Science Graduate Student
> >> > > > > > > > University of California, Santa Cruz
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> >> > > matei@cloudera.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Typically the data flow is like this:1) Client submits a
> job
> >> > > > > > description
> >> > > > > > > >> to
> >> > > > > > > >> the JobTracker.
> >> > > > > > > >> 2) JobTracker figures out block locations for the input
> >> > file(s)
> >> > > by
> >> > > > > > > talking
> >> > > > > > > >> to HDFS NameNode.
> >> > > > > > > >> 3) JobTracker creates a job description file in HDFS
> which
> >> > will
> >> > > be
> >> > > > > > read
> >> > > > > > > by
> >> > > > > > > >> the nodes to copy over the job's code etc.
> >> > > > > > > >> 4) JobTracker starts map tasks on the slaves
> (TaskTrackers)
> >> > with
> >> > > > the
> >> > > > > > > >> appropriate data blocks.
> >> > > > > > > >> 5) After running, maps create intermediate output files
> on
> >> > those
> >> > > > > > slaves.
> >> > > > > > > >> These are not in HDFS, they're in some temporary storage
> >> used
> >> > by
> >> > > > > > > >> MapReduce.
> >> > > > > > > >> 6) JobTracker starts reduces on a series of slaves, which
> >> copy
> >> > > > over
> >> > > > > > the
> >> > > > > > > >> appropriate map outputs, apply the reduce function, and
> >> write
> >> > > the
> >> > > > > > > outputs
> >> > > > > > > >> to
> >> > > > > > > >> HDFS (one output file per reducer).
> >> > > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> >> > > > > JobTracker.
> >> > > > > > > >>
> >> > > > > > > >> However, there is a big caveat, which is that the map and
> >> > reduce
> >> > > > > tasks
> >> > > > > > > run
> >> > > > > > > >> arbitrary code. It is not unusual to have a map that
> opens a
> >> > > > second
> >> > > > > > HDFS
> >> > > > > > > >> file to read some information (e.g. for doing a join of a
> >> > small
> >> > > > > table
> >> > > > > > > >> against a big file). If you use Hadoop Streaming or Pipes
> to
> >> > > write
> >> > > > a
> >> > > > > > job
> >> > > > > > > >> in
> >> > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> >> > processes
> >> > > > > which
> >> > > > > > > may
> >> > > > > > > >> also access external resources in this manner. Some
> people
> >> > also
> >> > > > > > > read/write
> >> > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> >> security
> >> > > > > > solution
> >> > > > > > > >> would ideally deal with these cases too.
> >> > > > > > > >>
> >> > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> >> > > > amansk@gmail.com
> >> > > > > >
> >> > > > > > > >> wrote:
> >> > > > > > > >>
> >> > > > > > > >> > A quick question here. How does a typical hadoop job
> work
> >> at
> >> > > the
> >> > > > > > > system
> >> > > > > > > >> > level? What are the various interactions and how does
> the
> >> > data
> >> > > > > flow?
> >> > > > > > > >> >
> >> > > > > > > >> > Amandeep
> >> > > > > > > >> >
> >> > > > > > > >> >
> >> > > > > > > >> > Amandeep Khurana
> >> > > > > > > >> > Computer Science Graduate Student
> >> > > > > > > >> > University of California, Santa Cruz
> >> > > > > > > >> >
> >> > > > > > > >> >
> >> > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> >> > > > > amansk@gmail.com
> >> > > > > > >
> >> > > > > > > >> > wrote:
> >> > > > > > > >> >
> >> > > > > > > >> > > Thanks Matei. If the basic architecture is similar to
> >> the
> >> > > > Google
> >> > > > > > > >> stuff, I
> >> > > > > > > >> > > can safely just work on the project using the
> >> information
> >> > > from
> >> > > > > the
> >> > > > > > > >> > papers.
> >> > > > > > > >> > >
> >> > > > > > > >> > > I am aware of the 4487 jira and the current status of
> >> the
> >> > > > > > > permissions
> >> > > > > > > >> > > mechanism. I had a look at them earlier.
> >> > > > > > > >> > >
> >> > > > > > > >> > > Cheers
> >> > > > > > > >> > > Amandeep
> >> > > > > > > >> > >
> >> > > > > > > >> > >
> >> > > > > > > >> > > Amandeep Khurana
> >> > > > > > > >> > > Computer Science Graduate Student
> >> > > > > > > >> > > University of California, Santa Cruz
> >> > > > > > > >> > >
> >> > > > > > > >> > >
> >> > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> >> > > > > > matei@cloudera.com>
> >> > > > > > > >> > wrote:
> >> > > > > > > >> > >
> >> > > > > > > >> > >> Forgot to add, this JIRA details the latest security
> >> > > features
> >> > > > > > that
> >> > > > > > > >> are
> >> > > > > > > >> > >> being
> >> > > > > > > >> > >> worked on in Hadoop trunk:
> >> > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> >> > > > > > > >> > >> This document describes the current status and
> >> > limitations
> >> > > of
> >> > > > > the
> >> > > > > > > >> > >> permissions mechanism:
> >> > > > > > > >> > >>
> >> > > > > > > >>
> >> > > > > >
> >> > >
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> >> > > > .
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> >> > > > > > matei@cloudera.com
> >> > > > > > > >
> >> > > > > > > >> > >> wrote:
> >> > > > > > > >> > >>
> >> > > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> >> > > > > > MapReduce/GFS
> >> > > > > > > at
> >> > > > > > > >> > the
> >> > > > > > > >> > >> > level described in those papers. In particular, in
> >> > HDFS,
> >> > > > > there
> >> > > > > > is
> >> > > > > > > a
> >> > > > > > > >> > >> master
> >> > > > > > > >> > >> > node containing metadata and a number of slave
> nodes
> >> > > > > > (datanodes)
> >> > > > > > > >> > >> containing
> >> > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> >> > master
> >> > > > to
> >> > > > > > list
> >> > > > > > > >> > >> > directories, etc. When they want to read a region
> of
> >> > some
> >> > > > > file,
> >> > > > > > > >> they
> >> > > > > > > >> > >> tell
> >> > > > > > > >> > >> > the master the filename and offset, and they
> receive
> >> a
> >> > > list
> >> > > > > of
> >> > > > > > > >> block
> >> > > > > > > >> > >> > locations (datanodes). They then contact the
> >> individual
> >> > > > > > datanodes
> >> > > > > > > >> to
> >> > > > > > > >> > >> read
> >> > > > > > > >> > >> > the blocks. When clients write a file, they first
> >> > obtain
> >> > > a
> >> > > > > new
> >> > > > > > > >> block
> >> > > > > > > >> > ID
> >> > > > > > > >> > >> and
> >> > > > > > > >> > >> > list of nodes to write it to from the master, then
> >> > > contact
> >> > > > > the
> >> > > > > > > >> > datanodes
> >> > > > > > > >> > >> to
> >> > > > > > > >> > >> > write it (actually, the datanodes pipeline the
> write
> >> as
> >> > > in
> >> > > > > GFS)
> >> > > > > > > and
> >> > > > > > > >> > >> report
> >> > > > > > > >> > >> > when the write is complete. HDFS actually has some
> >> > > security
> >> > > > > > > >> mechanisms
> >> > > > > > > >> > >> built
> >> > > > > > > >> > >> > in, authenticating users based on their Unix ID
> and
> >> > > > providing
> >> > > > > > > >> > Unix-like
> >> > > > > > > >> > >> file
> >> > > > > > > >> > >> > permissions. I don't know much about how these are
> >> > > > > implemented,
> >> > > > > > > but
> >> > > > > > > >> > they
> >> > > > > > > >> > >> > would be a good place to start looking.
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana
> <
> >> > > > > > > >> amansk@gmail.com
> >> > > > > > > >> > >> >wrote:
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >> >> Thanks Matie
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> I had gone through the architecture document
> online.
> >> I
> >> > > am
> >> > > > > > > >> currently
> >> > > > > > > >> > >> >> working
> >> > > > > > > >> > >> >> on a project towards Security in Hadoop. I do
> know
> >> how
> >> > > the
> >> > > > > > data
> >> > > > > > > >> moves
> >> > > > > > > >> > >> >> around
> >> > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does
> HDFS
> >> > > > follow
> >> > > > > > and
> >> > > > > > > >> how
> >> > > > > > > >> > >> >> different it is from GFS. Can you throw some
> light
> >> on
> >> > > > that?
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> >> > > following
> >> > > > > the
> >> > > > > > > same
> >> > > > > > > >> > >> >> protocols. Thats why the question about how does
> the
> >> > > > Hadoop
> >> > > > > > > >> framework
> >> > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> >> from
> >> > > Map
> >> > > > > > Reduce
> >> > > > > > > >> and
> >> > > > > > > >> > >> GFS.
> >> > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> >> information
> >> > on
> >> > > > how
> >> > > > > > > those
> >> > > > > > > >> > >> systems
> >> > > > > > > >> > >> >> are designed but there is nothing that concrete
> for
> >> > > Hadoop
> >> > > > > > that
> >> > > > > > > I
> >> > > > > > > >> > have
> >> > > > > > > >> > >> >> been
> >> > > > > > > >> > >> >> able to find.
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> Amandeep
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> Amandeep Khurana
> >> > > > > > > >> > >> >> Computer Science Graduate Student
> >> > > > > > > >> > >> >> University of California, Santa Cruz
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> >> > > > > > > >> matei@cloudera.com>
> >> > > > > > > >> > >> >> wrote:
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >> > Hi Amandeep,
> >> > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS
> and
> >> > > aims
> >> > > > to
> >> > > > > > > >> provide
> >> > > > > > > >> > >> those
> >> > > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> >> > > similar
> >> > > > to
> >> > > > > > GFS
> >> > > > > > > >> > (large
> >> > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> >> > missing
> >> > > > are
> >> > > > > > > >> > read-write
> >> > > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> >> > > provided
> >> > > > > > > because
> >> > > > > > > >> > few
> >> > > > > > > >> > >> >> Hadoop
> >> > > > > > > >> > >> >> > applications require it) and multiple appenders
> >> (the
> >> > > > > record
> >> > > > > > > >> append
> >> > > > > > > >> > >> >> > operation). You can read about HDFS
> architecture
> >> at
> >> > > > > > > >> > >> >> >
> >> > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> >> > > > > > .
> >> > > > > > > >> The
> >> > > > > > > >> > >> >> MapReduce
> >> > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same
> way
> >> > > that
> >> > > > > > > Google's
> >> > > > > > > >> > >> >> MapReduce
> >> > > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> >> > data),
> >> > > > > > > although
> >> > > > > > > >> > >> Hadoop
> >> > > > > > > >> > >> >> > MapReduce also supports running over other
> >> > distributed
> >> > > > > > > >> filesystems.
> >> > > > > > > >> > >> >> >
> >> > > > > > > >> > >> >> > Matei
> >> > > > > > > >> > >> >> >
> >> > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep
> Khurana
> >> <
> >> > > > > > > >> > amansk@gmail.com
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >> >> > wrote:
> >> > > > > > > >> > >> >> >
> >> > > > > > > >> > >> >> > > Hi
> >> > > > > > > >> > >> >> > >
> >> > > > > > > >> > >> >> > > Is the HDFS architecture completely based on
> the
> >> > > > Google
> >> > > > > > > >> > Filesystem?
> >> > > > > > > >> > >> If
> >> > > > > > > >> > >> >> it
> >> > > > > > > >> > >> >> > > isnt, what are the differences between the
> two?
> >> > > > > > > >> > >> >> > >
> >> > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> >> HDFS
> >> > > same
> >> > > > > as
> >> > > > > > > how
> >> > > > > > > >> it
> >> > > > > > > >> > is
> >> > > > > > > >> > >> >> > between
> >> > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> >> > > > > > > >> > >> >> > >
> >> > > > > > > >> > >> >> > > Amandeep
> >> > > > > > > >> > >> >> > >
> >> > > > > > > >> > >> >> > >
> >> > > > > > > >> > >> >> > > Amandeep Khurana
> >> > > > > > > >> > >> >> > > Computer Science Graduate Student
> >> > > > > > > >> > >> >> > > University of California, Santa Cruz
> >> > > > > > > >> > >> >> > >
> >> > > > > > > >> > >> >> >
> >> > > > > > > >> > >> >>
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >> >
> >> > > > > > > >> > >>
> >> > > > > > > >> > >
> >> > > > > > > >> > >
> >> > > > > > > >> >
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
>
>
> --
> M. Raşit ÖZDAŞ
>
Re: HDFS architecture based on GFS?
Posted by Rasit OZDAS <ra...@gmail.com>.
"If there was a
malicious process though, then I imagine it could talk to a datanode
directly and request a specific block."
I didn't understand usage of "malicuous" here,
but any process using HDFS api should first ask NameNode where the
file replications are.
Then - I assume - namenode returns the IP of best DataNode (or all IPs),
then call to specific DataNode is made.
Please correct me if I'm wrong.
Cheers,
Rasit
2009/2/16 Matei Zaharia <ma...@cloudera.com>:
> In general, yeah, the scripts can access any resource they want (within the
> permissions of the user that the task runs as). It's also possible to access
> HDFS from scripts because HDFS provides a FUSE interface that can make it
> look like a regular file system on the machine. (The FUSE module in turn
> talks to the namenode as a regular HDFS client.)
>
> On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com> wrote:
>
>> I dont know much about Hadoop streaming and have a quick question here.
>>
>> The snippets of code/programs that you attach into the map reduce job might
>> want to access outside resources (like you mentioned). Now these might not
>> need to go to the namenode right? For example a python script. How would it
>> access the data? Would it ask the parent java process (in the tasktracker)
>> to get the data or would it go and do stuff on its own?
>>
>>
>> Amandeep Khurana
>> Computer Science Graduate Student
>> University of California, Santa Cruz
>>
>>
>> On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>>
>> > Nope, typically the JobTracker just starts the process, and the
>> tasktracker
>> > talks directly to the namenode to get a pointer to the datanode, and then
>> > directly to the datanode.
>> >
>> > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
>> > wrote:
>> >
>> > > Alright.. Got it.
>> > >
>> > > Now, do the task trackers talk to the namenode and the data node
>> directly
>> > > or
>> > > do they go through the job tracker for it? So, if my code is such that
>> I
>> > > need to access more files from the hdfs, would the job tracker get
>> > involved
>> > > or not?
>> > >
>> > >
>> > >
>> > >
>> > > Amandeep Khurana
>> > > Computer Science Graduate Student
>> > > University of California, Santa Cruz
>> > >
>> > >
>> > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
>> > wrote:
>> > >
>> > > > Normally, HDFS files are accessed through the namenode. If there was
>> a
>> > > > malicious process though, then I imagine it could talk to a datanode
>> > > > directly and request a specific block.
>> > > >
>> > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Ok. Got it.
>> > > > >
>> > > > > Now, when my job needs to access another file, does it go to the
>> > > Namenode
>> > > > > to
>> > > > > get the block ids? How does the java process know where the files
>> are
>> > > and
>> > > > > how to access them?
>> > > > >
>> > > > >
>> > > > > Amandeep Khurana
>> > > > > Computer Science Graduate Student
>> > > > > University of California, Santa Cruz
>> > > > >
>> > > > >
>> > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <matei@cloudera.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > I mentioned this case because even jobs written in Java can use
>> the
>> > > > HDFS
>> > > > > > API
>> > > > > > to talk to the NameNode and access the filesystem. People often
>> do
>> > > this
>> > > > > > because their job needs to read a config file, some small data
>> > table,
>> > > > etc
>> > > > > > and use this information in its map or reduce functions. In this
>> > > case,
>> > > > > you
>> > > > > > open the second file separately in your mapper's init function
>> and
>> > > read
>> > > > > > whatever you need from it. In general I wanted to point out that
>> > you
>> > > > > can't
>> > > > > > know which files a job will access unless you look at its source
>> > code
>> > > > or
>> > > > > > monitor the calls it makes; the input file(s) you provide in the
>> > job
>> > > > > > description are a hint to the MapReduce framework to place your
>> job
>> > > on
>> > > > > > certain nodes, but it's reasonable for the job to access other
>> > files
>> > > as
>> > > > > > well.
>> > > > > >
>> > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
>> > amansk@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Another question that I have here - When the jobs run arbitrary
>> > > code
>> > > > > and
>> > > > > > > access data from the HDFS, do they go to the namenode to get
>> the
>> > > > block
>> > > > > > > information?
>> > > > > > >
>> > > > > > >
>> > > > > > > Amandeep Khurana
>> > > > > > > Computer Science Graduate Student
>> > > > > > > University of California, Santa Cruz
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
>> > > amansk@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Assuming that the job is purely in Java and not involving
>> > > streaming
>> > > > > or
>> > > > > > > > pipes, wouldnt the resources (files) required by the job as
>> > > inputs
>> > > > be
>> > > > > > > known
>> > > > > > > > beforehand? So, if the map task is accessing a second file,
>> how
>> > > > does
>> > > > > it
>> > > > > > > make
>> > > > > > > > it different except that there are multiple files. The
>> > JobTracker
>> > > > > would
>> > > > > > > know
>> > > > > > > > beforehand that multiple files would be accessed. Right?
>> > > > > > > >
>> > > > > > > > I am slightly confused why you have mentioned this case
>> > > > separately...
>> > > > > > Can
>> > > > > > > > you elaborate on it a little bit?
>> > > > > > > >
>> > > > > > > > Amandeep
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Amandeep Khurana
>> > > > > > > > Computer Science Graduate Student
>> > > > > > > > University of California, Santa Cruz
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
>> > > matei@cloudera.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Typically the data flow is like this:1) Client submits a job
>> > > > > > description
>> > > > > > > >> to
>> > > > > > > >> the JobTracker.
>> > > > > > > >> 2) JobTracker figures out block locations for the input
>> > file(s)
>> > > by
>> > > > > > > talking
>> > > > > > > >> to HDFS NameNode.
>> > > > > > > >> 3) JobTracker creates a job description file in HDFS which
>> > will
>> > > be
>> > > > > > read
>> > > > > > > by
>> > > > > > > >> the nodes to copy over the job's code etc.
>> > > > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers)
>> > with
>> > > > the
>> > > > > > > >> appropriate data blocks.
>> > > > > > > >> 5) After running, maps create intermediate output files on
>> > those
>> > > > > > slaves.
>> > > > > > > >> These are not in HDFS, they're in some temporary storage
>> used
>> > by
>> > > > > > > >> MapReduce.
>> > > > > > > >> 6) JobTracker starts reduces on a series of slaves, which
>> copy
>> > > > over
>> > > > > > the
>> > > > > > > >> appropriate map outputs, apply the reduce function, and
>> write
>> > > the
>> > > > > > > outputs
>> > > > > > > >> to
>> > > > > > > >> HDFS (one output file per reducer).
>> > > > > > > >> 7) Some logs for the job may also be put into HDFS by the
>> > > > > JobTracker.
>> > > > > > > >>
>> > > > > > > >> However, there is a big caveat, which is that the map and
>> > reduce
>> > > > > tasks
>> > > > > > > run
>> > > > > > > >> arbitrary code. It is not unusual to have a map that opens a
>> > > > second
>> > > > > > HDFS
>> > > > > > > >> file to read some information (e.g. for doing a join of a
>> > small
>> > > > > table
>> > > > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
>> > > write
>> > > > a
>> > > > > > job
>> > > > > > > >> in
>> > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
>> > processes
>> > > > > which
>> > > > > > > may
>> > > > > > > >> also access external resources in this manner. Some people
>> > also
>> > > > > > > read/write
>> > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
>> security
>> > > > > > solution
>> > > > > > > >> would ideally deal with these cases too.
>> > > > > > > >>
>> > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
>> > > > amansk@gmail.com
>> > > > > >
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >> > A quick question here. How does a typical hadoop job work
>> at
>> > > the
>> > > > > > > system
>> > > > > > > >> > level? What are the various interactions and how does the
>> > data
>> > > > > flow?
>> > > > > > > >> >
>> > > > > > > >> > Amandeep
>> > > > > > > >> >
>> > > > > > > >> >
>> > > > > > > >> > Amandeep Khurana
>> > > > > > > >> > Computer Science Graduate Student
>> > > > > > > >> > University of California, Santa Cruz
>> > > > > > > >> >
>> > > > > > > >> >
>> > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
>> > > > > amansk@gmail.com
>> > > > > > >
>> > > > > > > >> > wrote:
>> > > > > > > >> >
>> > > > > > > >> > > Thanks Matei. If the basic architecture is similar to
>> the
>> > > > Google
>> > > > > > > >> stuff, I
>> > > > > > > >> > > can safely just work on the project using the
>> information
>> > > from
>> > > > > the
>> > > > > > > >> > papers.
>> > > > > > > >> > >
>> > > > > > > >> > > I am aware of the 4487 jira and the current status of
>> the
>> > > > > > > permissions
>> > > > > > > >> > > mechanism. I had a look at them earlier.
>> > > > > > > >> > >
>> > > > > > > >> > > Cheers
>> > > > > > > >> > > Amandeep
>> > > > > > > >> > >
>> > > > > > > >> > >
>> > > > > > > >> > > Amandeep Khurana
>> > > > > > > >> > > Computer Science Graduate Student
>> > > > > > > >> > > University of California, Santa Cruz
>> > > > > > > >> > >
>> > > > > > > >> > >
>> > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
>> > > > > > matei@cloudera.com>
>> > > > > > > >> > wrote:
>> > > > > > > >> > >
>> > > > > > > >> > >> Forgot to add, this JIRA details the latest security
>> > > features
>> > > > > > that
>> > > > > > > >> are
>> > > > > > > >> > >> being
>> > > > > > > >> > >> worked on in Hadoop trunk:
>> > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
>> > > > > > > >> > >> This document describes the current status and
>> > limitations
>> > > of
>> > > > > the
>> > > > > > > >> > >> permissions mechanism:
>> > > > > > > >> > >>
>> > > > > > > >>
>> > > > > >
>> > > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
>> > > > .
>> > > > > > > >> > >>
>> > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
>> > > > > > matei@cloudera.com
>> > > > > > > >
>> > > > > > > >> > >> wrote:
>> > > > > > > >> > >>
>> > > > > > > >> > >> > I think it's safe to assume that Hadoop works like
>> > > > > > MapReduce/GFS
>> > > > > > > at
>> > > > > > > >> > the
>> > > > > > > >> > >> > level described in those papers. In particular, in
>> > HDFS,
>> > > > > there
>> > > > > > is
>> > > > > > > a
>> > > > > > > >> > >> master
>> > > > > > > >> > >> > node containing metadata and a number of slave nodes
>> > > > > > (datanodes)
>> > > > > > > >> > >> containing
>> > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
>> > master
>> > > > to
>> > > > > > list
>> > > > > > > >> > >> > directories, etc. When they want to read a region of
>> > some
>> > > > > file,
>> > > > > > > >> they
>> > > > > > > >> > >> tell
>> > > > > > > >> > >> > the master the filename and offset, and they receive
>> a
>> > > list
>> > > > > of
>> > > > > > > >> block
>> > > > > > > >> > >> > locations (datanodes). They then contact the
>> individual
>> > > > > > datanodes
>> > > > > > > >> to
>> > > > > > > >> > >> read
>> > > > > > > >> > >> > the blocks. When clients write a file, they first
>> > obtain
>> > > a
>> > > > > new
>> > > > > > > >> block
>> > > > > > > >> > ID
>> > > > > > > >> > >> and
>> > > > > > > >> > >> > list of nodes to write it to from the master, then
>> > > contact
>> > > > > the
>> > > > > > > >> > datanodes
>> > > > > > > >> > >> to
>> > > > > > > >> > >> > write it (actually, the datanodes pipeline the write
>> as
>> > > in
>> > > > > GFS)
>> > > > > > > and
>> > > > > > > >> > >> report
>> > > > > > > >> > >> > when the write is complete. HDFS actually has some
>> > > security
>> > > > > > > >> mechanisms
>> > > > > > > >> > >> built
>> > > > > > > >> > >> > in, authenticating users based on their Unix ID and
>> > > > providing
>> > > > > > > >> > Unix-like
>> > > > > > > >> > >> file
>> > > > > > > >> > >> > permissions. I don't know much about how these are
>> > > > > implemented,
>> > > > > > > but
>> > > > > > > >> > they
>> > > > > > > >> > >> > would be a good place to start looking.
>> > > > > > > >> > >> >
>> > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
>> > > > > > > >> amansk@gmail.com
>> > > > > > > >> > >> >wrote:
>> > > > > > > >> > >> >
>> > > > > > > >> > >> >> Thanks Matie
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> I had gone through the architecture document online.
>> I
>> > > am
>> > > > > > > >> currently
>> > > > > > > >> > >> >> working
>> > > > > > > >> > >> >> on a project towards Security in Hadoop. I do know
>> how
>> > > the
>> > > > > > data
>> > > > > > > >> moves
>> > > > > > > >> > >> >> around
>> > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
>> > > > follow
>> > > > > > and
>> > > > > > > >> how
>> > > > > > > >> > >> >> different it is from GFS. Can you throw some light
>> on
>> > > > that?
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
>> > > following
>> > > > > the
>> > > > > > > same
>> > > > > > > >> > >> >> protocols. Thats why the question about how does the
>> > > > Hadoop
>> > > > > > > >> framework
>> > > > > > > >> > >> >> integrate with the HDFS, and how different is it
>> from
>> > > Map
>> > > > > > Reduce
>> > > > > > > >> and
>> > > > > > > >> > >> GFS.
>> > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
>> information
>> > on
>> > > > how
>> > > > > > > those
>> > > > > > > >> > >> systems
>> > > > > > > >> > >> >> are designed but there is nothing that concrete for
>> > > Hadoop
>> > > > > > that
>> > > > > > > I
>> > > > > > > >> > have
>> > > > > > > >> > >> >> been
>> > > > > > > >> > >> >> able to find.
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> Amandeep
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> Amandeep Khurana
>> > > > > > > >> > >> >> Computer Science Graduate Student
>> > > > > > > >> > >> >> University of California, Santa Cruz
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
>> > > > > > > >> matei@cloudera.com>
>> > > > > > > >> > >> >> wrote:
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >> > Hi Amandeep,
>> > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
>> > > aims
>> > > > to
>> > > > > > > >> provide
>> > > > > > > >> > >> those
>> > > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
>> > > similar
>> > > > to
>> > > > > > GFS
>> > > > > > > >> > (large
>> > > > > > > >> > >> >> > blocks, replication, etc); some notable things
>> > missing
>> > > > are
>> > > > > > > >> > read-write
>> > > > > > > >> > >> >> > support in the middle of a file (unlikely to be
>> > > provided
>> > > > > > > because
>> > > > > > > >> > few
>> > > > > > > >> > >> >> Hadoop
>> > > > > > > >> > >> >> > applications require it) and multiple appenders
>> (the
>> > > > > record
>> > > > > > > >> append
>> > > > > > > >> > >> >> > operation). You can read about HDFS architecture
>> at
>> > > > > > > >> > >> >> >
>> > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
>> > > > > > .
>> > > > > > > >> The
>> > > > > > > >> > >> >> MapReduce
>> > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
>> > > that
>> > > > > > > Google's
>> > > > > > > >> > >> >> MapReduce
>> > > > > > > >> > >> >> > interacts with GFS (shipping computation to the
>> > data),
>> > > > > > > although
>> > > > > > > >> > >> Hadoop
>> > > > > > > >> > >> >> > MapReduce also supports running over other
>> > distributed
>> > > > > > > >> filesystems.
>> > > > > > > >> > >> >> >
>> > > > > > > >> > >> >> > Matei
>> > > > > > > >> > >> >> >
>> > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana
>> <
>> > > > > > > >> > amansk@gmail.com
>> > > > > > > >> > >> >
>> > > > > > > >> > >> >> > wrote:
>> > > > > > > >> > >> >> >
>> > > > > > > >> > >> >> > > Hi
>> > > > > > > >> > >> >> > >
>> > > > > > > >> > >> >> > > Is the HDFS architecture completely based on the
>> > > > Google
>> > > > > > > >> > Filesystem?
>> > > > > > > >> > >> If
>> > > > > > > >> > >> >> it
>> > > > > > > >> > >> >> > > isnt, what are the differences between the two?
>> > > > > > > >> > >> >> > >
>> > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
>> HDFS
>> > > same
>> > > > > as
>> > > > > > > how
>> > > > > > > >> it
>> > > > > > > >> > is
>> > > > > > > >> > >> >> > between
>> > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
>> > > > > > > >> > >> >> > >
>> > > > > > > >> > >> >> > > Amandeep
>> > > > > > > >> > >> >> > >
>> > > > > > > >> > >> >> > >
>> > > > > > > >> > >> >> > > Amandeep Khurana
>> > > > > > > >> > >> >> > > Computer Science Graduate Student
>> > > > > > > >> > >> >> > > University of California, Santa Cruz
>> > > > > > > >> > >> >> > >
>> > > > > > > >> > >> >> >
>> > > > > > > >> > >> >>
>> > > > > > > >> > >> >
>> > > > > > > >> > >> >
>> > > > > > > >> > >>
>> > > > > > > >> > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
--
M. Raşit ÖZDAŞ
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
In general, yeah, the scripts can access any resource they want (within the
permissions of the user that the task runs as). It's also possible to access
HDFS from scripts because HDFS provides a FUSE interface that can make it
look like a regular file system on the machine. (The FUSE module in turn
talks to the namenode as a regular HDFS client.)
On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com> wrote:
> I dont know much about Hadoop streaming and have a quick question here.
>
> The snippets of code/programs that you attach into the map reduce job might
> want to access outside resources (like you mentioned). Now these might not
> need to go to the namenode right? For example a python script. How would it
> access the data? Would it ask the parent java process (in the tasktracker)
> to get the data or would it go and do stuff on its own?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > Nope, typically the JobTracker just starts the process, and the
> tasktracker
> > talks directly to the namenode to get a pointer to the datanode, and then
> > directly to the datanode.
> >
> > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Alright.. Got it.
> > >
> > > Now, do the task trackers talk to the namenode and the data node
> directly
> > > or
> > > do they go through the job tracker for it? So, if my code is such that
> I
> > > need to access more files from the hdfs, would the job tracker get
> > involved
> > > or not?
> > >
> > >
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > > > Normally, HDFS files are accessed through the namenode. If there was
> a
> > > > malicious process though, then I imagine it could talk to a datanode
> > > > directly and request a specific block.
> > > >
> > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Ok. Got it.
> > > > >
> > > > > Now, when my job needs to access another file, does it go to the
> > > Namenode
> > > > > to
> > > > > get the block ids? How does the java process know where the files
> are
> > > and
> > > > > how to access them?
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <matei@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > > > I mentioned this case because even jobs written in Java can use
> the
> > > > HDFS
> > > > > > API
> > > > > > to talk to the NameNode and access the filesystem. People often
> do
> > > this
> > > > > > because their job needs to read a config file, some small data
> > table,
> > > > etc
> > > > > > and use this information in its map or reduce functions. In this
> > > case,
> > > > > you
> > > > > > open the second file separately in your mapper's init function
> and
> > > read
> > > > > > whatever you need from it. In general I wanted to point out that
> > you
> > > > > can't
> > > > > > know which files a job will access unless you look at its source
> > code
> > > > or
> > > > > > monitor the calls it makes; the input file(s) you provide in the
> > job
> > > > > > description are a hint to the MapReduce framework to place your
> job
> > > on
> > > > > > certain nodes, but it's reasonable for the job to access other
> > files
> > > as
> > > > > > well.
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Another question that I have here - When the jobs run arbitrary
> > > code
> > > > > and
> > > > > > > access data from the HDFS, do they go to the namenode to get
> the
> > > > block
> > > > > > > information?
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > > amansk@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Assuming that the job is purely in Java and not involving
> > > streaming
> > > > > or
> > > > > > > > pipes, wouldnt the resources (files) required by the job as
> > > inputs
> > > > be
> > > > > > > known
> > > > > > > > beforehand? So, if the map task is accessing a second file,
> how
> > > > does
> > > > > it
> > > > > > > make
> > > > > > > > it different except that there are multiple files. The
> > JobTracker
> > > > > would
> > > > > > > know
> > > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > > >
> > > > > > > > I am slightly confused why you have mentioned this case
> > > > separately...
> > > > > > Can
> > > > > > > > you elaborate on it a little bit?
> > > > > > > >
> > > > > > > > Amandeep
> > > > > > > >
> > > > > > > >
> > > > > > > > Amandeep Khurana
> > > > > > > > Computer Science Graduate Student
> > > > > > > > University of California, Santa Cruz
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > > matei@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Typically the data flow is like this:1) Client submits a job
> > > > > > description
> > > > > > > >> to
> > > > > > > >> the JobTracker.
> > > > > > > >> 2) JobTracker figures out block locations for the input
> > file(s)
> > > by
> > > > > > > talking
> > > > > > > >> to HDFS NameNode.
> > > > > > > >> 3) JobTracker creates a job description file in HDFS which
> > will
> > > be
> > > > > > read
> > > > > > > by
> > > > > > > >> the nodes to copy over the job's code etc.
> > > > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers)
> > with
> > > > the
> > > > > > > >> appropriate data blocks.
> > > > > > > >> 5) After running, maps create intermediate output files on
> > those
> > > > > > slaves.
> > > > > > > >> These are not in HDFS, they're in some temporary storage
> used
> > by
> > > > > > > >> MapReduce.
> > > > > > > >> 6) JobTracker starts reduces on a series of slaves, which
> copy
> > > > over
> > > > > > the
> > > > > > > >> appropriate map outputs, apply the reduce function, and
> write
> > > the
> > > > > > > outputs
> > > > > > > >> to
> > > > > > > >> HDFS (one output file per reducer).
> > > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > > > JobTracker.
> > > > > > > >>
> > > > > > > >> However, there is a big caveat, which is that the map and
> > reduce
> > > > > tasks
> > > > > > > run
> > > > > > > >> arbitrary code. It is not unusual to have a map that opens a
> > > > second
> > > > > > HDFS
> > > > > > > >> file to read some information (e.g. for doing a join of a
> > small
> > > > > table
> > > > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
> > > write
> > > > a
> > > > > > job
> > > > > > > >> in
> > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> > processes
> > > > > which
> > > > > > > may
> > > > > > > >> also access external resources in this manner. Some people
> > also
> > > > > > > read/write
> > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> security
> > > > > > solution
> > > > > > > >> would ideally deal with these cases too.
> > > > > > > >>
> > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > > amansk@gmail.com
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > A quick question here. How does a typical hadoop job work
> at
> > > the
> > > > > > > system
> > > > > > > >> > level? What are the various interactions and how does the
> > data
> > > > > flow?
> > > > > > > >> >
> > > > > > > >> > Amandeep
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Amandeep Khurana
> > > > > > > >> > Computer Science Graduate Student
> > > > > > > >> > University of California, Santa Cruz
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > > amansk@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Thanks Matei. If the basic architecture is similar to
> the
> > > > Google
> > > > > > > >> stuff, I
> > > > > > > >> > > can safely just work on the project using the
> information
> > > from
> > > > > the
> > > > > > > >> > papers.
> > > > > > > >> > >
> > > > > > > >> > > I am aware of the 4487 jira and the current status of
> the
> > > > > > > permissions
> > > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > > >> > >
> > > > > > > >> > > Cheers
> > > > > > > >> > > Amandeep
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Amandeep Khurana
> > > > > > > >> > > Computer Science Graduate Student
> > > > > > > >> > > University of California, Santa Cruz
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > > matei@cloudera.com>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> Forgot to add, this JIRA details the latest security
> > > features
> > > > > > that
> > > > > > > >> are
> > > > > > > >> > >> being
> > > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > > >> > >> This document describes the current status and
> > limitations
> > > of
> > > > > the
> > > > > > > >> > >> permissions mechanism:
> > > > > > > >> > >>
> > > > > > > >>
> > > > > >
> > > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > > .
> > > > > > > >> > >>
> > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > > matei@cloudera.com
> > > > > > > >
> > > > > > > >> > >> wrote:
> > > > > > > >> > >>
> > > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > > > MapReduce/GFS
> > > > > > > at
> > > > > > > >> > the
> > > > > > > >> > >> > level described in those papers. In particular, in
> > HDFS,
> > > > > there
> > > > > > is
> > > > > > > a
> > > > > > > >> > >> master
> > > > > > > >> > >> > node containing metadata and a number of slave nodes
> > > > > > (datanodes)
> > > > > > > >> > >> containing
> > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> > master
> > > > to
> > > > > > list
> > > > > > > >> > >> > directories, etc. When they want to read a region of
> > some
> > > > > file,
> > > > > > > >> they
> > > > > > > >> > >> tell
> > > > > > > >> > >> > the master the filename and offset, and they receive
> a
> > > list
> > > > > of
> > > > > > > >> block
> > > > > > > >> > >> > locations (datanodes). They then contact the
> individual
> > > > > > datanodes
> > > > > > > >> to
> > > > > > > >> > >> read
> > > > > > > >> > >> > the blocks. When clients write a file, they first
> > obtain
> > > a
> > > > > new
> > > > > > > >> block
> > > > > > > >> > ID
> > > > > > > >> > >> and
> > > > > > > >> > >> > list of nodes to write it to from the master, then
> > > contact
> > > > > the
> > > > > > > >> > datanodes
> > > > > > > >> > >> to
> > > > > > > >> > >> > write it (actually, the datanodes pipeline the write
> as
> > > in
> > > > > GFS)
> > > > > > > and
> > > > > > > >> > >> report
> > > > > > > >> > >> > when the write is complete. HDFS actually has some
> > > security
> > > > > > > >> mechanisms
> > > > > > > >> > >> built
> > > > > > > >> > >> > in, authenticating users based on their Unix ID and
> > > > providing
> > > > > > > >> > Unix-like
> > > > > > > >> > >> file
> > > > > > > >> > >> > permissions. I don't know much about how these are
> > > > > implemented,
> > > > > > > but
> > > > > > > >> > they
> > > > > > > >> > >> > would be a good place to start looking.
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > > > >> amansk@gmail.com
> > > > > > > >> > >> >wrote:
> > > > > > > >> > >> >
> > > > > > > >> > >> >> Thanks Matie
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> I had gone through the architecture document online.
> I
> > > am
> > > > > > > >> currently
> > > > > > > >> > >> >> working
> > > > > > > >> > >> >> on a project towards Security in Hadoop. I do know
> how
> > > the
> > > > > > data
> > > > > > > >> moves
> > > > > > > >> > >> >> around
> > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> > > > follow
> > > > > > and
> > > > > > > >> how
> > > > > > > >> > >> >> different it is from GFS. Can you throw some light
> on
> > > > that?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > > following
> > > > > the
> > > > > > > same
> > > > > > > >> > >> >> protocols. Thats why the question about how does the
> > > > Hadoop
> > > > > > > >> framework
> > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> from
> > > Map
> > > > > > Reduce
> > > > > > > >> and
> > > > > > > >> > >> GFS.
> > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> information
> > on
> > > > how
> > > > > > > those
> > > > > > > >> > >> systems
> > > > > > > >> > >> >> are designed but there is nothing that concrete for
> > > Hadoop
> > > > > > that
> > > > > > > I
> > > > > > > >> > have
> > > > > > > >> > >> >> been
> > > > > > > >> > >> >> able to find.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Amandeep
> > > > > > > >> > >> >>
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Amandeep Khurana
> > > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > > >> > >> >>
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > > > >> matei@cloudera.com>
> > > > > > > >> > >> >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> > Hi Amandeep,
> > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
> > > aims
> > > > to
> > > > > > > >> provide
> > > > > > > >> > >> those
> > > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> > > similar
> > > > to
> > > > > > GFS
> > > > > > > >> > (large
> > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> > missing
> > > > are
> > > > > > > >> > read-write
> > > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> > > provided
> > > > > > > because
> > > > > > > >> > few
> > > > > > > >> > >> >> Hadoop
> > > > > > > >> > >> >> > applications require it) and multiple appenders
> (the
> > > > > record
> > > > > > > >> append
> > > > > > > >> > >> >> > operation). You can read about HDFS architecture
> at
> > > > > > > >> > >> >> >
> > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > > .
> > > > > > > >> The
> > > > > > > >> > >> >> MapReduce
> > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
> > > that
> > > > > > > Google's
> > > > > > > >> > >> >> MapReduce
> > > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> > data),
> > > > > > > although
> > > > > > > >> > >> Hadoop
> > > > > > > >> > >> >> > MapReduce also supports running over other
> > distributed
> > > > > > > >> filesystems.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Matei
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana
> <
> > > > > > > >> > amansk@gmail.com
> > > > > > > >> > >> >
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > Hi
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Is the HDFS architecture completely based on the
> > > > Google
> > > > > > > >> > Filesystem?
> > > > > > > >> > >> If
> > > > > > > >> > >> >> it
> > > > > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> HDFS
> > > same
> > > > > as
> > > > > > > how
> > > > > > > >> it
> > > > > > > >> > is
> > > > > > > >> > >> >> > between
> > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Amandeep
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >>
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
In general, yeah, the scripts can access any resource they want (within the
permissions of the user that the task runs as). It's also possible to access
HDFS from scripts because HDFS provides a FUSE interface that can make it
look like a regular file system on the machine. (The FUSE module in turn
talks to the namenode as a regular HDFS client.)
On Sun, Feb 15, 2009 at 8:43 PM, Amandeep Khurana <am...@gmail.com> wrote:
> I dont know much about Hadoop streaming and have a quick question here.
>
> The snippets of code/programs that you attach into the map reduce job might
> want to access outside resources (like you mentioned). Now these might not
> need to go to the namenode right? For example a python script. How would it
> access the data? Would it ask the parent java process (in the tasktracker)
> to get the data or would it go and do stuff on its own?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > Nope, typically the JobTracker just starts the process, and the
> tasktracker
> > talks directly to the namenode to get a pointer to the datanode, and then
> > directly to the datanode.
> >
> > On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Alright.. Got it.
> > >
> > > Now, do the task trackers talk to the namenode and the data node
> directly
> > > or
> > > do they go through the job tracker for it? So, if my code is such that
> I
> > > need to access more files from the hdfs, would the job tracker get
> > involved
> > > or not?
> > >
> > >
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > > > Normally, HDFS files are accessed through the namenode. If there was
> a
> > > > malicious process though, then I imagine it could talk to a datanode
> > > > directly and request a specific block.
> > > >
> > > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Ok. Got it.
> > > > >
> > > > > Now, when my job needs to access another file, does it go to the
> > > Namenode
> > > > > to
> > > > > get the block ids? How does the java process know where the files
> are
> > > and
> > > > > how to access them?
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <matei@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > > > I mentioned this case because even jobs written in Java can use
> the
> > > > HDFS
> > > > > > API
> > > > > > to talk to the NameNode and access the filesystem. People often
> do
> > > this
> > > > > > because their job needs to read a config file, some small data
> > table,
> > > > etc
> > > > > > and use this information in its map or reduce functions. In this
> > > case,
> > > > > you
> > > > > > open the second file separately in your mapper's init function
> and
> > > read
> > > > > > whatever you need from it. In general I wanted to point out that
> > you
> > > > > can't
> > > > > > know which files a job will access unless you look at its source
> > code
> > > > or
> > > > > > monitor the calls it makes; the input file(s) you provide in the
> > job
> > > > > > description are a hint to the MapReduce framework to place your
> job
> > > on
> > > > > > certain nodes, but it's reasonable for the job to access other
> > files
> > > as
> > > > > > well.
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Another question that I have here - When the jobs run arbitrary
> > > code
> > > > > and
> > > > > > > access data from the HDFS, do they go to the namenode to get
> the
> > > > block
> > > > > > > information?
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > > amansk@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Assuming that the job is purely in Java and not involving
> > > streaming
> > > > > or
> > > > > > > > pipes, wouldnt the resources (files) required by the job as
> > > inputs
> > > > be
> > > > > > > known
> > > > > > > > beforehand? So, if the map task is accessing a second file,
> how
> > > > does
> > > > > it
> > > > > > > make
> > > > > > > > it different except that there are multiple files. The
> > JobTracker
> > > > > would
> > > > > > > know
> > > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > > >
> > > > > > > > I am slightly confused why you have mentioned this case
> > > > separately...
> > > > > > Can
> > > > > > > > you elaborate on it a little bit?
> > > > > > > >
> > > > > > > > Amandeep
> > > > > > > >
> > > > > > > >
> > > > > > > > Amandeep Khurana
> > > > > > > > Computer Science Graduate Student
> > > > > > > > University of California, Santa Cruz
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > > matei@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Typically the data flow is like this:1) Client submits a job
> > > > > > description
> > > > > > > >> to
> > > > > > > >> the JobTracker.
> > > > > > > >> 2) JobTracker figures out block locations for the input
> > file(s)
> > > by
> > > > > > > talking
> > > > > > > >> to HDFS NameNode.
> > > > > > > >> 3) JobTracker creates a job description file in HDFS which
> > will
> > > be
> > > > > > read
> > > > > > > by
> > > > > > > >> the nodes to copy over the job's code etc.
> > > > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers)
> > with
> > > > the
> > > > > > > >> appropriate data blocks.
> > > > > > > >> 5) After running, maps create intermediate output files on
> > those
> > > > > > slaves.
> > > > > > > >> These are not in HDFS, they're in some temporary storage
> used
> > by
> > > > > > > >> MapReduce.
> > > > > > > >> 6) JobTracker starts reduces on a series of slaves, which
> copy
> > > > over
> > > > > > the
> > > > > > > >> appropriate map outputs, apply the reduce function, and
> write
> > > the
> > > > > > > outputs
> > > > > > > >> to
> > > > > > > >> HDFS (one output file per reducer).
> > > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > > > JobTracker.
> > > > > > > >>
> > > > > > > >> However, there is a big caveat, which is that the map and
> > reduce
> > > > > tasks
> > > > > > > run
> > > > > > > >> arbitrary code. It is not unusual to have a map that opens a
> > > > second
> > > > > > HDFS
> > > > > > > >> file to read some information (e.g. for doing a join of a
> > small
> > > > > table
> > > > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
> > > write
> > > > a
> > > > > > job
> > > > > > > >> in
> > > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> > processes
> > > > > which
> > > > > > > may
> > > > > > > >> also access external resources in this manner. Some people
> > also
> > > > > > > read/write
> > > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive
> security
> > > > > > solution
> > > > > > > >> would ideally deal with these cases too.
> > > > > > > >>
> > > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > > amansk@gmail.com
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > A quick question here. How does a typical hadoop job work
> at
> > > the
> > > > > > > system
> > > > > > > >> > level? What are the various interactions and how does the
> > data
> > > > > flow?
> > > > > > > >> >
> > > > > > > >> > Amandeep
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Amandeep Khurana
> > > > > > > >> > Computer Science Graduate Student
> > > > > > > >> > University of California, Santa Cruz
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > > amansk@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Thanks Matei. If the basic architecture is similar to
> the
> > > > Google
> > > > > > > >> stuff, I
> > > > > > > >> > > can safely just work on the project using the
> information
> > > from
> > > > > the
> > > > > > > >> > papers.
> > > > > > > >> > >
> > > > > > > >> > > I am aware of the 4487 jira and the current status of
> the
> > > > > > > permissions
> > > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > > >> > >
> > > > > > > >> > > Cheers
> > > > > > > >> > > Amandeep
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Amandeep Khurana
> > > > > > > >> > > Computer Science Graduate Student
> > > > > > > >> > > University of California, Santa Cruz
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > > matei@cloudera.com>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> Forgot to add, this JIRA details the latest security
> > > features
> > > > > > that
> > > > > > > >> are
> > > > > > > >> > >> being
> > > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > > >> > >> This document describes the current status and
> > limitations
> > > of
> > > > > the
> > > > > > > >> > >> permissions mechanism:
> > > > > > > >> > >>
> > > > > > > >>
> > > > > >
> > > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > > .
> > > > > > > >> > >>
> > > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > > matei@cloudera.com
> > > > > > > >
> > > > > > > >> > >> wrote:
> > > > > > > >> > >>
> > > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > > > MapReduce/GFS
> > > > > > > at
> > > > > > > >> > the
> > > > > > > >> > >> > level described in those papers. In particular, in
> > HDFS,
> > > > > there
> > > > > > is
> > > > > > > a
> > > > > > > >> > >> master
> > > > > > > >> > >> > node containing metadata and a number of slave nodes
> > > > > > (datanodes)
> > > > > > > >> > >> containing
> > > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> > master
> > > > to
> > > > > > list
> > > > > > > >> > >> > directories, etc. When they want to read a region of
> > some
> > > > > file,
> > > > > > > >> they
> > > > > > > >> > >> tell
> > > > > > > >> > >> > the master the filename and offset, and they receive
> a
> > > list
> > > > > of
> > > > > > > >> block
> > > > > > > >> > >> > locations (datanodes). They then contact the
> individual
> > > > > > datanodes
> > > > > > > >> to
> > > > > > > >> > >> read
> > > > > > > >> > >> > the blocks. When clients write a file, they first
> > obtain
> > > a
> > > > > new
> > > > > > > >> block
> > > > > > > >> > ID
> > > > > > > >> > >> and
> > > > > > > >> > >> > list of nodes to write it to from the master, then
> > > contact
> > > > > the
> > > > > > > >> > datanodes
> > > > > > > >> > >> to
> > > > > > > >> > >> > write it (actually, the datanodes pipeline the write
> as
> > > in
> > > > > GFS)
> > > > > > > and
> > > > > > > >> > >> report
> > > > > > > >> > >> > when the write is complete. HDFS actually has some
> > > security
> > > > > > > >> mechanisms
> > > > > > > >> > >> built
> > > > > > > >> > >> > in, authenticating users based on their Unix ID and
> > > > providing
> > > > > > > >> > Unix-like
> > > > > > > >> > >> file
> > > > > > > >> > >> > permissions. I don't know much about how these are
> > > > > implemented,
> > > > > > > but
> > > > > > > >> > they
> > > > > > > >> > >> > would be a good place to start looking.
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > > > >> amansk@gmail.com
> > > > > > > >> > >> >wrote:
> > > > > > > >> > >> >
> > > > > > > >> > >> >> Thanks Matie
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> I had gone through the architecture document online.
> I
> > > am
> > > > > > > >> currently
> > > > > > > >> > >> >> working
> > > > > > > >> > >> >> on a project towards Security in Hadoop. I do know
> how
> > > the
> > > > > > data
> > > > > > > >> moves
> > > > > > > >> > >> >> around
> > > > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> > > > follow
> > > > > > and
> > > > > > > >> how
> > > > > > > >> > >> >> different it is from GFS. Can you throw some light
> on
> > > > that?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > > following
> > > > > the
> > > > > > > same
> > > > > > > >> > >> >> protocols. Thats why the question about how does the
> > > > Hadoop
> > > > > > > >> framework
> > > > > > > >> > >> >> integrate with the HDFS, and how different is it
> from
> > > Map
> > > > > > Reduce
> > > > > > > >> and
> > > > > > > >> > >> GFS.
> > > > > > > >> > >> >> The GFS and Map Reduce papers give a good
> information
> > on
> > > > how
> > > > > > > those
> > > > > > > >> > >> systems
> > > > > > > >> > >> >> are designed but there is nothing that concrete for
> > > Hadoop
> > > > > > that
> > > > > > > I
> > > > > > > >> > have
> > > > > > > >> > >> >> been
> > > > > > > >> > >> >> able to find.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Amandeep
> > > > > > > >> > >> >>
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Amandeep Khurana
> > > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > > >> > >> >>
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > > > >> matei@cloudera.com>
> > > > > > > >> > >> >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> > Hi Amandeep,
> > > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
> > > aims
> > > > to
> > > > > > > >> provide
> > > > > > > >> > >> those
> > > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> > > similar
> > > > to
> > > > > > GFS
> > > > > > > >> > (large
> > > > > > > >> > >> >> > blocks, replication, etc); some notable things
> > missing
> > > > are
> > > > > > > >> > read-write
> > > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> > > provided
> > > > > > > because
> > > > > > > >> > few
> > > > > > > >> > >> >> Hadoop
> > > > > > > >> > >> >> > applications require it) and multiple appenders
> (the
> > > > > record
> > > > > > > >> append
> > > > > > > >> > >> >> > operation). You can read about HDFS architecture
> at
> > > > > > > >> > >> >> >
> > > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > > .
> > > > > > > >> The
> > > > > > > >> > >> >> MapReduce
> > > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
> > > that
> > > > > > > Google's
> > > > > > > >> > >> >> MapReduce
> > > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> > data),
> > > > > > > although
> > > > > > > >> > >> Hadoop
> > > > > > > >> > >> >> > MapReduce also supports running over other
> > distributed
> > > > > > > >> filesystems.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Matei
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana
> <
> > > > > > > >> > amansk@gmail.com
> > > > > > > >> > >> >
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > Hi
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Is the HDFS architecture completely based on the
> > > > Google
> > > > > > > >> > Filesystem?
> > > > > > > >> > >> If
> > > > > > > >> > >> >> it
> > > > > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and
> HDFS
> > > same
> > > > > as
> > > > > > > how
> > > > > > > >> it
> > > > > > > >> > is
> > > > > > > >> > >> >> > between
> > > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Amandeep
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >>
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
I dont know much about Hadoop streaming and have a quick question here.
The snippets of code/programs that you attach into the map reduce job might
want to access outside resources (like you mentioned). Now these might not
need to go to the namenode right? For example a python script. How would it
access the data? Would it ask the parent java process (in the tasktracker)
to get the data or would it go and do stuff on its own?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Nope, typically the JobTracker just starts the process, and the tasktracker
> talks directly to the namenode to get a pointer to the datanode, and then
> directly to the datanode.
>
> On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Alright.. Got it.
> >
> > Now, do the task trackers talk to the namenode and the data node directly
> > or
> > do they go through the job tracker for it? So, if my code is such that I
> > need to access more files from the hdfs, would the job tracker get
> involved
> > or not?
> >
> >
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> > > Normally, HDFS files are accessed through the namenode. If there was a
> > > malicious process though, then I imagine it could talk to a datanode
> > > directly and request a specific block.
> > >
> > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Ok. Got it.
> > > >
> > > > Now, when my job needs to access another file, does it go to the
> > Namenode
> > > > to
> > > > get the block ids? How does the java process know where the files are
> > and
> > > > how to access them?
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
> > > wrote:
> > > >
> > > > > I mentioned this case because even jobs written in Java can use the
> > > HDFS
> > > > > API
> > > > > to talk to the NameNode and access the filesystem. People often do
> > this
> > > > > because their job needs to read a config file, some small data
> table,
> > > etc
> > > > > and use this information in its map or reduce functions. In this
> > case,
> > > > you
> > > > > open the second file separately in your mapper's init function and
> > read
> > > > > whatever you need from it. In general I wanted to point out that
> you
> > > > can't
> > > > > know which files a job will access unless you look at its source
> code
> > > or
> > > > > monitor the calls it makes; the input file(s) you provide in the
> job
> > > > > description are a hint to the MapReduce framework to place your job
> > on
> > > > > certain nodes, but it's reasonable for the job to access other
> files
> > as
> > > > > well.
> > > > >
> > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Another question that I have here - When the jobs run arbitrary
> > code
> > > > and
> > > > > > access data from the HDFS, do they go to the namenode to get the
> > > block
> > > > > > information?
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Assuming that the job is purely in Java and not involving
> > streaming
> > > > or
> > > > > > > pipes, wouldnt the resources (files) required by the job as
> > inputs
> > > be
> > > > > > known
> > > > > > > beforehand? So, if the map task is accessing a second file, how
> > > does
> > > > it
> > > > > > make
> > > > > > > it different except that there are multiple files. The
> JobTracker
> > > > would
> > > > > > know
> > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > >
> > > > > > > I am slightly confused why you have mentioned this case
> > > separately...
> > > > > Can
> > > > > > > you elaborate on it a little bit?
> > > > > > >
> > > > > > > Amandeep
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > matei@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Typically the data flow is like this:1) Client submits a job
> > > > > description
> > > > > > >> to
> > > > > > >> the JobTracker.
> > > > > > >> 2) JobTracker figures out block locations for the input
> file(s)
> > by
> > > > > > talking
> > > > > > >> to HDFS NameNode.
> > > > > > >> 3) JobTracker creates a job description file in HDFS which
> will
> > be
> > > > > read
> > > > > > by
> > > > > > >> the nodes to copy over the job's code etc.
> > > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers)
> with
> > > the
> > > > > > >> appropriate data blocks.
> > > > > > >> 5) After running, maps create intermediate output files on
> those
> > > > > slaves.
> > > > > > >> These are not in HDFS, they're in some temporary storage used
> by
> > > > > > >> MapReduce.
> > > > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
> > > over
> > > > > the
> > > > > > >> appropriate map outputs, apply the reduce function, and write
> > the
> > > > > > outputs
> > > > > > >> to
> > > > > > >> HDFS (one output file per reducer).
> > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > > JobTracker.
> > > > > > >>
> > > > > > >> However, there is a big caveat, which is that the map and
> reduce
> > > > tasks
> > > > > > run
> > > > > > >> arbitrary code. It is not unusual to have a map that opens a
> > > second
> > > > > HDFS
> > > > > > >> file to read some information (e.g. for doing a join of a
> small
> > > > table
> > > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
> > write
> > > a
> > > > > job
> > > > > > >> in
> > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> processes
> > > > which
> > > > > > may
> > > > > > >> also access external resources in this manner. Some people
> also
> > > > > > read/write
> > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > > > > solution
> > > > > > >> would ideally deal with these cases too.
> > > > > > >>
> > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > amansk@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > A quick question here. How does a typical hadoop job work at
> > the
> > > > > > system
> > > > > > >> > level? What are the various interactions and how does the
> data
> > > > flow?
> > > > > > >> >
> > > > > > >> > Amandeep
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Amandeep Khurana
> > > > > > >> > Computer Science Graduate Student
> > > > > > >> > University of California, Santa Cruz
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > amansk@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Thanks Matei. If the basic architecture is similar to the
> > > Google
> > > > > > >> stuff, I
> > > > > > >> > > can safely just work on the project using the information
> > from
> > > > the
> > > > > > >> > papers.
> > > > > > >> > >
> > > > > > >> > > I am aware of the 4487 jira and the current status of the
> > > > > > permissions
> > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > >> > >
> > > > > > >> > > Cheers
> > > > > > >> > > Amandeep
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Amandeep Khurana
> > > > > > >> > > Computer Science Graduate Student
> > > > > > >> > > University of California, Santa Cruz
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > matei@cloudera.com>
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > >> Forgot to add, this JIRA details the latest security
> > features
> > > > > that
> > > > > > >> are
> > > > > > >> > >> being
> > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > >> > >> This document describes the current status and
> limitations
> > of
> > > > the
> > > > > > >> > >> permissions mechanism:
> > > > > > >> > >>
> > > > > > >>
> > > > >
> > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > .
> > > > > > >> > >>
> > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > matei@cloudera.com
> > > > > > >
> > > > > > >> > >> wrote:
> > > > > > >> > >>
> > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > > MapReduce/GFS
> > > > > > at
> > > > > > >> > the
> > > > > > >> > >> > level described in those papers. In particular, in
> HDFS,
> > > > there
> > > > > is
> > > > > > a
> > > > > > >> > >> master
> > > > > > >> > >> > node containing metadata and a number of slave nodes
> > > > > (datanodes)
> > > > > > >> > >> containing
> > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> master
> > > to
> > > > > list
> > > > > > >> > >> > directories, etc. When they want to read a region of
> some
> > > > file,
> > > > > > >> they
> > > > > > >> > >> tell
> > > > > > >> > >> > the master the filename and offset, and they receive a
> > list
> > > > of
> > > > > > >> block
> > > > > > >> > >> > locations (datanodes). They then contact the individual
> > > > > datanodes
> > > > > > >> to
> > > > > > >> > >> read
> > > > > > >> > >> > the blocks. When clients write a file, they first
> obtain
> > a
> > > > new
> > > > > > >> block
> > > > > > >> > ID
> > > > > > >> > >> and
> > > > > > >> > >> > list of nodes to write it to from the master, then
> > contact
> > > > the
> > > > > > >> > datanodes
> > > > > > >> > >> to
> > > > > > >> > >> > write it (actually, the datanodes pipeline the write as
> > in
> > > > GFS)
> > > > > > and
> > > > > > >> > >> report
> > > > > > >> > >> > when the write is complete. HDFS actually has some
> > security
> > > > > > >> mechanisms
> > > > > > >> > >> built
> > > > > > >> > >> > in, authenticating users based on their Unix ID and
> > > providing
> > > > > > >> > Unix-like
> > > > > > >> > >> file
> > > > > > >> > >> > permissions. I don't know much about how these are
> > > > implemented,
> > > > > > but
> > > > > > >> > they
> > > > > > >> > >> > would be a good place to start looking.
> > > > > > >> > >> >
> > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > > >> amansk@gmail.com
> > > > > > >> > >> >wrote:
> > > > > > >> > >> >
> > > > > > >> > >> >> Thanks Matie
> > > > > > >> > >> >>
> > > > > > >> > >> >> I had gone through the architecture document online. I
> > am
> > > > > > >> currently
> > > > > > >> > >> >> working
> > > > > > >> > >> >> on a project towards Security in Hadoop. I do know how
> > the
> > > > > data
> > > > > > >> moves
> > > > > > >> > >> >> around
> > > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> > > follow
> > > > > and
> > > > > > >> how
> > > > > > >> > >> >> different it is from GFS. Can you throw some light on
> > > that?
> > > > > > >> > >> >>
> > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > following
> > > > the
> > > > > > same
> > > > > > >> > >> >> protocols. Thats why the question about how does the
> > > Hadoop
> > > > > > >> framework
> > > > > > >> > >> >> integrate with the HDFS, and how different is it from
> > Map
> > > > > Reduce
> > > > > > >> and
> > > > > > >> > >> GFS.
> > > > > > >> > >> >> The GFS and Map Reduce papers give a good information
> on
> > > how
> > > > > > those
> > > > > > >> > >> systems
> > > > > > >> > >> >> are designed but there is nothing that concrete for
> > Hadoop
> > > > > that
> > > > > > I
> > > > > > >> > have
> > > > > > >> > >> >> been
> > > > > > >> > >> >> able to find.
> > > > > > >> > >> >>
> > > > > > >> > >> >> Amandeep
> > > > > > >> > >> >>
> > > > > > >> > >> >>
> > > > > > >> > >> >> Amandeep Khurana
> > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > >> > >> >>
> > > > > > >> > >> >>
> > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > > >> matei@cloudera.com>
> > > > > > >> > >> >> wrote:
> > > > > > >> > >> >>
> > > > > > >> > >> >> > Hi Amandeep,
> > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
> > aims
> > > to
> > > > > > >> provide
> > > > > > >> > >> those
> > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> > similar
> > > to
> > > > > GFS
> > > > > > >> > (large
> > > > > > >> > >> >> > blocks, replication, etc); some notable things
> missing
> > > are
> > > > > > >> > read-write
> > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> > provided
> > > > > > because
> > > > > > >> > few
> > > > > > >> > >> >> Hadoop
> > > > > > >> > >> >> > applications require it) and multiple appenders (the
> > > > record
> > > > > > >> append
> > > > > > >> > >> >> > operation). You can read about HDFS architecture at
> > > > > > >> > >> >> >
> > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > .
> > > > > > >> The
> > > > > > >> > >> >> MapReduce
> > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
> > that
> > > > > > Google's
> > > > > > >> > >> >> MapReduce
> > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> data),
> > > > > > although
> > > > > > >> > >> Hadoop
> > > > > > >> > >> >> > MapReduce also supports running over other
> distributed
> > > > > > >> filesystems.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Matei
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > > > > >> > amansk@gmail.com
> > > > > > >> > >> >
> > > > > > >> > >> >> > wrote:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > > Hi
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Is the HDFS architecture completely based on the
> > > Google
> > > > > > >> > Filesystem?
> > > > > > >> > >> If
> > > > > > >> > >> >> it
> > > > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS
> > same
> > > > as
> > > > > > how
> > > > > > >> it
> > > > > > >> > is
> > > > > > >> > >> >> > between
> > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Amandeep
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> >
> > > > > > >> > >> >>
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >>
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Steve Loughran <st...@apache.org>.
kang_min82 wrote:
> Hello Matei,
>
> Which Tasktracker did you mean here ?
>
> I don't understand that. In general we have mane Tasktrackers and each of
> them runs on one separate Datanode. Why doesn't the JobTracker talk directly
> to the Namenode for a list of Datanodes and then performs the MapReduce
> tasks there.
1. There's no requirement for a 1:1 mapping of task-trackers to
datanodes. You could bring up TT's on any machine with spare CPU cycles
on your network, talking to a long lived filesystem built from a few
datanodes
2. There's no requirement for HDFS. You could have a cluster of
MapReduce nodes talking to other filesystems. Locality of data helps,
but is not needed.
3. Layering makes for cleaner code.
Re: HDFS architecture based on GFS?
Posted by kang_min82 <ka...@yahoo.com>.
Hello Matei,
Which Tasktracker did you mean here ?
I don't understand that. In general we have mane Tasktrackers and each of
them runs on one separate Datanode. Why doesn't the JobTracker talk directly
to the Namenode for a list of Datanodes and then performs the MapReduce
tasks there.
Any help is appreciated.
Kang
Matei Zaharia-2 wrote:
>
> Nope, typically the JobTracker just starts the process, and the
> tasktracker
> talks directly to the namenode to get a pointer to the datanode, and then
> directly to the datanode.
>
> On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
>> Alright.. Got it.
>>
>> Now, do the task trackers talk to the namenode and the data node directly
>> or
>> do they go through the job tracker for it? So, if my code is such that I
>> need to access more files from the hdfs, would the job tracker get
>> involved
>> or not?
>>
>>
>>
>>
>> Amandeep Khurana
>> Computer Science Graduate Student
>> University of California, Santa Cruz
>>
>>
>> On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
>> wrote:
>>
>> > Normally, HDFS files are accessed through the namenode. If there was a
>> > malicious process though, then I imagine it could talk to a datanode
>> > directly and request a specific block.
>> >
>> > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
>> > wrote:
>> >
>> > > Ok. Got it.
>> > >
>> > > Now, when my job needs to access another file, does it go to the
>> Namenode
>> > > to
>> > > get the block ids? How does the java process know where the files are
>> and
>> > > how to access them?
>> > >
>> > >
>> > > Amandeep Khurana
>> > > Computer Science Graduate Student
>> > > University of California, Santa Cruz
>> > >
>> > >
>> > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
>> > wrote:
>> > >
>> > > > I mentioned this case because even jobs written in Java can use the
>> > HDFS
>> > > > API
>> > > > to talk to the NameNode and access the filesystem. People often do
>> this
>> > > > because their job needs to read a config file, some small data
>> table,
>> > etc
>> > > > and use this information in its map or reduce functions. In this
>> case,
>> > > you
>> > > > open the second file separately in your mapper's init function and
>> read
>> > > > whatever you need from it. In general I wanted to point out that
>> you
>> > > can't
>> > > > know which files a job will access unless you look at its source
>> code
>> > or
>> > > > monitor the calls it makes; the input file(s) you provide in the
>> job
>> > > > description are a hint to the MapReduce framework to place your job
>> on
>> > > > certain nodes, but it's reasonable for the job to access other
>> files
>> as
>> > > > well.
>> > > >
>> > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana
>> <am...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Another question that I have here - When the jobs run arbitrary
>> code
>> > > and
>> > > > > access data from the HDFS, do they go to the namenode to get the
>> > block
>> > > > > information?
>> > > > >
>> > > > >
>> > > > > Amandeep Khurana
>> > > > > Computer Science Graduate Student
>> > > > > University of California, Santa Cruz
>> > > > >
>> > > > >
>> > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
>> amansk@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Assuming that the job is purely in Java and not involving
>> streaming
>> > > or
>> > > > > > pipes, wouldnt the resources (files) required by the job as
>> inputs
>> > be
>> > > > > known
>> > > > > > beforehand? So, if the map task is accessing a second file, how
>> > does
>> > > it
>> > > > > make
>> > > > > > it different except that there are multiple files. The
>> JobTracker
>> > > would
>> > > > > know
>> > > > > > beforehand that multiple files would be accessed. Right?
>> > > > > >
>> > > > > > I am slightly confused why you have mentioned this case
>> > separately...
>> > > > Can
>> > > > > > you elaborate on it a little bit?
>> > > > > >
>> > > > > > Amandeep
>> > > > > >
>> > > > > >
>> > > > > > Amandeep Khurana
>> > > > > > Computer Science Graduate Student
>> > > > > > University of California, Santa Cruz
>> > > > > >
>> > > > > >
>> > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
>> matei@cloudera.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > >> Typically the data flow is like this:1) Client submits a job
>> > > > description
>> > > > > >> to
>> > > > > >> the JobTracker.
>> > > > > >> 2) JobTracker figures out block locations for the input
>> file(s)
>> by
>> > > > > talking
>> > > > > >> to HDFS NameNode.
>> > > > > >> 3) JobTracker creates a job description file in HDFS which
>> will
>> be
>> > > > read
>> > > > > by
>> > > > > >> the nodes to copy over the job's code etc.
>> > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers)
>> with
>> > the
>> > > > > >> appropriate data blocks.
>> > > > > >> 5) After running, maps create intermediate output files on
>> those
>> > > > slaves.
>> > > > > >> These are not in HDFS, they're in some temporary storage used
>> by
>> > > > > >> MapReduce.
>> > > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
>> > over
>> > > > the
>> > > > > >> appropriate map outputs, apply the reduce function, and write
>> the
>> > > > > outputs
>> > > > > >> to
>> > > > > >> HDFS (one output file per reducer).
>> > > > > >> 7) Some logs for the job may also be put into HDFS by the
>> > > JobTracker.
>> > > > > >>
>> > > > > >> However, there is a big caveat, which is that the map and
>> reduce
>> > > tasks
>> > > > > run
>> > > > > >> arbitrary code. It is not unusual to have a map that opens a
>> > second
>> > > > HDFS
>> > > > > >> file to read some information (e.g. for doing a join of a
>> small
>> > > table
>> > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
>> write
>> > a
>> > > > job
>> > > > > >> in
>> > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
>> processes
>> > > which
>> > > > > may
>> > > > > >> also access external resources in this manner. Some people
>> also
>> > > > > read/write
>> > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
>> > > > solution
>> > > > > >> would ideally deal with these cases too.
>> > > > > >>
>> > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
>> > amansk@gmail.com
>> > > >
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >> > A quick question here. How does a typical hadoop job work at
>> the
>> > > > > system
>> > > > > >> > level? What are the various interactions and how does the
>> data
>> > > flow?
>> > > > > >> >
>> > > > > >> > Amandeep
>> > > > > >> >
>> > > > > >> >
>> > > > > >> > Amandeep Khurana
>> > > > > >> > Computer Science Graduate Student
>> > > > > >> > University of California, Santa Cruz
>> > > > > >> >
>> > > > > >> >
>> > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
>> > > amansk@gmail.com
>> > > > >
>> > > > > >> > wrote:
>> > > > > >> >
>> > > > > >> > > Thanks Matei. If the basic architecture is similar to the
>> > Google
>> > > > > >> stuff, I
>> > > > > >> > > can safely just work on the project using the information
>> from
>> > > the
>> > > > > >> > papers.
>> > > > > >> > >
>> > > > > >> > > I am aware of the 4487 jira and the current status of the
>> > > > > permissions
>> > > > > >> > > mechanism. I had a look at them earlier.
>> > > > > >> > >
>> > > > > >> > > Cheers
>> > > > > >> > > Amandeep
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > Amandeep Khurana
>> > > > > >> > > Computer Science Graduate Student
>> > > > > >> > > University of California, Santa Cruz
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
>> > > > matei@cloudera.com>
>> > > > > >> > wrote:
>> > > > > >> > >
>> > > > > >> > >> Forgot to add, this JIRA details the latest security
>> features
>> > > > that
>> > > > > >> are
>> > > > > >> > >> being
>> > > > > >> > >> worked on in Hadoop trunk:
>> > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
>> > > > > >> > >> This document describes the current status and
>> limitations
>> of
>> > > the
>> > > > > >> > >> permissions mechanism:
>> > > > > >> > >>
>> > > > > >>
>> > > >
>> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
>> > .
>> > > > > >> > >>
>> > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
>> > > > matei@cloudera.com
>> > > > > >
>> > > > > >> > >> wrote:
>> > > > > >> > >>
>> > > > > >> > >> > I think it's safe to assume that Hadoop works like
>> > > > MapReduce/GFS
>> > > > > at
>> > > > > >> > the
>> > > > > >> > >> > level described in those papers. In particular, in
>> HDFS,
>> > > there
>> > > > is
>> > > > > a
>> > > > > >> > >> master
>> > > > > >> > >> > node containing metadata and a number of slave nodes
>> > > > (datanodes)
>> > > > > >> > >> containing
>> > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
>> master
>> > to
>> > > > list
>> > > > > >> > >> > directories, etc. When they want to read a region of
>> some
>> > > file,
>> > > > > >> they
>> > > > > >> > >> tell
>> > > > > >> > >> > the master the filename and offset, and they receive a
>> list
>> > > of
>> > > > > >> block
>> > > > > >> > >> > locations (datanodes). They then contact the individual
>> > > > datanodes
>> > > > > >> to
>> > > > > >> > >> read
>> > > > > >> > >> > the blocks. When clients write a file, they first
>> obtain
>> a
>> > > new
>> > > > > >> block
>> > > > > >> > ID
>> > > > > >> > >> and
>> > > > > >> > >> > list of nodes to write it to from the master, then
>> contact
>> > > the
>> > > > > >> > datanodes
>> > > > > >> > >> to
>> > > > > >> > >> > write it (actually, the datanodes pipeline the write as
>> in
>> > > GFS)
>> > > > > and
>> > > > > >> > >> report
>> > > > > >> > >> > when the write is complete. HDFS actually has some
>> security
>> > > > > >> mechanisms
>> > > > > >> > >> built
>> > > > > >> > >> > in, authenticating users based on their Unix ID and
>> > providing
>> > > > > >> > Unix-like
>> > > > > >> > >> file
>> > > > > >> > >> > permissions. I don't know much about how these are
>> > > implemented,
>> > > > > but
>> > > > > >> > they
>> > > > > >> > >> > would be a good place to start looking.
>> > > > > >> > >> >
>> > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
>> > > > > >> amansk@gmail.com
>> > > > > >> > >> >wrote:
>> > > > > >> > >> >
>> > > > > >> > >> >> Thanks Matie
>> > > > > >> > >> >>
>> > > > > >> > >> >> I had gone through the architecture document online. I
>> am
>> > > > > >> currently
>> > > > > >> > >> >> working
>> > > > > >> > >> >> on a project towards Security in Hadoop. I do know how
>> the
>> > > > data
>> > > > > >> moves
>> > > > > >> > >> >> around
>> > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
>> > follow
>> > > > and
>> > > > > >> how
>> > > > > >> > >> >> different it is from GFS. Can you throw some light on
>> > that?
>> > > > > >> > >> >>
>> > > > > >> > >> >> Security would also involve the Map Reduce jobs
>> following
>> > > the
>> > > > > same
>> > > > > >> > >> >> protocols. Thats why the question about how does the
>> > Hadoop
>> > > > > >> framework
>> > > > > >> > >> >> integrate with the HDFS, and how different is it from
>> Map
>> > > > Reduce
>> > > > > >> and
>> > > > > >> > >> GFS.
>> > > > > >> > >> >> The GFS and Map Reduce papers give a good information
>> on
>> > how
>> > > > > those
>> > > > > >> > >> systems
>> > > > > >> > >> >> are designed but there is nothing that concrete for
>> Hadoop
>> > > > that
>> > > > > I
>> > > > > >> > have
>> > > > > >> > >> >> been
>> > > > > >> > >> >> able to find.
>> > > > > >> > >> >>
>> > > > > >> > >> >> Amandeep
>> > > > > >> > >> >>
>> > > > > >> > >> >>
>> > > > > >> > >> >> Amandeep Khurana
>> > > > > >> > >> >> Computer Science Graduate Student
>> > > > > >> > >> >> University of California, Santa Cruz
>> > > > > >> > >> >>
>> > > > > >> > >> >>
>> > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
>> > > > > >> matei@cloudera.com>
>> > > > > >> > >> >> wrote:
>> > > > > >> > >> >>
>> > > > > >> > >> >> > Hi Amandeep,
>> > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
>> aims
>> > to
>> > > > > >> provide
>> > > > > >> > >> those
>> > > > > >> > >> >> > capabilities as an open-source project. HDFS is
>> similar
>> > to
>> > > > GFS
>> > > > > >> > (large
>> > > > > >> > >> >> > blocks, replication, etc); some notable things
>> missing
>> > are
>> > > > > >> > read-write
>> > > > > >> > >> >> > support in the middle of a file (unlikely to be
>> provided
>> > > > > because
>> > > > > >> > few
>> > > > > >> > >> >> Hadoop
>> > > > > >> > >> >> > applications require it) and multiple appenders (the
>> > > record
>> > > > > >> append
>> > > > > >> > >> >> > operation). You can read about HDFS architecture at
>> > > > > >> > >> >> >
>> > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
>> > > > .
>> > > > > >> The
>> > > > > >> > >> >> MapReduce
>> > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
>> that
>> > > > > Google's
>> > > > > >> > >> >> MapReduce
>> > > > > >> > >> >> > interacts with GFS (shipping computation to the
>> data),
>> > > > > although
>> > > > > >> > >> Hadoop
>> > > > > >> > >> >> > MapReduce also supports running over other
>> distributed
>> > > > > >> filesystems.
>> > > > > >> > >> >> >
>> > > > > >> > >> >> > Matei
>> > > > > >> > >> >> >
>> > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
>> > > > > >> > amansk@gmail.com
>> > > > > >> > >> >
>> > > > > >> > >> >> > wrote:
>> > > > > >> > >> >> >
>> > > > > >> > >> >> > > Hi
>> > > > > >> > >> >> > >
>> > > > > >> > >> >> > > Is the HDFS architecture completely based on the
>> > Google
>> > > > > >> > Filesystem?
>> > > > > >> > >> If
>> > > > > >> > >> >> it
>> > > > > >> > >> >> > > isnt, what are the differences between the two?
>> > > > > >> > >> >> > >
>> > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS
>> same
>> > > as
>> > > > > how
>> > > > > >> it
>> > > > > >> > is
>> > > > > >> > >> >> > between
>> > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
>> > > > > >> > >> >> > >
>> > > > > >> > >> >> > > Amandeep
>> > > > > >> > >> >> > >
>> > > > > >> > >> >> > >
>> > > > > >> > >> >> > > Amandeep Khurana
>> > > > > >> > >> >> > > Computer Science Graduate Student
>> > > > > >> > >> >> > > University of California, Santa Cruz
>> > > > > >> > >> >> > >
>> > > > > >> > >> >> >
>> > > > > >> > >> >>
>> > > > > >> > >> >
>> > > > > >> > >> >
>> > > > > >> > >>
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
--
View this message in context: http://www.nabble.com/HDFS-architecture-based-on-GFS--tp22026711p22237160.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
I dont know much about Hadoop streaming and have a quick question here.
The snippets of code/programs that you attach into the map reduce job might
want to access outside resources (like you mentioned). Now these might not
need to go to the namenode right? For example a python script. How would it
access the data? Would it ask the parent java process (in the tasktracker)
to get the data or would it go and do stuff on its own?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 8:23 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Nope, typically the JobTracker just starts the process, and the tasktracker
> talks directly to the namenode to get a pointer to the datanode, and then
> directly to the datanode.
>
> On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Alright.. Got it.
> >
> > Now, do the task trackers talk to the namenode and the data node directly
> > or
> > do they go through the job tracker for it? So, if my code is such that I
> > need to access more files from the hdfs, would the job tracker get
> involved
> > or not?
> >
> >
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> > > Normally, HDFS files are accessed through the namenode. If there was a
> > > malicious process though, then I imagine it could talk to a datanode
> > > directly and request a specific block.
> > >
> > > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Ok. Got it.
> > > >
> > > > Now, when my job needs to access another file, does it go to the
> > Namenode
> > > > to
> > > > get the block ids? How does the java process know where the files are
> > and
> > > > how to access them?
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
> > > wrote:
> > > >
> > > > > I mentioned this case because even jobs written in Java can use the
> > > HDFS
> > > > > API
> > > > > to talk to the NameNode and access the filesystem. People often do
> > this
> > > > > because their job needs to read a config file, some small data
> table,
> > > etc
> > > > > and use this information in its map or reduce functions. In this
> > case,
> > > > you
> > > > > open the second file separately in your mapper's init function and
> > read
> > > > > whatever you need from it. In general I wanted to point out that
> you
> > > > can't
> > > > > know which files a job will access unless you look at its source
> code
> > > or
> > > > > monitor the calls it makes; the input file(s) you provide in the
> job
> > > > > description are a hint to the MapReduce framework to place your job
> > on
> > > > > certain nodes, but it's reasonable for the job to access other
> files
> > as
> > > > > well.
> > > > >
> > > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Another question that I have here - When the jobs run arbitrary
> > code
> > > > and
> > > > > > access data from the HDFS, do they go to the namenode to get the
> > > block
> > > > > > information?
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> > amansk@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Assuming that the job is purely in Java and not involving
> > streaming
> > > > or
> > > > > > > pipes, wouldnt the resources (files) required by the job as
> > inputs
> > > be
> > > > > > known
> > > > > > > beforehand? So, if the map task is accessing a second file, how
> > > does
> > > > it
> > > > > > make
> > > > > > > it different except that there are multiple files. The
> JobTracker
> > > > would
> > > > > > know
> > > > > > > beforehand that multiple files would be accessed. Right?
> > > > > > >
> > > > > > > I am slightly confused why you have mentioned this case
> > > separately...
> > > > > Can
> > > > > > > you elaborate on it a little bit?
> > > > > > >
> > > > > > > Amandeep
> > > > > > >
> > > > > > >
> > > > > > > Amandeep Khurana
> > > > > > > Computer Science Graduate Student
> > > > > > > University of California, Santa Cruz
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> > matei@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Typically the data flow is like this:1) Client submits a job
> > > > > description
> > > > > > >> to
> > > > > > >> the JobTracker.
> > > > > > >> 2) JobTracker figures out block locations for the input
> file(s)
> > by
> > > > > > talking
> > > > > > >> to HDFS NameNode.
> > > > > > >> 3) JobTracker creates a job description file in HDFS which
> will
> > be
> > > > > read
> > > > > > by
> > > > > > >> the nodes to copy over the job's code etc.
> > > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers)
> with
> > > the
> > > > > > >> appropriate data blocks.
> > > > > > >> 5) After running, maps create intermediate output files on
> those
> > > > > slaves.
> > > > > > >> These are not in HDFS, they're in some temporary storage used
> by
> > > > > > >> MapReduce.
> > > > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
> > > over
> > > > > the
> > > > > > >> appropriate map outputs, apply the reduce function, and write
> > the
> > > > > > outputs
> > > > > > >> to
> > > > > > >> HDFS (one output file per reducer).
> > > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > > JobTracker.
> > > > > > >>
> > > > > > >> However, there is a big caveat, which is that the map and
> reduce
> > > > tasks
> > > > > > run
> > > > > > >> arbitrary code. It is not unusual to have a map that opens a
> > > second
> > > > > HDFS
> > > > > > >> file to read some information (e.g. for doing a join of a
> small
> > > > table
> > > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
> > write
> > > a
> > > > > job
> > > > > > >> in
> > > > > > >> Python, Ruby, C, etc, then you are launching arbitrary
> processes
> > > > which
> > > > > > may
> > > > > > >> also access external resources in this manner. Some people
> also
> > > > > > read/write
> > > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > > > > solution
> > > > > > >> would ideally deal with these cases too.
> > > > > > >>
> > > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > > amansk@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > A quick question here. How does a typical hadoop job work at
> > the
> > > > > > system
> > > > > > >> > level? What are the various interactions and how does the
> data
> > > > flow?
> > > > > > >> >
> > > > > > >> > Amandeep
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Amandeep Khurana
> > > > > > >> > Computer Science Graduate Student
> > > > > > >> > University of California, Santa Cruz
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > > amansk@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Thanks Matei. If the basic architecture is similar to the
> > > Google
> > > > > > >> stuff, I
> > > > > > >> > > can safely just work on the project using the information
> > from
> > > > the
> > > > > > >> > papers.
> > > > > > >> > >
> > > > > > >> > > I am aware of the 4487 jira and the current status of the
> > > > > > permissions
> > > > > > >> > > mechanism. I had a look at them earlier.
> > > > > > >> > >
> > > > > > >> > > Cheers
> > > > > > >> > > Amandeep
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Amandeep Khurana
> > > > > > >> > > Computer Science Graduate Student
> > > > > > >> > > University of California, Santa Cruz
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > > matei@cloudera.com>
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > >> Forgot to add, this JIRA details the latest security
> > features
> > > > > that
> > > > > > >> are
> > > > > > >> > >> being
> > > > > > >> > >> worked on in Hadoop trunk:
> > > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > > >> > >> This document describes the current status and
> limitations
> > of
> > > > the
> > > > > > >> > >> permissions mechanism:
> > > > > > >> > >>
> > > > > > >>
> > > > >
> > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > > .
> > > > > > >> > >>
> > > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > > matei@cloudera.com
> > > > > > >
> > > > > > >> > >> wrote:
> > > > > > >> > >>
> > > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > > MapReduce/GFS
> > > > > > at
> > > > > > >> > the
> > > > > > >> > >> > level described in those papers. In particular, in
> HDFS,
> > > > there
> > > > > is
> > > > > > a
> > > > > > >> > >> master
> > > > > > >> > >> > node containing metadata and a number of slave nodes
> > > > > (datanodes)
> > > > > > >> > >> containing
> > > > > > >> > >> > blocks, as in GFS. Clients start by talking to the
> master
> > > to
> > > > > list
> > > > > > >> > >> > directories, etc. When they want to read a region of
> some
> > > > file,
> > > > > > >> they
> > > > > > >> > >> tell
> > > > > > >> > >> > the master the filename and offset, and they receive a
> > list
> > > > of
> > > > > > >> block
> > > > > > >> > >> > locations (datanodes). They then contact the individual
> > > > > datanodes
> > > > > > >> to
> > > > > > >> > >> read
> > > > > > >> > >> > the blocks. When clients write a file, they first
> obtain
> > a
> > > > new
> > > > > > >> block
> > > > > > >> > ID
> > > > > > >> > >> and
> > > > > > >> > >> > list of nodes to write it to from the master, then
> > contact
> > > > the
> > > > > > >> > datanodes
> > > > > > >> > >> to
> > > > > > >> > >> > write it (actually, the datanodes pipeline the write as
> > in
> > > > GFS)
> > > > > > and
> > > > > > >> > >> report
> > > > > > >> > >> > when the write is complete. HDFS actually has some
> > security
> > > > > > >> mechanisms
> > > > > > >> > >> built
> > > > > > >> > >> > in, authenticating users based on their Unix ID and
> > > providing
> > > > > > >> > Unix-like
> > > > > > >> > >> file
> > > > > > >> > >> > permissions. I don't know much about how these are
> > > > implemented,
> > > > > > but
> > > > > > >> > they
> > > > > > >> > >> > would be a good place to start looking.
> > > > > > >> > >> >
> > > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > > >> amansk@gmail.com
> > > > > > >> > >> >wrote:
> > > > > > >> > >> >
> > > > > > >> > >> >> Thanks Matie
> > > > > > >> > >> >>
> > > > > > >> > >> >> I had gone through the architecture document online. I
> > am
> > > > > > >> currently
> > > > > > >> > >> >> working
> > > > > > >> > >> >> on a project towards Security in Hadoop. I do know how
> > the
> > > > > data
> > > > > > >> moves
> > > > > > >> > >> >> around
> > > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> > > follow
> > > > > and
> > > > > > >> how
> > > > > > >> > >> >> different it is from GFS. Can you throw some light on
> > > that?
> > > > > > >> > >> >>
> > > > > > >> > >> >> Security would also involve the Map Reduce jobs
> > following
> > > > the
> > > > > > same
> > > > > > >> > >> >> protocols. Thats why the question about how does the
> > > Hadoop
> > > > > > >> framework
> > > > > > >> > >> >> integrate with the HDFS, and how different is it from
> > Map
> > > > > Reduce
> > > > > > >> and
> > > > > > >> > >> GFS.
> > > > > > >> > >> >> The GFS and Map Reduce papers give a good information
> on
> > > how
> > > > > > those
> > > > > > >> > >> systems
> > > > > > >> > >> >> are designed but there is nothing that concrete for
> > Hadoop
> > > > > that
> > > > > > I
> > > > > > >> > have
> > > > > > >> > >> >> been
> > > > > > >> > >> >> able to find.
> > > > > > >> > >> >>
> > > > > > >> > >> >> Amandeep
> > > > > > >> > >> >>
> > > > > > >> > >> >>
> > > > > > >> > >> >> Amandeep Khurana
> > > > > > >> > >> >> Computer Science Graduate Student
> > > > > > >> > >> >> University of California, Santa Cruz
> > > > > > >> > >> >>
> > > > > > >> > >> >>
> > > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > > >> matei@cloudera.com>
> > > > > > >> > >> >> wrote:
> > > > > > >> > >> >>
> > > > > > >> > >> >> > Hi Amandeep,
> > > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
> > aims
> > > to
> > > > > > >> provide
> > > > > > >> > >> those
> > > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> > similar
> > > to
> > > > > GFS
> > > > > > >> > (large
> > > > > > >> > >> >> > blocks, replication, etc); some notable things
> missing
> > > are
> > > > > > >> > read-write
> > > > > > >> > >> >> > support in the middle of a file (unlikely to be
> > provided
> > > > > > because
> > > > > > >> > few
> > > > > > >> > >> >> Hadoop
> > > > > > >> > >> >> > applications require it) and multiple appenders (the
> > > > record
> > > > > > >> append
> > > > > > >> > >> >> > operation). You can read about HDFS architecture at
> > > > > > >> > >> >> >
> > > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > > .
> > > > > > >> The
> > > > > > >> > >> >> MapReduce
> > > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
> > that
> > > > > > Google's
> > > > > > >> > >> >> MapReduce
> > > > > > >> > >> >> > interacts with GFS (shipping computation to the
> data),
> > > > > > although
> > > > > > >> > >> Hadoop
> > > > > > >> > >> >> > MapReduce also supports running over other
> distributed
> > > > > > >> filesystems.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Matei
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > > > > >> > amansk@gmail.com
> > > > > > >> > >> >
> > > > > > >> > >> >> > wrote:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > > Hi
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Is the HDFS architecture completely based on the
> > > Google
> > > > > > >> > Filesystem?
> > > > > > >> > >> If
> > > > > > >> > >> >> it
> > > > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS
> > same
> > > > as
> > > > > > how
> > > > > > >> it
> > > > > > >> > is
> > > > > > >> > >> >> > between
> > > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Amandeep
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > Amandeep Khurana
> > > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> >
> > > > > > >> > >> >>
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >>
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Nope, typically the JobTracker just starts the process, and the tasktracker
talks directly to the namenode to get a pointer to the datanode, and then
directly to the datanode.
On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Alright.. Got it.
>
> Now, do the task trackers talk to the namenode and the data node directly
> or
> do they go through the job tracker for it? So, if my code is such that I
> need to access more files from the hdfs, would the job tracker get involved
> or not?
>
>
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > Normally, HDFS files are accessed through the namenode. If there was a
> > malicious process though, then I imagine it could talk to a datanode
> > directly and request a specific block.
> >
> > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Ok. Got it.
> > >
> > > Now, when my job needs to access another file, does it go to the
> Namenode
> > > to
> > > get the block ids? How does the java process know where the files are
> and
> > > how to access them?
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > > > I mentioned this case because even jobs written in Java can use the
> > HDFS
> > > > API
> > > > to talk to the NameNode and access the filesystem. People often do
> this
> > > > because their job needs to read a config file, some small data table,
> > etc
> > > > and use this information in its map or reduce functions. In this
> case,
> > > you
> > > > open the second file separately in your mapper's init function and
> read
> > > > whatever you need from it. In general I wanted to point out that you
> > > can't
> > > > know which files a job will access unless you look at its source code
> > or
> > > > monitor the calls it makes; the input file(s) you provide in the job
> > > > description are a hint to the MapReduce framework to place your job
> on
> > > > certain nodes, but it's reasonable for the job to access other files
> as
> > > > well.
> > > >
> > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Another question that I have here - When the jobs run arbitrary
> code
> > > and
> > > > > access data from the HDFS, do they go to the namenode to get the
> > block
> > > > > information?
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Assuming that the job is purely in Java and not involving
> streaming
> > > or
> > > > > > pipes, wouldnt the resources (files) required by the job as
> inputs
> > be
> > > > > known
> > > > > > beforehand? So, if the map task is accessing a second file, how
> > does
> > > it
> > > > > make
> > > > > > it different except that there are multiple files. The JobTracker
> > > would
> > > > > know
> > > > > > beforehand that multiple files would be accessed. Right?
> > > > > >
> > > > > > I am slightly confused why you have mentioned this case
> > separately...
> > > > Can
> > > > > > you elaborate on it a little bit?
> > > > > >
> > > > > > Amandeep
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> matei@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Typically the data flow is like this:1) Client submits a job
> > > > description
> > > > > >> to
> > > > > >> the JobTracker.
> > > > > >> 2) JobTracker figures out block locations for the input file(s)
> by
> > > > > talking
> > > > > >> to HDFS NameNode.
> > > > > >> 3) JobTracker creates a job description file in HDFS which will
> be
> > > > read
> > > > > by
> > > > > >> the nodes to copy over the job's code etc.
> > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with
> > the
> > > > > >> appropriate data blocks.
> > > > > >> 5) After running, maps create intermediate output files on those
> > > > slaves.
> > > > > >> These are not in HDFS, they're in some temporary storage used by
> > > > > >> MapReduce.
> > > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
> > over
> > > > the
> > > > > >> appropriate map outputs, apply the reduce function, and write
> the
> > > > > outputs
> > > > > >> to
> > > > > >> HDFS (one output file per reducer).
> > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > JobTracker.
> > > > > >>
> > > > > >> However, there is a big caveat, which is that the map and reduce
> > > tasks
> > > > > run
> > > > > >> arbitrary code. It is not unusual to have a map that opens a
> > second
> > > > HDFS
> > > > > >> file to read some information (e.g. for doing a join of a small
> > > table
> > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
> write
> > a
> > > > job
> > > > > >> in
> > > > > >> Python, Ruby, C, etc, then you are launching arbitrary processes
> > > which
> > > > > may
> > > > > >> also access external resources in this manner. Some people also
> > > > > read/write
> > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > > > solution
> > > > > >> would ideally deal with these cases too.
> > > > > >>
> > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > amansk@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > A quick question here. How does a typical hadoop job work at
> the
> > > > > system
> > > > > >> > level? What are the various interactions and how does the data
> > > flow?
> > > > > >> >
> > > > > >> > Amandeep
> > > > > >> >
> > > > > >> >
> > > > > >> > Amandeep Khurana
> > > > > >> > Computer Science Graduate Student
> > > > > >> > University of California, Santa Cruz
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > amansk@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Thanks Matei. If the basic architecture is similar to the
> > Google
> > > > > >> stuff, I
> > > > > >> > > can safely just work on the project using the information
> from
> > > the
> > > > > >> > papers.
> > > > > >> > >
> > > > > >> > > I am aware of the 4487 jira and the current status of the
> > > > > permissions
> > > > > >> > > mechanism. I had a look at them earlier.
> > > > > >> > >
> > > > > >> > > Cheers
> > > > > >> > > Amandeep
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Amandeep Khurana
> > > > > >> > > Computer Science Graduate Student
> > > > > >> > > University of California, Santa Cruz
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > matei@cloudera.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> Forgot to add, this JIRA details the latest security
> features
> > > > that
> > > > > >> are
> > > > > >> > >> being
> > > > > >> > >> worked on in Hadoop trunk:
> > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > >> > >> This document describes the current status and limitations
> of
> > > the
> > > > > >> > >> permissions mechanism:
> > > > > >> > >>
> > > > > >>
> > > >
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > .
> > > > > >> > >>
> > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > matei@cloudera.com
> > > > > >
> > > > > >> > >> wrote:
> > > > > >> > >>
> > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > MapReduce/GFS
> > > > > at
> > > > > >> > the
> > > > > >> > >> > level described in those papers. In particular, in HDFS,
> > > there
> > > > is
> > > > > a
> > > > > >> > >> master
> > > > > >> > >> > node containing metadata and a number of slave nodes
> > > > (datanodes)
> > > > > >> > >> containing
> > > > > >> > >> > blocks, as in GFS. Clients start by talking to the master
> > to
> > > > list
> > > > > >> > >> > directories, etc. When they want to read a region of some
> > > file,
> > > > > >> they
> > > > > >> > >> tell
> > > > > >> > >> > the master the filename and offset, and they receive a
> list
> > > of
> > > > > >> block
> > > > > >> > >> > locations (datanodes). They then contact the individual
> > > > datanodes
> > > > > >> to
> > > > > >> > >> read
> > > > > >> > >> > the blocks. When clients write a file, they first obtain
> a
> > > new
> > > > > >> block
> > > > > >> > ID
> > > > > >> > >> and
> > > > > >> > >> > list of nodes to write it to from the master, then
> contact
> > > the
> > > > > >> > datanodes
> > > > > >> > >> to
> > > > > >> > >> > write it (actually, the datanodes pipeline the write as
> in
> > > GFS)
> > > > > and
> > > > > >> > >> report
> > > > > >> > >> > when the write is complete. HDFS actually has some
> security
> > > > > >> mechanisms
> > > > > >> > >> built
> > > > > >> > >> > in, authenticating users based on their Unix ID and
> > providing
> > > > > >> > Unix-like
> > > > > >> > >> file
> > > > > >> > >> > permissions. I don't know much about how these are
> > > implemented,
> > > > > but
> > > > > >> > they
> > > > > >> > >> > would be a good place to start looking.
> > > > > >> > >> >
> > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > >> amansk@gmail.com
> > > > > >> > >> >wrote:
> > > > > >> > >> >
> > > > > >> > >> >> Thanks Matie
> > > > > >> > >> >>
> > > > > >> > >> >> I had gone through the architecture document online. I
> am
> > > > > >> currently
> > > > > >> > >> >> working
> > > > > >> > >> >> on a project towards Security in Hadoop. I do know how
> the
> > > > data
> > > > > >> moves
> > > > > >> > >> >> around
> > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> > follow
> > > > and
> > > > > >> how
> > > > > >> > >> >> different it is from GFS. Can you throw some light on
> > that?
> > > > > >> > >> >>
> > > > > >> > >> >> Security would also involve the Map Reduce jobs
> following
> > > the
> > > > > same
> > > > > >> > >> >> protocols. Thats why the question about how does the
> > Hadoop
> > > > > >> framework
> > > > > >> > >> >> integrate with the HDFS, and how different is it from
> Map
> > > > Reduce
> > > > > >> and
> > > > > >> > >> GFS.
> > > > > >> > >> >> The GFS and Map Reduce papers give a good information on
> > how
> > > > > those
> > > > > >> > >> systems
> > > > > >> > >> >> are designed but there is nothing that concrete for
> Hadoop
> > > > that
> > > > > I
> > > > > >> > have
> > > > > >> > >> >> been
> > > > > >> > >> >> able to find.
> > > > > >> > >> >>
> > > > > >> > >> >> Amandeep
> > > > > >> > >> >>
> > > > > >> > >> >>
> > > > > >> > >> >> Amandeep Khurana
> > > > > >> > >> >> Computer Science Graduate Student
> > > > > >> > >> >> University of California, Santa Cruz
> > > > > >> > >> >>
> > > > > >> > >> >>
> > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > >> matei@cloudera.com>
> > > > > >> > >> >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> > Hi Amandeep,
> > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
> aims
> > to
> > > > > >> provide
> > > > > >> > >> those
> > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> similar
> > to
> > > > GFS
> > > > > >> > (large
> > > > > >> > >> >> > blocks, replication, etc); some notable things missing
> > are
> > > > > >> > read-write
> > > > > >> > >> >> > support in the middle of a file (unlikely to be
> provided
> > > > > because
> > > > > >> > few
> > > > > >> > >> >> Hadoop
> > > > > >> > >> >> > applications require it) and multiple appenders (the
> > > record
> > > > > >> append
> > > > > >> > >> >> > operation). You can read about HDFS architecture at
> > > > > >> > >> >> >
> > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > .
> > > > > >> The
> > > > > >> > >> >> MapReduce
> > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
> that
> > > > > Google's
> > > > > >> > >> >> MapReduce
> > > > > >> > >> >> > interacts with GFS (shipping computation to the data),
> > > > > although
> > > > > >> > >> Hadoop
> > > > > >> > >> >> > MapReduce also supports running over other distributed
> > > > > >> filesystems.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Matei
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > > > >> > amansk@gmail.com
> > > > > >> > >> >
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > Hi
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Is the HDFS architecture completely based on the
> > Google
> > > > > >> > Filesystem?
> > > > > >> > >> If
> > > > > >> > >> >> it
> > > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS
> same
> > > as
> > > > > how
> > > > > >> it
> > > > > >> > is
> > > > > >> > >> >> > between
> > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Amandeep
> > > > > >> > >> >> > >
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Amandeep Khurana
> > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > >> > >> >> > >
> > > > > >> > >> >> >
> > > > > >> > >> >>
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Nope, typically the JobTracker just starts the process, and the tasktracker
talks directly to the namenode to get a pointer to the datanode, and then
directly to the datanode.
On Sun, Feb 15, 2009 at 8:07 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Alright.. Got it.
>
> Now, do the task trackers talk to the namenode and the data node directly
> or
> do they go through the job tracker for it? So, if my code is such that I
> need to access more files from the hdfs, would the job tracker get involved
> or not?
>
>
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > Normally, HDFS files are accessed through the namenode. If there was a
> > malicious process though, then I imagine it could talk to a datanode
> > directly and request a specific block.
> >
> > On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Ok. Got it.
> > >
> > > Now, when my job needs to access another file, does it go to the
> Namenode
> > > to
> > > get the block ids? How does the java process know where the files are
> and
> > > how to access them?
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > > > I mentioned this case because even jobs written in Java can use the
> > HDFS
> > > > API
> > > > to talk to the NameNode and access the filesystem. People often do
> this
> > > > because their job needs to read a config file, some small data table,
> > etc
> > > > and use this information in its map or reduce functions. In this
> case,
> > > you
> > > > open the second file separately in your mapper's init function and
> read
> > > > whatever you need from it. In general I wanted to point out that you
> > > can't
> > > > know which files a job will access unless you look at its source code
> > or
> > > > monitor the calls it makes; the input file(s) you provide in the job
> > > > description are a hint to the MapReduce framework to place your job
> on
> > > > certain nodes, but it's reasonable for the job to access other files
> as
> > > > well.
> > > >
> > > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Another question that I have here - When the jobs run arbitrary
> code
> > > and
> > > > > access data from the HDFS, do they go to the namenode to get the
> > block
> > > > > information?
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Assuming that the job is purely in Java and not involving
> streaming
> > > or
> > > > > > pipes, wouldnt the resources (files) required by the job as
> inputs
> > be
> > > > > known
> > > > > > beforehand? So, if the map task is accessing a second file, how
> > does
> > > it
> > > > > make
> > > > > > it different except that there are multiple files. The JobTracker
> > > would
> > > > > know
> > > > > > beforehand that multiple files would be accessed. Right?
> > > > > >
> > > > > > I am slightly confused why you have mentioned this case
> > separately...
> > > > Can
> > > > > > you elaborate on it a little bit?
> > > > > >
> > > > > > Amandeep
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <
> matei@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Typically the data flow is like this:1) Client submits a job
> > > > description
> > > > > >> to
> > > > > >> the JobTracker.
> > > > > >> 2) JobTracker figures out block locations for the input file(s)
> by
> > > > > talking
> > > > > >> to HDFS NameNode.
> > > > > >> 3) JobTracker creates a job description file in HDFS which will
> be
> > > > read
> > > > > by
> > > > > >> the nodes to copy over the job's code etc.
> > > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with
> > the
> > > > > >> appropriate data blocks.
> > > > > >> 5) After running, maps create intermediate output files on those
> > > > slaves.
> > > > > >> These are not in HDFS, they're in some temporary storage used by
> > > > > >> MapReduce.
> > > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
> > over
> > > > the
> > > > > >> appropriate map outputs, apply the reduce function, and write
> the
> > > > > outputs
> > > > > >> to
> > > > > >> HDFS (one output file per reducer).
> > > > > >> 7) Some logs for the job may also be put into HDFS by the
> > > JobTracker.
> > > > > >>
> > > > > >> However, there is a big caveat, which is that the map and reduce
> > > tasks
> > > > > run
> > > > > >> arbitrary code. It is not unusual to have a map that opens a
> > second
> > > > HDFS
> > > > > >> file to read some information (e.g. for doing a join of a small
> > > table
> > > > > >> against a big file). If you use Hadoop Streaming or Pipes to
> write
> > a
> > > > job
> > > > > >> in
> > > > > >> Python, Ruby, C, etc, then you are launching arbitrary processes
> > > which
> > > > > may
> > > > > >> also access external resources in this manner. Some people also
> > > > > read/write
> > > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > > > solution
> > > > > >> would ideally deal with these cases too.
> > > > > >>
> > > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> > amansk@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > A quick question here. How does a typical hadoop job work at
> the
> > > > > system
> > > > > >> > level? What are the various interactions and how does the data
> > > flow?
> > > > > >> >
> > > > > >> > Amandeep
> > > > > >> >
> > > > > >> >
> > > > > >> > Amandeep Khurana
> > > > > >> > Computer Science Graduate Student
> > > > > >> > University of California, Santa Cruz
> > > > > >> >
> > > > > >> >
> > > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > > amansk@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Thanks Matei. If the basic architecture is similar to the
> > Google
> > > > > >> stuff, I
> > > > > >> > > can safely just work on the project using the information
> from
> > > the
> > > > > >> > papers.
> > > > > >> > >
> > > > > >> > > I am aware of the 4487 jira and the current status of the
> > > > > permissions
> > > > > >> > > mechanism. I had a look at them earlier.
> > > > > >> > >
> > > > > >> > > Cheers
> > > > > >> > > Amandeep
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Amandeep Khurana
> > > > > >> > > Computer Science Graduate Student
> > > > > >> > > University of California, Santa Cruz
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > > matei@cloudera.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> Forgot to add, this JIRA details the latest security
> features
> > > > that
> > > > > >> are
> > > > > >> > >> being
> > > > > >> > >> worked on in Hadoop trunk:
> > > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > > >> > >> This document describes the current status and limitations
> of
> > > the
> > > > > >> > >> permissions mechanism:
> > > > > >> > >>
> > > > > >>
> > > >
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> > .
> > > > > >> > >>
> > > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > > matei@cloudera.com
> > > > > >
> > > > > >> > >> wrote:
> > > > > >> > >>
> > > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > > MapReduce/GFS
> > > > > at
> > > > > >> > the
> > > > > >> > >> > level described in those papers. In particular, in HDFS,
> > > there
> > > > is
> > > > > a
> > > > > >> > >> master
> > > > > >> > >> > node containing metadata and a number of slave nodes
> > > > (datanodes)
> > > > > >> > >> containing
> > > > > >> > >> > blocks, as in GFS. Clients start by talking to the master
> > to
> > > > list
> > > > > >> > >> > directories, etc. When they want to read a region of some
> > > file,
> > > > > >> they
> > > > > >> > >> tell
> > > > > >> > >> > the master the filename and offset, and they receive a
> list
> > > of
> > > > > >> block
> > > > > >> > >> > locations (datanodes). They then contact the individual
> > > > datanodes
> > > > > >> to
> > > > > >> > >> read
> > > > > >> > >> > the blocks. When clients write a file, they first obtain
> a
> > > new
> > > > > >> block
> > > > > >> > ID
> > > > > >> > >> and
> > > > > >> > >> > list of nodes to write it to from the master, then
> contact
> > > the
> > > > > >> > datanodes
> > > > > >> > >> to
> > > > > >> > >> > write it (actually, the datanodes pipeline the write as
> in
> > > GFS)
> > > > > and
> > > > > >> > >> report
> > > > > >> > >> > when the write is complete. HDFS actually has some
> security
> > > > > >> mechanisms
> > > > > >> > >> built
> > > > > >> > >> > in, authenticating users based on their Unix ID and
> > providing
> > > > > >> > Unix-like
> > > > > >> > >> file
> > > > > >> > >> > permissions. I don't know much about how these are
> > > implemented,
> > > > > but
> > > > > >> > they
> > > > > >> > >> > would be a good place to start looking.
> > > > > >> > >> >
> > > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > > >> amansk@gmail.com
> > > > > >> > >> >wrote:
> > > > > >> > >> >
> > > > > >> > >> >> Thanks Matie
> > > > > >> > >> >>
> > > > > >> > >> >> I had gone through the architecture document online. I
> am
> > > > > >> currently
> > > > > >> > >> >> working
> > > > > >> > >> >> on a project towards Security in Hadoop. I do know how
> the
> > > > data
> > > > > >> moves
> > > > > >> > >> >> around
> > > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> > follow
> > > > and
> > > > > >> how
> > > > > >> > >> >> different it is from GFS. Can you throw some light on
> > that?
> > > > > >> > >> >>
> > > > > >> > >> >> Security would also involve the Map Reduce jobs
> following
> > > the
> > > > > same
> > > > > >> > >> >> protocols. Thats why the question about how does the
> > Hadoop
> > > > > >> framework
> > > > > >> > >> >> integrate with the HDFS, and how different is it from
> Map
> > > > Reduce
> > > > > >> and
> > > > > >> > >> GFS.
> > > > > >> > >> >> The GFS and Map Reduce papers give a good information on
> > how
> > > > > those
> > > > > >> > >> systems
> > > > > >> > >> >> are designed but there is nothing that concrete for
> Hadoop
> > > > that
> > > > > I
> > > > > >> > have
> > > > > >> > >> >> been
> > > > > >> > >> >> able to find.
> > > > > >> > >> >>
> > > > > >> > >> >> Amandeep
> > > > > >> > >> >>
> > > > > >> > >> >>
> > > > > >> > >> >> Amandeep Khurana
> > > > > >> > >> >> Computer Science Graduate Student
> > > > > >> > >> >> University of California, Santa Cruz
> > > > > >> > >> >>
> > > > > >> > >> >>
> > > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > > >> matei@cloudera.com>
> > > > > >> > >> >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> > Hi Amandeep,
> > > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and
> aims
> > to
> > > > > >> provide
> > > > > >> > >> those
> > > > > >> > >> >> > capabilities as an open-source project. HDFS is
> similar
> > to
> > > > GFS
> > > > > >> > (large
> > > > > >> > >> >> > blocks, replication, etc); some notable things missing
> > are
> > > > > >> > read-write
> > > > > >> > >> >> > support in the middle of a file (unlikely to be
> provided
> > > > > because
> > > > > >> > few
> > > > > >> > >> >> Hadoop
> > > > > >> > >> >> > applications require it) and multiple appenders (the
> > > record
> > > > > >> append
> > > > > >> > >> >> > operation). You can read about HDFS architecture at
> > > > > >> > >> >> >
> > > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > > .
> > > > > >> The
> > > > > >> > >> >> MapReduce
> > > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way
> that
> > > > > Google's
> > > > > >> > >> >> MapReduce
> > > > > >> > >> >> > interacts with GFS (shipping computation to the data),
> > > > > although
> > > > > >> > >> Hadoop
> > > > > >> > >> >> > MapReduce also supports running over other distributed
> > > > > >> filesystems.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Matei
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > > > >> > amansk@gmail.com
> > > > > >> > >> >
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > Hi
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Is the HDFS architecture completely based on the
> > Google
> > > > > >> > Filesystem?
> > > > > >> > >> If
> > > > > >> > >> >> it
> > > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS
> same
> > > as
> > > > > how
> > > > > >> it
> > > > > >> > is
> > > > > >> > >> >> > between
> > > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Amandeep
> > > > > >> > >> >> > >
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > Amandeep Khurana
> > > > > >> > >> >> > > Computer Science Graduate Student
> > > > > >> > >> >> > > University of California, Santa Cruz
> > > > > >> > >> >> > >
> > > > > >> > >> >> >
> > > > > >> > >> >>
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Alright.. Got it.
Now, do the task trackers talk to the namenode and the data node directly or
do they go through the job tracker for it? So, if my code is such that I
need to access more files from the hdfs, would the job tracker get involved
or not?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Normally, HDFS files are accessed through the namenode. If there was a
> malicious process though, then I imagine it could talk to a datanode
> directly and request a specific block.
>
> On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Ok. Got it.
> >
> > Now, when my job needs to access another file, does it go to the Namenode
> > to
> > get the block ids? How does the java process know where the files are and
> > how to access them?
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> > > I mentioned this case because even jobs written in Java can use the
> HDFS
> > > API
> > > to talk to the NameNode and access the filesystem. People often do this
> > > because their job needs to read a config file, some small data table,
> etc
> > > and use this information in its map or reduce functions. In this case,
> > you
> > > open the second file separately in your mapper's init function and read
> > > whatever you need from it. In general I wanted to point out that you
> > can't
> > > know which files a job will access unless you look at its source code
> or
> > > monitor the calls it makes; the input file(s) you provide in the job
> > > description are a hint to the MapReduce framework to place your job on
> > > certain nodes, but it's reasonable for the job to access other files as
> > > well.
> > >
> > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Another question that I have here - When the jobs run arbitrary code
> > and
> > > > access data from the HDFS, do they go to the namenode to get the
> block
> > > > information?
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Assuming that the job is purely in Java and not involving streaming
> > or
> > > > > pipes, wouldnt the resources (files) required by the job as inputs
> be
> > > > known
> > > > > beforehand? So, if the map task is accessing a second file, how
> does
> > it
> > > > make
> > > > > it different except that there are multiple files. The JobTracker
> > would
> > > > know
> > > > > beforehand that multiple files would be accessed. Right?
> > > > >
> > > > > I am slightly confused why you have mentioned this case
> separately...
> > > Can
> > > > > you elaborate on it a little bit?
> > > > >
> > > > > Amandeep
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <matei@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > >> Typically the data flow is like this:1) Client submits a job
> > > description
> > > > >> to
> > > > >> the JobTracker.
> > > > >> 2) JobTracker figures out block locations for the input file(s) by
> > > > talking
> > > > >> to HDFS NameNode.
> > > > >> 3) JobTracker creates a job description file in HDFS which will be
> > > read
> > > > by
> > > > >> the nodes to copy over the job's code etc.
> > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with
> the
> > > > >> appropriate data blocks.
> > > > >> 5) After running, maps create intermediate output files on those
> > > slaves.
> > > > >> These are not in HDFS, they're in some temporary storage used by
> > > > >> MapReduce.
> > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
> over
> > > the
> > > > >> appropriate map outputs, apply the reduce function, and write the
> > > > outputs
> > > > >> to
> > > > >> HDFS (one output file per reducer).
> > > > >> 7) Some logs for the job may also be put into HDFS by the
> > JobTracker.
> > > > >>
> > > > >> However, there is a big caveat, which is that the map and reduce
> > tasks
> > > > run
> > > > >> arbitrary code. It is not unusual to have a map that opens a
> second
> > > HDFS
> > > > >> file to read some information (e.g. for doing a join of a small
> > table
> > > > >> against a big file). If you use Hadoop Streaming or Pipes to write
> a
> > > job
> > > > >> in
> > > > >> Python, Ruby, C, etc, then you are launching arbitrary processes
> > which
> > > > may
> > > > >> also access external resources in this manner. Some people also
> > > > read/write
> > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > > solution
> > > > >> would ideally deal with these cases too.
> > > > >>
> > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> amansk@gmail.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > A quick question here. How does a typical hadoop job work at the
> > > > system
> > > > >> > level? What are the various interactions and how does the data
> > flow?
> > > > >> >
> > > > >> > Amandeep
> > > > >> >
> > > > >> >
> > > > >> > Amandeep Khurana
> > > > >> > Computer Science Graduate Student
> > > > >> > University of California, Santa Cruz
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > amansk@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Thanks Matei. If the basic architecture is similar to the
> Google
> > > > >> stuff, I
> > > > >> > > can safely just work on the project using the information from
> > the
> > > > >> > papers.
> > > > >> > >
> > > > >> > > I am aware of the 4487 jira and the current status of the
> > > > permissions
> > > > >> > > mechanism. I had a look at them earlier.
> > > > >> > >
> > > > >> > > Cheers
> > > > >> > > Amandeep
> > > > >> > >
> > > > >> > >
> > > > >> > > Amandeep Khurana
> > > > >> > > Computer Science Graduate Student
> > > > >> > > University of California, Santa Cruz
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > matei@cloudera.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> Forgot to add, this JIRA details the latest security features
> > > that
> > > > >> are
> > > > >> > >> being
> > > > >> > >> worked on in Hadoop trunk:
> > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > >> > >> This document describes the current status and limitations of
> > the
> > > > >> > >> permissions mechanism:
> > > > >> > >>
> > > > >>
> > > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> .
> > > > >> > >>
> > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > matei@cloudera.com
> > > > >
> > > > >> > >> wrote:
> > > > >> > >>
> > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > MapReduce/GFS
> > > > at
> > > > >> > the
> > > > >> > >> > level described in those papers. In particular, in HDFS,
> > there
> > > is
> > > > a
> > > > >> > >> master
> > > > >> > >> > node containing metadata and a number of slave nodes
> > > (datanodes)
> > > > >> > >> containing
> > > > >> > >> > blocks, as in GFS. Clients start by talking to the master
> to
> > > list
> > > > >> > >> > directories, etc. When they want to read a region of some
> > file,
> > > > >> they
> > > > >> > >> tell
> > > > >> > >> > the master the filename and offset, and they receive a list
> > of
> > > > >> block
> > > > >> > >> > locations (datanodes). They then contact the individual
> > > datanodes
> > > > >> to
> > > > >> > >> read
> > > > >> > >> > the blocks. When clients write a file, they first obtain a
> > new
> > > > >> block
> > > > >> > ID
> > > > >> > >> and
> > > > >> > >> > list of nodes to write it to from the master, then contact
> > the
> > > > >> > datanodes
> > > > >> > >> to
> > > > >> > >> > write it (actually, the datanodes pipeline the write as in
> > GFS)
> > > > and
> > > > >> > >> report
> > > > >> > >> > when the write is complete. HDFS actually has some security
> > > > >> mechanisms
> > > > >> > >> built
> > > > >> > >> > in, authenticating users based on their Unix ID and
> providing
> > > > >> > Unix-like
> > > > >> > >> file
> > > > >> > >> > permissions. I don't know much about how these are
> > implemented,
> > > > but
> > > > >> > they
> > > > >> > >> > would be a good place to start looking.
> > > > >> > >> >
> > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > >> amansk@gmail.com
> > > > >> > >> >wrote:
> > > > >> > >> >
> > > > >> > >> >> Thanks Matie
> > > > >> > >> >>
> > > > >> > >> >> I had gone through the architecture document online. I am
> > > > >> currently
> > > > >> > >> >> working
> > > > >> > >> >> on a project towards Security in Hadoop. I do know how the
> > > data
> > > > >> moves
> > > > >> > >> >> around
> > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> follow
> > > and
> > > > >> how
> > > > >> > >> >> different it is from GFS. Can you throw some light on
> that?
> > > > >> > >> >>
> > > > >> > >> >> Security would also involve the Map Reduce jobs following
> > the
> > > > same
> > > > >> > >> >> protocols. Thats why the question about how does the
> Hadoop
> > > > >> framework
> > > > >> > >> >> integrate with the HDFS, and how different is it from Map
> > > Reduce
> > > > >> and
> > > > >> > >> GFS.
> > > > >> > >> >> The GFS and Map Reduce papers give a good information on
> how
> > > > those
> > > > >> > >> systems
> > > > >> > >> >> are designed but there is nothing that concrete for Hadoop
> > > that
> > > > I
> > > > >> > have
> > > > >> > >> >> been
> > > > >> > >> >> able to find.
> > > > >> > >> >>
> > > > >> > >> >> Amandeep
> > > > >> > >> >>
> > > > >> > >> >>
> > > > >> > >> >> Amandeep Khurana
> > > > >> > >> >> Computer Science Graduate Student
> > > > >> > >> >> University of California, Santa Cruz
> > > > >> > >> >>
> > > > >> > >> >>
> > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > >> matei@cloudera.com>
> > > > >> > >> >> wrote:
> > > > >> > >> >>
> > > > >> > >> >> > Hi Amandeep,
> > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims
> to
> > > > >> provide
> > > > >> > >> those
> > > > >> > >> >> > capabilities as an open-source project. HDFS is similar
> to
> > > GFS
> > > > >> > (large
> > > > >> > >> >> > blocks, replication, etc); some notable things missing
> are
> > > > >> > read-write
> > > > >> > >> >> > support in the middle of a file (unlikely to be provided
> > > > because
> > > > >> > few
> > > > >> > >> >> Hadoop
> > > > >> > >> >> > applications require it) and multiple appenders (the
> > record
> > > > >> append
> > > > >> > >> >> > operation). You can read about HDFS architecture at
> > > > >> > >> >> >
> > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > .
> > > > >> The
> > > > >> > >> >> MapReduce
> > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> > > > Google's
> > > > >> > >> >> MapReduce
> > > > >> > >> >> > interacts with GFS (shipping computation to the data),
> > > > although
> > > > >> > >> Hadoop
> > > > >> > >> >> > MapReduce also supports running over other distributed
> > > > >> filesystems.
> > > > >> > >> >> >
> > > > >> > >> >> > Matei
> > > > >> > >> >> >
> > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > > >> > amansk@gmail.com
> > > > >> > >> >
> > > > >> > >> >> > wrote:
> > > > >> > >> >> >
> > > > >> > >> >> > > Hi
> > > > >> > >> >> > >
> > > > >> > >> >> > > Is the HDFS architecture completely based on the
> Google
> > > > >> > Filesystem?
> > > > >> > >> If
> > > > >> > >> >> it
> > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > >> > >> >> > >
> > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same
> > as
> > > > how
> > > > >> it
> > > > >> > is
> > > > >> > >> >> > between
> > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > >> > >> >> > >
> > > > >> > >> >> > > Amandeep
> > > > >> > >> >> > >
> > > > >> > >> >> > >
> > > > >> > >> >> > > Amandeep Khurana
> > > > >> > >> >> > > Computer Science Graduate Student
> > > > >> > >> >> > > University of California, Santa Cruz
> > > > >> > >> >> > >
> > > > >> > >> >> >
> > > > >> > >> >>
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Alright.. Got it.
Now, do the task trackers talk to the namenode and the data node directly or
do they go through the job tracker for it? So, if my code is such that I
need to access more files from the hdfs, would the job tracker get involved
or not?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 7:20 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Normally, HDFS files are accessed through the namenode. If there was a
> malicious process though, then I imagine it could talk to a datanode
> directly and request a specific block.
>
> On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Ok. Got it.
> >
> > Now, when my job needs to access another file, does it go to the Namenode
> > to
> > get the block ids? How does the java process know where the files are and
> > how to access them?
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> > > I mentioned this case because even jobs written in Java can use the
> HDFS
> > > API
> > > to talk to the NameNode and access the filesystem. People often do this
> > > because their job needs to read a config file, some small data table,
> etc
> > > and use this information in its map or reduce functions. In this case,
> > you
> > > open the second file separately in your mapper's init function and read
> > > whatever you need from it. In general I wanted to point out that you
> > can't
> > > know which files a job will access unless you look at its source code
> or
> > > monitor the calls it makes; the input file(s) you provide in the job
> > > description are a hint to the MapReduce framework to place your job on
> > > certain nodes, but it's reasonable for the job to access other files as
> > > well.
> > >
> > > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Another question that I have here - When the jobs run arbitrary code
> > and
> > > > access data from the HDFS, do they go to the namenode to get the
> block
> > > > information?
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> > > > wrote:
> > > >
> > > > > Assuming that the job is purely in Java and not involving streaming
> > or
> > > > > pipes, wouldnt the resources (files) required by the job as inputs
> be
> > > > known
> > > > > beforehand? So, if the map task is accessing a second file, how
> does
> > it
> > > > make
> > > > > it different except that there are multiple files. The JobTracker
> > would
> > > > know
> > > > > beforehand that multiple files would be accessed. Right?
> > > > >
> > > > > I am slightly confused why you have mentioned this case
> separately...
> > > Can
> > > > > you elaborate on it a little bit?
> > > > >
> > > > > Amandeep
> > > > >
> > > > >
> > > > > Amandeep Khurana
> > > > > Computer Science Graduate Student
> > > > > University of California, Santa Cruz
> > > > >
> > > > >
> > > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <matei@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > >> Typically the data flow is like this:1) Client submits a job
> > > description
> > > > >> to
> > > > >> the JobTracker.
> > > > >> 2) JobTracker figures out block locations for the input file(s) by
> > > > talking
> > > > >> to HDFS NameNode.
> > > > >> 3) JobTracker creates a job description file in HDFS which will be
> > > read
> > > > by
> > > > >> the nodes to copy over the job's code etc.
> > > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with
> the
> > > > >> appropriate data blocks.
> > > > >> 5) After running, maps create intermediate output files on those
> > > slaves.
> > > > >> These are not in HDFS, they're in some temporary storage used by
> > > > >> MapReduce.
> > > > >> 6) JobTracker starts reduces on a series of slaves, which copy
> over
> > > the
> > > > >> appropriate map outputs, apply the reduce function, and write the
> > > > outputs
> > > > >> to
> > > > >> HDFS (one output file per reducer).
> > > > >> 7) Some logs for the job may also be put into HDFS by the
> > JobTracker.
> > > > >>
> > > > >> However, there is a big caveat, which is that the map and reduce
> > tasks
> > > > run
> > > > >> arbitrary code. It is not unusual to have a map that opens a
> second
> > > HDFS
> > > > >> file to read some information (e.g. for doing a join of a small
> > table
> > > > >> against a big file). If you use Hadoop Streaming or Pipes to write
> a
> > > job
> > > > >> in
> > > > >> Python, Ruby, C, etc, then you are launching arbitrary processes
> > which
> > > > may
> > > > >> also access external resources in this manner. Some people also
> > > > read/write
> > > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > > solution
> > > > >> would ideally deal with these cases too.
> > > > >>
> > > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <
> amansk@gmail.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > A quick question here. How does a typical hadoop job work at the
> > > > system
> > > > >> > level? What are the various interactions and how does the data
> > flow?
> > > > >> >
> > > > >> > Amandeep
> > > > >> >
> > > > >> >
> > > > >> > Amandeep Khurana
> > > > >> > Computer Science Graduate Student
> > > > >> > University of California, Santa Cruz
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> > amansk@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Thanks Matei. If the basic architecture is similar to the
> Google
> > > > >> stuff, I
> > > > >> > > can safely just work on the project using the information from
> > the
> > > > >> > papers.
> > > > >> > >
> > > > >> > > I am aware of the 4487 jira and the current status of the
> > > > permissions
> > > > >> > > mechanism. I had a look at them earlier.
> > > > >> > >
> > > > >> > > Cheers
> > > > >> > > Amandeep
> > > > >> > >
> > > > >> > >
> > > > >> > > Amandeep Khurana
> > > > >> > > Computer Science Graduate Student
> > > > >> > > University of California, Santa Cruz
> > > > >> > >
> > > > >> > >
> > > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > > matei@cloudera.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> Forgot to add, this JIRA details the latest security features
> > > that
> > > > >> are
> > > > >> > >> being
> > > > >> > >> worked on in Hadoop trunk:
> > > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > > >> > >> This document describes the current status and limitations of
> > the
> > > > >> > >> permissions mechanism:
> > > > >> > >>
> > > > >>
> > > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html
> .
> > > > >> > >>
> > > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > > matei@cloudera.com
> > > > >
> > > > >> > >> wrote:
> > > > >> > >>
> > > > >> > >> > I think it's safe to assume that Hadoop works like
> > > MapReduce/GFS
> > > > at
> > > > >> > the
> > > > >> > >> > level described in those papers. In particular, in HDFS,
> > there
> > > is
> > > > a
> > > > >> > >> master
> > > > >> > >> > node containing metadata and a number of slave nodes
> > > (datanodes)
> > > > >> > >> containing
> > > > >> > >> > blocks, as in GFS. Clients start by talking to the master
> to
> > > list
> > > > >> > >> > directories, etc. When they want to read a region of some
> > file,
> > > > >> they
> > > > >> > >> tell
> > > > >> > >> > the master the filename and offset, and they receive a list
> > of
> > > > >> block
> > > > >> > >> > locations (datanodes). They then contact the individual
> > > datanodes
> > > > >> to
> > > > >> > >> read
> > > > >> > >> > the blocks. When clients write a file, they first obtain a
> > new
> > > > >> block
> > > > >> > ID
> > > > >> > >> and
> > > > >> > >> > list of nodes to write it to from the master, then contact
> > the
> > > > >> > datanodes
> > > > >> > >> to
> > > > >> > >> > write it (actually, the datanodes pipeline the write as in
> > GFS)
> > > > and
> > > > >> > >> report
> > > > >> > >> > when the write is complete. HDFS actually has some security
> > > > >> mechanisms
> > > > >> > >> built
> > > > >> > >> > in, authenticating users based on their Unix ID and
> providing
> > > > >> > Unix-like
> > > > >> > >> file
> > > > >> > >> > permissions. I don't know much about how these are
> > implemented,
> > > > but
> > > > >> > they
> > > > >> > >> > would be a good place to start looking.
> > > > >> > >> >
> > > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > > >> amansk@gmail.com
> > > > >> > >> >wrote:
> > > > >> > >> >
> > > > >> > >> >> Thanks Matie
> > > > >> > >> >>
> > > > >> > >> >> I had gone through the architecture document online. I am
> > > > >> currently
> > > > >> > >> >> working
> > > > >> > >> >> on a project towards Security in Hadoop. I do know how the
> > > data
> > > > >> moves
> > > > >> > >> >> around
> > > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS
> follow
> > > and
> > > > >> how
> > > > >> > >> >> different it is from GFS. Can you throw some light on
> that?
> > > > >> > >> >>
> > > > >> > >> >> Security would also involve the Map Reduce jobs following
> > the
> > > > same
> > > > >> > >> >> protocols. Thats why the question about how does the
> Hadoop
> > > > >> framework
> > > > >> > >> >> integrate with the HDFS, and how different is it from Map
> > > Reduce
> > > > >> and
> > > > >> > >> GFS.
> > > > >> > >> >> The GFS and Map Reduce papers give a good information on
> how
> > > > those
> > > > >> > >> systems
> > > > >> > >> >> are designed but there is nothing that concrete for Hadoop
> > > that
> > > > I
> > > > >> > have
> > > > >> > >> >> been
> > > > >> > >> >> able to find.
> > > > >> > >> >>
> > > > >> > >> >> Amandeep
> > > > >> > >> >>
> > > > >> > >> >>
> > > > >> > >> >> Amandeep Khurana
> > > > >> > >> >> Computer Science Graduate Student
> > > > >> > >> >> University of California, Santa Cruz
> > > > >> > >> >>
> > > > >> > >> >>
> > > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > > >> matei@cloudera.com>
> > > > >> > >> >> wrote:
> > > > >> > >> >>
> > > > >> > >> >> > Hi Amandeep,
> > > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims
> to
> > > > >> provide
> > > > >> > >> those
> > > > >> > >> >> > capabilities as an open-source project. HDFS is similar
> to
> > > GFS
> > > > >> > (large
> > > > >> > >> >> > blocks, replication, etc); some notable things missing
> are
> > > > >> > read-write
> > > > >> > >> >> > support in the middle of a file (unlikely to be provided
> > > > because
> > > > >> > few
> > > > >> > >> >> Hadoop
> > > > >> > >> >> > applications require it) and multiple appenders (the
> > record
> > > > >> append
> > > > >> > >> >> > operation). You can read about HDFS architecture at
> > > > >> > >> >> >
> > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > > .
> > > > >> The
> > > > >> > >> >> MapReduce
> > > > >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> > > > Google's
> > > > >> > >> >> MapReduce
> > > > >> > >> >> > interacts with GFS (shipping computation to the data),
> > > > although
> > > > >> > >> Hadoop
> > > > >> > >> >> > MapReduce also supports running over other distributed
> > > > >> filesystems.
> > > > >> > >> >> >
> > > > >> > >> >> > Matei
> > > > >> > >> >> >
> > > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > > >> > amansk@gmail.com
> > > > >> > >> >
> > > > >> > >> >> > wrote:
> > > > >> > >> >> >
> > > > >> > >> >> > > Hi
> > > > >> > >> >> > >
> > > > >> > >> >> > > Is the HDFS architecture completely based on the
> Google
> > > > >> > Filesystem?
> > > > >> > >> If
> > > > >> > >> >> it
> > > > >> > >> >> > > isnt, what are the differences between the two?
> > > > >> > >> >> > >
> > > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same
> > as
> > > > how
> > > > >> it
> > > > >> > is
> > > > >> > >> >> > between
> > > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > > >> > >> >> > >
> > > > >> > >> >> > > Amandeep
> > > > >> > >> >> > >
> > > > >> > >> >> > >
> > > > >> > >> >> > > Amandeep Khurana
> > > > >> > >> >> > > Computer Science Graduate Student
> > > > >> > >> >> > > University of California, Santa Cruz
> > > > >> > >> >> > >
> > > > >> > >> >> >
> > > > >> > >> >>
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Normally, HDFS files are accessed through the namenode. If there was a
malicious process though, then I imagine it could talk to a datanode
directly and request a specific block.
On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Ok. Got it.
>
> Now, when my job needs to access another file, does it go to the Namenode
> to
> get the block ids? How does the java process know where the files are and
> how to access them?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > I mentioned this case because even jobs written in Java can use the HDFS
> > API
> > to talk to the NameNode and access the filesystem. People often do this
> > because their job needs to read a config file, some small data table, etc
> > and use this information in its map or reduce functions. In this case,
> you
> > open the second file separately in your mapper's init function and read
> > whatever you need from it. In general I wanted to point out that you
> can't
> > know which files a job will access unless you look at its source code or
> > monitor the calls it makes; the input file(s) you provide in the job
> > description are a hint to the MapReduce framework to place your job on
> > certain nodes, but it's reasonable for the job to access other files as
> > well.
> >
> > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Another question that I have here - When the jobs run arbitrary code
> and
> > > access data from the HDFS, do they go to the namenode to get the block
> > > information?
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Assuming that the job is purely in Java and not involving streaming
> or
> > > > pipes, wouldnt the resources (files) required by the job as inputs be
> > > known
> > > > beforehand? So, if the map task is accessing a second file, how does
> it
> > > make
> > > > it different except that there are multiple files. The JobTracker
> would
> > > know
> > > > beforehand that multiple files would be accessed. Right?
> > > >
> > > > I am slightly confused why you have mentioned this case separately...
> > Can
> > > > you elaborate on it a little bit?
> > > >
> > > > Amandeep
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com>
> > > wrote:
> > > >
> > > >> Typically the data flow is like this:1) Client submits a job
> > description
> > > >> to
> > > >> the JobTracker.
> > > >> 2) JobTracker figures out block locations for the input file(s) by
> > > talking
> > > >> to HDFS NameNode.
> > > >> 3) JobTracker creates a job description file in HDFS which will be
> > read
> > > by
> > > >> the nodes to copy over the job's code etc.
> > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> > > >> appropriate data blocks.
> > > >> 5) After running, maps create intermediate output files on those
> > slaves.
> > > >> These are not in HDFS, they're in some temporary storage used by
> > > >> MapReduce.
> > > >> 6) JobTracker starts reduces on a series of slaves, which copy over
> > the
> > > >> appropriate map outputs, apply the reduce function, and write the
> > > outputs
> > > >> to
> > > >> HDFS (one output file per reducer).
> > > >> 7) Some logs for the job may also be put into HDFS by the
> JobTracker.
> > > >>
> > > >> However, there is a big caveat, which is that the map and reduce
> tasks
> > > run
> > > >> arbitrary code. It is not unusual to have a map that opens a second
> > HDFS
> > > >> file to read some information (e.g. for doing a join of a small
> table
> > > >> against a big file). If you use Hadoop Streaming or Pipes to write a
> > job
> > > >> in
> > > >> Python, Ruby, C, etc, then you are launching arbitrary processes
> which
> > > may
> > > >> also access external resources in this manner. Some people also
> > > read/write
> > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > solution
> > > >> would ideally deal with these cases too.
> > > >>
> > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <amansk@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > A quick question here. How does a typical hadoop job work at the
> > > system
> > > >> > level? What are the various interactions and how does the data
> flow?
> > > >> >
> > > >> > Amandeep
> > > >> >
> > > >> >
> > > >> > Amandeep Khurana
> > > >> > Computer Science Graduate Student
> > > >> > University of California, Santa Cruz
> > > >> >
> > > >> >
> > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> amansk@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks Matei. If the basic architecture is similar to the Google
> > > >> stuff, I
> > > >> > > can safely just work on the project using the information from
> the
> > > >> > papers.
> > > >> > >
> > > >> > > I am aware of the 4487 jira and the current status of the
> > > permissions
> > > >> > > mechanism. I had a look at them earlier.
> > > >> > >
> > > >> > > Cheers
> > > >> > > Amandeep
> > > >> > >
> > > >> > >
> > > >> > > Amandeep Khurana
> > > >> > > Computer Science Graduate Student
> > > >> > > University of California, Santa Cruz
> > > >> > >
> > > >> > >
> > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > matei@cloudera.com>
> > > >> > wrote:
> > > >> > >
> > > >> > >> Forgot to add, this JIRA details the latest security features
> > that
> > > >> are
> > > >> > >> being
> > > >> > >> worked on in Hadoop trunk:
> > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > >> > >> This document describes the current status and limitations of
> the
> > > >> > >> permissions mechanism:
> > > >> > >>
> > > >>
> > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > > >> > >>
> > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > matei@cloudera.com
> > > >
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > I think it's safe to assume that Hadoop works like
> > MapReduce/GFS
> > > at
> > > >> > the
> > > >> > >> > level described in those papers. In particular, in HDFS,
> there
> > is
> > > a
> > > >> > >> master
> > > >> > >> > node containing metadata and a number of slave nodes
> > (datanodes)
> > > >> > >> containing
> > > >> > >> > blocks, as in GFS. Clients start by talking to the master to
> > list
> > > >> > >> > directories, etc. When they want to read a region of some
> file,
> > > >> they
> > > >> > >> tell
> > > >> > >> > the master the filename and offset, and they receive a list
> of
> > > >> block
> > > >> > >> > locations (datanodes). They then contact the individual
> > datanodes
> > > >> to
> > > >> > >> read
> > > >> > >> > the blocks. When clients write a file, they first obtain a
> new
> > > >> block
> > > >> > ID
> > > >> > >> and
> > > >> > >> > list of nodes to write it to from the master, then contact
> the
> > > >> > datanodes
> > > >> > >> to
> > > >> > >> > write it (actually, the datanodes pipeline the write as in
> GFS)
> > > and
> > > >> > >> report
> > > >> > >> > when the write is complete. HDFS actually has some security
> > > >> mechanisms
> > > >> > >> built
> > > >> > >> > in, authenticating users based on their Unix ID and providing
> > > >> > Unix-like
> > > >> > >> file
> > > >> > >> > permissions. I don't know much about how these are
> implemented,
> > > but
> > > >> > they
> > > >> > >> > would be a good place to start looking.
> > > >> > >> >
> > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > >> amansk@gmail.com
> > > >> > >> >wrote:
> > > >> > >> >
> > > >> > >> >> Thanks Matie
> > > >> > >> >>
> > > >> > >> >> I had gone through the architecture document online. I am
> > > >> currently
> > > >> > >> >> working
> > > >> > >> >> on a project towards Security in Hadoop. I do know how the
> > data
> > > >> moves
> > > >> > >> >> around
> > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS follow
> > and
> > > >> how
> > > >> > >> >> different it is from GFS. Can you throw some light on that?
> > > >> > >> >>
> > > >> > >> >> Security would also involve the Map Reduce jobs following
> the
> > > same
> > > >> > >> >> protocols. Thats why the question about how does the Hadoop
> > > >> framework
> > > >> > >> >> integrate with the HDFS, and how different is it from Map
> > Reduce
> > > >> and
> > > >> > >> GFS.
> > > >> > >> >> The GFS and Map Reduce papers give a good information on how
> > > those
> > > >> > >> systems
> > > >> > >> >> are designed but there is nothing that concrete for Hadoop
> > that
> > > I
> > > >> > have
> > > >> > >> >> been
> > > >> > >> >> able to find.
> > > >> > >> >>
> > > >> > >> >> Amandeep
> > > >> > >> >>
> > > >> > >> >>
> > > >> > >> >> Amandeep Khurana
> > > >> > >> >> Computer Science Graduate Student
> > > >> > >> >> University of California, Santa Cruz
> > > >> > >> >>
> > > >> > >> >>
> > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > >> matei@cloudera.com>
> > > >> > >> >> wrote:
> > > >> > >> >>
> > > >> > >> >> > Hi Amandeep,
> > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> > > >> provide
> > > >> > >> those
> > > >> > >> >> > capabilities as an open-source project. HDFS is similar to
> > GFS
> > > >> > (large
> > > >> > >> >> > blocks, replication, etc); some notable things missing are
> > > >> > read-write
> > > >> > >> >> > support in the middle of a file (unlikely to be provided
> > > because
> > > >> > few
> > > >> > >> >> Hadoop
> > > >> > >> >> > applications require it) and multiple appenders (the
> record
> > > >> append
> > > >> > >> >> > operation). You can read about HDFS architecture at
> > > >> > >> >> >
> http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > .
> > > >> The
> > > >> > >> >> MapReduce
> > > >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> > > Google's
> > > >> > >> >> MapReduce
> > > >> > >> >> > interacts with GFS (shipping computation to the data),
> > > although
> > > >> > >> Hadoop
> > > >> > >> >> > MapReduce also supports running over other distributed
> > > >> filesystems.
> > > >> > >> >> >
> > > >> > >> >> > Matei
> > > >> > >> >> >
> > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > >> > amansk@gmail.com
> > > >> > >> >
> > > >> > >> >> > wrote:
> > > >> > >> >> >
> > > >> > >> >> > > Hi
> > > >> > >> >> > >
> > > >> > >> >> > > Is the HDFS architecture completely based on the Google
> > > >> > Filesystem?
> > > >> > >> If
> > > >> > >> >> it
> > > >> > >> >> > > isnt, what are the differences between the two?
> > > >> > >> >> > >
> > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same
> as
> > > how
> > > >> it
> > > >> > is
> > > >> > >> >> > between
> > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > >> > >> >> > >
> > > >> > >> >> > > Amandeep
> > > >> > >> >> > >
> > > >> > >> >> > >
> > > >> > >> >> > > Amandeep Khurana
> > > >> > >> >> > > Computer Science Graduate Student
> > > >> > >> >> > > University of California, Santa Cruz
> > > >> > >> >> > >
> > > >> > >> >> >
> > > >> > >> >>
> > > >> > >> >
> > > >> > >> >
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Normally, HDFS files are accessed through the namenode. If there was a
malicious process though, then I imagine it could talk to a datanode
directly and request a specific block.
On Sun, Feb 15, 2009 at 7:15 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Ok. Got it.
>
> Now, when my job needs to access another file, does it go to the Namenode
> to
> get the block ids? How does the java process know where the files are and
> how to access them?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > I mentioned this case because even jobs written in Java can use the HDFS
> > API
> > to talk to the NameNode and access the filesystem. People often do this
> > because their job needs to read a config file, some small data table, etc
> > and use this information in its map or reduce functions. In this case,
> you
> > open the second file separately in your mapper's init function and read
> > whatever you need from it. In general I wanted to point out that you
> can't
> > know which files a job will access unless you look at its source code or
> > monitor the calls it makes; the input file(s) you provide in the job
> > description are a hint to the MapReduce framework to place your job on
> > certain nodes, but it's reasonable for the job to access other files as
> > well.
> >
> > On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Another question that I have here - When the jobs run arbitrary code
> and
> > > access data from the HDFS, do they go to the namenode to get the block
> > > information?
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> > > wrote:
> > >
> > > > Assuming that the job is purely in Java and not involving streaming
> or
> > > > pipes, wouldnt the resources (files) required by the job as inputs be
> > > known
> > > > beforehand? So, if the map task is accessing a second file, how does
> it
> > > make
> > > > it different except that there are multiple files. The JobTracker
> would
> > > know
> > > > beforehand that multiple files would be accessed. Right?
> > > >
> > > > I am slightly confused why you have mentioned this case separately...
> > Can
> > > > you elaborate on it a little bit?
> > > >
> > > > Amandeep
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com>
> > > wrote:
> > > >
> > > >> Typically the data flow is like this:1) Client submits a job
> > description
> > > >> to
> > > >> the JobTracker.
> > > >> 2) JobTracker figures out block locations for the input file(s) by
> > > talking
> > > >> to HDFS NameNode.
> > > >> 3) JobTracker creates a job description file in HDFS which will be
> > read
> > > by
> > > >> the nodes to copy over the job's code etc.
> > > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> > > >> appropriate data blocks.
> > > >> 5) After running, maps create intermediate output files on those
> > slaves.
> > > >> These are not in HDFS, they're in some temporary storage used by
> > > >> MapReduce.
> > > >> 6) JobTracker starts reduces on a series of slaves, which copy over
> > the
> > > >> appropriate map outputs, apply the reduce function, and write the
> > > outputs
> > > >> to
> > > >> HDFS (one output file per reducer).
> > > >> 7) Some logs for the job may also be put into HDFS by the
> JobTracker.
> > > >>
> > > >> However, there is a big caveat, which is that the map and reduce
> tasks
> > > run
> > > >> arbitrary code. It is not unusual to have a map that opens a second
> > HDFS
> > > >> file to read some information (e.g. for doing a join of a small
> table
> > > >> against a big file). If you use Hadoop Streaming or Pipes to write a
> > job
> > > >> in
> > > >> Python, Ruby, C, etc, then you are launching arbitrary processes
> which
> > > may
> > > >> also access external resources in this manner. Some people also
> > > read/write
> > > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> > solution
> > > >> would ideally deal with these cases too.
> > > >>
> > > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <amansk@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > A quick question here. How does a typical hadoop job work at the
> > > system
> > > >> > level? What are the various interactions and how does the data
> flow?
> > > >> >
> > > >> > Amandeep
> > > >> >
> > > >> >
> > > >> > Amandeep Khurana
> > > >> > Computer Science Graduate Student
> > > >> > University of California, Santa Cruz
> > > >> >
> > > >> >
> > > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <
> amansk@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks Matei. If the basic architecture is similar to the Google
> > > >> stuff, I
> > > >> > > can safely just work on the project using the information from
> the
> > > >> > papers.
> > > >> > >
> > > >> > > I am aware of the 4487 jira and the current status of the
> > > permissions
> > > >> > > mechanism. I had a look at them earlier.
> > > >> > >
> > > >> > > Cheers
> > > >> > > Amandeep
> > > >> > >
> > > >> > >
> > > >> > > Amandeep Khurana
> > > >> > > Computer Science Graduate Student
> > > >> > > University of California, Santa Cruz
> > > >> > >
> > > >> > >
> > > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> > matei@cloudera.com>
> > > >> > wrote:
> > > >> > >
> > > >> > >> Forgot to add, this JIRA details the latest security features
> > that
> > > >> are
> > > >> > >> being
> > > >> > >> worked on in Hadoop trunk:
> > > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > > >> > >> This document describes the current status and limitations of
> the
> > > >> > >> permissions mechanism:
> > > >> > >>
> > > >>
> > http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > > >> > >>
> > > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> > matei@cloudera.com
> > > >
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > I think it's safe to assume that Hadoop works like
> > MapReduce/GFS
> > > at
> > > >> > the
> > > >> > >> > level described in those papers. In particular, in HDFS,
> there
> > is
> > > a
> > > >> > >> master
> > > >> > >> > node containing metadata and a number of slave nodes
> > (datanodes)
> > > >> > >> containing
> > > >> > >> > blocks, as in GFS. Clients start by talking to the master to
> > list
> > > >> > >> > directories, etc. When they want to read a region of some
> file,
> > > >> they
> > > >> > >> tell
> > > >> > >> > the master the filename and offset, and they receive a list
> of
> > > >> block
> > > >> > >> > locations (datanodes). They then contact the individual
> > datanodes
> > > >> to
> > > >> > >> read
> > > >> > >> > the blocks. When clients write a file, they first obtain a
> new
> > > >> block
> > > >> > ID
> > > >> > >> and
> > > >> > >> > list of nodes to write it to from the master, then contact
> the
> > > >> > datanodes
> > > >> > >> to
> > > >> > >> > write it (actually, the datanodes pipeline the write as in
> GFS)
> > > and
> > > >> > >> report
> > > >> > >> > when the write is complete. HDFS actually has some security
> > > >> mechanisms
> > > >> > >> built
> > > >> > >> > in, authenticating users based on their Unix ID and providing
> > > >> > Unix-like
> > > >> > >> file
> > > >> > >> > permissions. I don't know much about how these are
> implemented,
> > > but
> > > >> > they
> > > >> > >> > would be a good place to start looking.
> > > >> > >> >
> > > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > > >> amansk@gmail.com
> > > >> > >> >wrote:
> > > >> > >> >
> > > >> > >> >> Thanks Matie
> > > >> > >> >>
> > > >> > >> >> I had gone through the architecture document online. I am
> > > >> currently
> > > >> > >> >> working
> > > >> > >> >> on a project towards Security in Hadoop. I do know how the
> > data
> > > >> moves
> > > >> > >> >> around
> > > >> > >> >> in the GFS but wasnt sure how much of that does HDFS follow
> > and
> > > >> how
> > > >> > >> >> different it is from GFS. Can you throw some light on that?
> > > >> > >> >>
> > > >> > >> >> Security would also involve the Map Reduce jobs following
> the
> > > same
> > > >> > >> >> protocols. Thats why the question about how does the Hadoop
> > > >> framework
> > > >> > >> >> integrate with the HDFS, and how different is it from Map
> > Reduce
> > > >> and
> > > >> > >> GFS.
> > > >> > >> >> The GFS and Map Reduce papers give a good information on how
> > > those
> > > >> > >> systems
> > > >> > >> >> are designed but there is nothing that concrete for Hadoop
> > that
> > > I
> > > >> > have
> > > >> > >> >> been
> > > >> > >> >> able to find.
> > > >> > >> >>
> > > >> > >> >> Amandeep
> > > >> > >> >>
> > > >> > >> >>
> > > >> > >> >> Amandeep Khurana
> > > >> > >> >> Computer Science Graduate Student
> > > >> > >> >> University of California, Santa Cruz
> > > >> > >> >>
> > > >> > >> >>
> > > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > > >> matei@cloudera.com>
> > > >> > >> >> wrote:
> > > >> > >> >>
> > > >> > >> >> > Hi Amandeep,
> > > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> > > >> provide
> > > >> > >> those
> > > >> > >> >> > capabilities as an open-source project. HDFS is similar to
> > GFS
> > > >> > (large
> > > >> > >> >> > blocks, replication, etc); some notable things missing are
> > > >> > read-write
> > > >> > >> >> > support in the middle of a file (unlikely to be provided
> > > because
> > > >> > few
> > > >> > >> >> Hadoop
> > > >> > >> >> > applications require it) and multiple appenders (the
> record
> > > >> append
> > > >> > >> >> > operation). You can read about HDFS architecture at
> > > >> > >> >> >
> http://hadoop.apache.org/core/docs/current/hdfs_design.html
> > .
> > > >> The
> > > >> > >> >> MapReduce
> > > >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> > > Google's
> > > >> > >> >> MapReduce
> > > >> > >> >> > interacts with GFS (shipping computation to the data),
> > > although
> > > >> > >> Hadoop
> > > >> > >> >> > MapReduce also supports running over other distributed
> > > >> filesystems.
> > > >> > >> >> >
> > > >> > >> >> > Matei
> > > >> > >> >> >
> > > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > > >> > amansk@gmail.com
> > > >> > >> >
> > > >> > >> >> > wrote:
> > > >> > >> >> >
> > > >> > >> >> > > Hi
> > > >> > >> >> > >
> > > >> > >> >> > > Is the HDFS architecture completely based on the Google
> > > >> > Filesystem?
> > > >> > >> If
> > > >> > >> >> it
> > > >> > >> >> > > isnt, what are the differences between the two?
> > > >> > >> >> > >
> > > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same
> as
> > > how
> > > >> it
> > > >> > is
> > > >> > >> >> > between
> > > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > > >> > >> >> > >
> > > >> > >> >> > > Amandeep
> > > >> > >> >> > >
> > > >> > >> >> > >
> > > >> > >> >> > > Amandeep Khurana
> > > >> > >> >> > > Computer Science Graduate Student
> > > >> > >> >> > > University of California, Santa Cruz
> > > >> > >> >> > >
> > > >> > >> >> >
> > > >> > >> >>
> > > >> > >> >
> > > >> > >> >
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Ok. Got it.
Now, when my job needs to access another file, does it go to the Namenode to
get the block ids? How does the java process know where the files are and
how to access them?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> I mentioned this case because even jobs written in Java can use the HDFS
> API
> to talk to the NameNode and access the filesystem. People often do this
> because their job needs to read a config file, some small data table, etc
> and use this information in its map or reduce functions. In this case, you
> open the second file separately in your mapper's init function and read
> whatever you need from it. In general I wanted to point out that you can't
> know which files a job will access unless you look at its source code or
> monitor the calls it makes; the input file(s) you provide in the job
> description are a hint to the MapReduce framework to place your job on
> certain nodes, but it's reasonable for the job to access other files as
> well.
>
> On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Another question that I have here - When the jobs run arbitrary code and
> > access data from the HDFS, do they go to the namenode to get the block
> > information?
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Assuming that the job is purely in Java and not involving streaming or
> > > pipes, wouldnt the resources (files) required by the job as inputs be
> > known
> > > beforehand? So, if the map task is accessing a second file, how does it
> > make
> > > it different except that there are multiple files. The JobTracker would
> > know
> > > beforehand that multiple files would be accessed. Right?
> > >
> > > I am slightly confused why you have mentioned this case separately...
> Can
> > > you elaborate on it a little bit?
> > >
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > >> Typically the data flow is like this:1) Client submits a job
> description
> > >> to
> > >> the JobTracker.
> > >> 2) JobTracker figures out block locations for the input file(s) by
> > talking
> > >> to HDFS NameNode.
> > >> 3) JobTracker creates a job description file in HDFS which will be
> read
> > by
> > >> the nodes to copy over the job's code etc.
> > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> > >> appropriate data blocks.
> > >> 5) After running, maps create intermediate output files on those
> slaves.
> > >> These are not in HDFS, they're in some temporary storage used by
> > >> MapReduce.
> > >> 6) JobTracker starts reduces on a series of slaves, which copy over
> the
> > >> appropriate map outputs, apply the reduce function, and write the
> > outputs
> > >> to
> > >> HDFS (one output file per reducer).
> > >> 7) Some logs for the job may also be put into HDFS by the JobTracker.
> > >>
> > >> However, there is a big caveat, which is that the map and reduce tasks
> > run
> > >> arbitrary code. It is not unusual to have a map that opens a second
> HDFS
> > >> file to read some information (e.g. for doing a join of a small table
> > >> against a big file). If you use Hadoop Streaming or Pipes to write a
> job
> > >> in
> > >> Python, Ruby, C, etc, then you are launching arbitrary processes which
> > may
> > >> also access external resources in this manner. Some people also
> > read/write
> > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> solution
> > >> would ideally deal with these cases too.
> > >>
> > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> > >> wrote:
> > >>
> > >> > A quick question here. How does a typical hadoop job work at the
> > system
> > >> > level? What are the various interactions and how does the data flow?
> > >> >
> > >> > Amandeep
> > >> >
> > >> >
> > >> > Amandeep Khurana
> > >> > Computer Science Graduate Student
> > >> > University of California, Santa Cruz
> > >> >
> > >> >
> > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <amansk@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > Thanks Matei. If the basic architecture is similar to the Google
> > >> stuff, I
> > >> > > can safely just work on the project using the information from the
> > >> > papers.
> > >> > >
> > >> > > I am aware of the 4487 jira and the current status of the
> > permissions
> > >> > > mechanism. I had a look at them earlier.
> > >> > >
> > >> > > Cheers
> > >> > > Amandeep
> > >> > >
> > >> > >
> > >> > > Amandeep Khurana
> > >> > > Computer Science Graduate Student
> > >> > > University of California, Santa Cruz
> > >> > >
> > >> > >
> > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> matei@cloudera.com>
> > >> > wrote:
> > >> > >
> > >> > >> Forgot to add, this JIRA details the latest security features
> that
> > >> are
> > >> > >> being
> > >> > >> worked on in Hadoop trunk:
> > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > >> > >> This document describes the current status and limitations of the
> > >> > >> permissions mechanism:
> > >> > >>
> > >>
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > >> > >>
> > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> matei@cloudera.com
> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > I think it's safe to assume that Hadoop works like
> MapReduce/GFS
> > at
> > >> > the
> > >> > >> > level described in those papers. In particular, in HDFS, there
> is
> > a
> > >> > >> master
> > >> > >> > node containing metadata and a number of slave nodes
> (datanodes)
> > >> > >> containing
> > >> > >> > blocks, as in GFS. Clients start by talking to the master to
> list
> > >> > >> > directories, etc. When they want to read a region of some file,
> > >> they
> > >> > >> tell
> > >> > >> > the master the filename and offset, and they receive a list of
> > >> block
> > >> > >> > locations (datanodes). They then contact the individual
> datanodes
> > >> to
> > >> > >> read
> > >> > >> > the blocks. When clients write a file, they first obtain a new
> > >> block
> > >> > ID
> > >> > >> and
> > >> > >> > list of nodes to write it to from the master, then contact the
> > >> > datanodes
> > >> > >> to
> > >> > >> > write it (actually, the datanodes pipeline the write as in GFS)
> > and
> > >> > >> report
> > >> > >> > when the write is complete. HDFS actually has some security
> > >> mechanisms
> > >> > >> built
> > >> > >> > in, authenticating users based on their Unix ID and providing
> > >> > Unix-like
> > >> > >> file
> > >> > >> > permissions. I don't know much about how these are implemented,
> > but
> > >> > they
> > >> > >> > would be a good place to start looking.
> > >> > >> >
> > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > >> amansk@gmail.com
> > >> > >> >wrote:
> > >> > >> >
> > >> > >> >> Thanks Matie
> > >> > >> >>
> > >> > >> >> I had gone through the architecture document online. I am
> > >> currently
> > >> > >> >> working
> > >> > >> >> on a project towards Security in Hadoop. I do know how the
> data
> > >> moves
> > >> > >> >> around
> > >> > >> >> in the GFS but wasnt sure how much of that does HDFS follow
> and
> > >> how
> > >> > >> >> different it is from GFS. Can you throw some light on that?
> > >> > >> >>
> > >> > >> >> Security would also involve the Map Reduce jobs following the
> > same
> > >> > >> >> protocols. Thats why the question about how does the Hadoop
> > >> framework
> > >> > >> >> integrate with the HDFS, and how different is it from Map
> Reduce
> > >> and
> > >> > >> GFS.
> > >> > >> >> The GFS and Map Reduce papers give a good information on how
> > those
> > >> > >> systems
> > >> > >> >> are designed but there is nothing that concrete for Hadoop
> that
> > I
> > >> > have
> > >> > >> >> been
> > >> > >> >> able to find.
> > >> > >> >>
> > >> > >> >> Amandeep
> > >> > >> >>
> > >> > >> >>
> > >> > >> >> Amandeep Khurana
> > >> > >> >> Computer Science Graduate Student
> > >> > >> >> University of California, Santa Cruz
> > >> > >> >>
> > >> > >> >>
> > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > >> matei@cloudera.com>
> > >> > >> >> wrote:
> > >> > >> >>
> > >> > >> >> > Hi Amandeep,
> > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> > >> provide
> > >> > >> those
> > >> > >> >> > capabilities as an open-source project. HDFS is similar to
> GFS
> > >> > (large
> > >> > >> >> > blocks, replication, etc); some notable things missing are
> > >> > read-write
> > >> > >> >> > support in the middle of a file (unlikely to be provided
> > because
> > >> > few
> > >> > >> >> Hadoop
> > >> > >> >> > applications require it) and multiple appenders (the record
> > >> append
> > >> > >> >> > operation). You can read about HDFS architecture at
> > >> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> .
> > >> The
> > >> > >> >> MapReduce
> > >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> > Google's
> > >> > >> >> MapReduce
> > >> > >> >> > interacts with GFS (shipping computation to the data),
> > although
> > >> > >> Hadoop
> > >> > >> >> > MapReduce also supports running over other distributed
> > >> filesystems.
> > >> > >> >> >
> > >> > >> >> > Matei
> > >> > >> >> >
> > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > >> > amansk@gmail.com
> > >> > >> >
> > >> > >> >> > wrote:
> > >> > >> >> >
> > >> > >> >> > > Hi
> > >> > >> >> > >
> > >> > >> >> > > Is the HDFS architecture completely based on the Google
> > >> > Filesystem?
> > >> > >> If
> > >> > >> >> it
> > >> > >> >> > > isnt, what are the differences between the two?
> > >> > >> >> > >
> > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as
> > how
> > >> it
> > >> > is
> > >> > >> >> > between
> > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > >> > >> >> > >
> > >> > >> >> > > Amandeep
> > >> > >> >> > >
> > >> > >> >> > >
> > >> > >> >> > > Amandeep Khurana
> > >> > >> >> > > Computer Science Graduate Student
> > >> > >> >> > > University of California, Santa Cruz
> > >> > >> >> > >
> > >> > >> >> >
> > >> > >> >>
> > >> > >> >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Ok. Got it.
Now, when my job needs to access another file, does it go to the Namenode to
get the block ids? How does the java process know where the files are and
how to access them?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 7:05 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> I mentioned this case because even jobs written in Java can use the HDFS
> API
> to talk to the NameNode and access the filesystem. People often do this
> because their job needs to read a config file, some small data table, etc
> and use this information in its map or reduce functions. In this case, you
> open the second file separately in your mapper's init function and read
> whatever you need from it. In general I wanted to point out that you can't
> know which files a job will access unless you look at its source code or
> monitor the calls it makes; the input file(s) you provide in the job
> description are a hint to the MapReduce framework to place your job on
> certain nodes, but it's reasonable for the job to access other files as
> well.
>
> On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Another question that I have here - When the jobs run arbitrary code and
> > access data from the HDFS, do they go to the namenode to get the block
> > information?
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Assuming that the job is purely in Java and not involving streaming or
> > > pipes, wouldnt the resources (files) required by the job as inputs be
> > known
> > > beforehand? So, if the map task is accessing a second file, how does it
> > make
> > > it different except that there are multiple files. The JobTracker would
> > know
> > > beforehand that multiple files would be accessed. Right?
> > >
> > > I am slightly confused why you have mentioned this case separately...
> Can
> > > you elaborate on it a little bit?
> > >
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > >> Typically the data flow is like this:1) Client submits a job
> description
> > >> to
> > >> the JobTracker.
> > >> 2) JobTracker figures out block locations for the input file(s) by
> > talking
> > >> to HDFS NameNode.
> > >> 3) JobTracker creates a job description file in HDFS which will be
> read
> > by
> > >> the nodes to copy over the job's code etc.
> > >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> > >> appropriate data blocks.
> > >> 5) After running, maps create intermediate output files on those
> slaves.
> > >> These are not in HDFS, they're in some temporary storage used by
> > >> MapReduce.
> > >> 6) JobTracker starts reduces on a series of slaves, which copy over
> the
> > >> appropriate map outputs, apply the reduce function, and write the
> > outputs
> > >> to
> > >> HDFS (one output file per reducer).
> > >> 7) Some logs for the job may also be put into HDFS by the JobTracker.
> > >>
> > >> However, there is a big caveat, which is that the map and reduce tasks
> > run
> > >> arbitrary code. It is not unusual to have a map that opens a second
> HDFS
> > >> file to read some information (e.g. for doing a join of a small table
> > >> against a big file). If you use Hadoop Streaming or Pipes to write a
> job
> > >> in
> > >> Python, Ruby, C, etc, then you are launching arbitrary processes which
> > may
> > >> also access external resources in this manner. Some people also
> > read/write
> > >> to DBs (e.g. MySQL) from their tasks. A comprehensive security
> solution
> > >> would ideally deal with these cases too.
> > >>
> > >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> > >> wrote:
> > >>
> > >> > A quick question here. How does a typical hadoop job work at the
> > system
> > >> > level? What are the various interactions and how does the data flow?
> > >> >
> > >> > Amandeep
> > >> >
> > >> >
> > >> > Amandeep Khurana
> > >> > Computer Science Graduate Student
> > >> > University of California, Santa Cruz
> > >> >
> > >> >
> > >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <amansk@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > Thanks Matei. If the basic architecture is similar to the Google
> > >> stuff, I
> > >> > > can safely just work on the project using the information from the
> > >> > papers.
> > >> > >
> > >> > > I am aware of the 4487 jira and the current status of the
> > permissions
> > >> > > mechanism. I had a look at them earlier.
> > >> > >
> > >> > > Cheers
> > >> > > Amandeep
> > >> > >
> > >> > >
> > >> > > Amandeep Khurana
> > >> > > Computer Science Graduate Student
> > >> > > University of California, Santa Cruz
> > >> > >
> > >> > >
> > >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <
> matei@cloudera.com>
> > >> > wrote:
> > >> > >
> > >> > >> Forgot to add, this JIRA details the latest security features
> that
> > >> are
> > >> > >> being
> > >> > >> worked on in Hadoop trunk:
> > >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > >> > >> This document describes the current status and limitations of the
> > >> > >> permissions mechanism:
> > >> > >>
> > >>
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > >> > >>
> > >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <
> matei@cloudera.com
> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > I think it's safe to assume that Hadoop works like
> MapReduce/GFS
> > at
> > >> > the
> > >> > >> > level described in those papers. In particular, in HDFS, there
> is
> > a
> > >> > >> master
> > >> > >> > node containing metadata and a number of slave nodes
> (datanodes)
> > >> > >> containing
> > >> > >> > blocks, as in GFS. Clients start by talking to the master to
> list
> > >> > >> > directories, etc. When they want to read a region of some file,
> > >> they
> > >> > >> tell
> > >> > >> > the master the filename and offset, and they receive a list of
> > >> block
> > >> > >> > locations (datanodes). They then contact the individual
> datanodes
> > >> to
> > >> > >> read
> > >> > >> > the blocks. When clients write a file, they first obtain a new
> > >> block
> > >> > ID
> > >> > >> and
> > >> > >> > list of nodes to write it to from the master, then contact the
> > >> > datanodes
> > >> > >> to
> > >> > >> > write it (actually, the datanodes pipeline the write as in GFS)
> > and
> > >> > >> report
> > >> > >> > when the write is complete. HDFS actually has some security
> > >> mechanisms
> > >> > >> built
> > >> > >> > in, authenticating users based on their Unix ID and providing
> > >> > Unix-like
> > >> > >> file
> > >> > >> > permissions. I don't know much about how these are implemented,
> > but
> > >> > they
> > >> > >> > would be a good place to start looking.
> > >> > >> >
> > >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> > >> amansk@gmail.com
> > >> > >> >wrote:
> > >> > >> >
> > >> > >> >> Thanks Matie
> > >> > >> >>
> > >> > >> >> I had gone through the architecture document online. I am
> > >> currently
> > >> > >> >> working
> > >> > >> >> on a project towards Security in Hadoop. I do know how the
> data
> > >> moves
> > >> > >> >> around
> > >> > >> >> in the GFS but wasnt sure how much of that does HDFS follow
> and
> > >> how
> > >> > >> >> different it is from GFS. Can you throw some light on that?
> > >> > >> >>
> > >> > >> >> Security would also involve the Map Reduce jobs following the
> > same
> > >> > >> >> protocols. Thats why the question about how does the Hadoop
> > >> framework
> > >> > >> >> integrate with the HDFS, and how different is it from Map
> Reduce
> > >> and
> > >> > >> GFS.
> > >> > >> >> The GFS and Map Reduce papers give a good information on how
> > those
> > >> > >> systems
> > >> > >> >> are designed but there is nothing that concrete for Hadoop
> that
> > I
> > >> > have
> > >> > >> >> been
> > >> > >> >> able to find.
> > >> > >> >>
> > >> > >> >> Amandeep
> > >> > >> >>
> > >> > >> >>
> > >> > >> >> Amandeep Khurana
> > >> > >> >> Computer Science Graduate Student
> > >> > >> >> University of California, Santa Cruz
> > >> > >> >>
> > >> > >> >>
> > >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> > >> matei@cloudera.com>
> > >> > >> >> wrote:
> > >> > >> >>
> > >> > >> >> > Hi Amandeep,
> > >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> > >> provide
> > >> > >> those
> > >> > >> >> > capabilities as an open-source project. HDFS is similar to
> GFS
> > >> > (large
> > >> > >> >> > blocks, replication, etc); some notable things missing are
> > >> > read-write
> > >> > >> >> > support in the middle of a file (unlikely to be provided
> > because
> > >> > few
> > >> > >> >> Hadoop
> > >> > >> >> > applications require it) and multiple appenders (the record
> > >> append
> > >> > >> >> > operation). You can read about HDFS architecture at
> > >> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html
> .
> > >> The
> > >> > >> >> MapReduce
> > >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> > Google's
> > >> > >> >> MapReduce
> > >> > >> >> > interacts with GFS (shipping computation to the data),
> > although
> > >> > >> Hadoop
> > >> > >> >> > MapReduce also supports running over other distributed
> > >> filesystems.
> > >> > >> >> >
> > >> > >> >> > Matei
> > >> > >> >> >
> > >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > >> > amansk@gmail.com
> > >> > >> >
> > >> > >> >> > wrote:
> > >> > >> >> >
> > >> > >> >> > > Hi
> > >> > >> >> > >
> > >> > >> >> > > Is the HDFS architecture completely based on the Google
> > >> > Filesystem?
> > >> > >> If
> > >> > >> >> it
> > >> > >> >> > > isnt, what are the differences between the two?
> > >> > >> >> > >
> > >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as
> > how
> > >> it
> > >> > is
> > >> > >> >> > between
> > >> > >> >> > > the Google's version of Map Reduce and GFS?
> > >> > >> >> > >
> > >> > >> >> > > Amandeep
> > >> > >> >> > >
> > >> > >> >> > >
> > >> > >> >> > > Amandeep Khurana
> > >> > >> >> > > Computer Science Graduate Student
> > >> > >> >> > > University of California, Santa Cruz
> > >> > >> >> > >
> > >> > >> >> >
> > >> > >> >>
> > >> > >> >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
I mentioned this case because even jobs written in Java can use the HDFS API
to talk to the NameNode and access the filesystem. People often do this
because their job needs to read a config file, some small data table, etc
and use this information in its map or reduce functions. In this case, you
open the second file separately in your mapper's init function and read
whatever you need from it. In general I wanted to point out that you can't
know which files a job will access unless you look at its source code or
monitor the calls it makes; the input file(s) you provide in the job
description are a hint to the MapReduce framework to place your job on
certain nodes, but it's reasonable for the job to access other files as
well.
On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Another question that I have here - When the jobs run arbitrary code and
> access data from the HDFS, do they go to the namenode to get the block
> information?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Assuming that the job is purely in Java and not involving streaming or
> > pipes, wouldnt the resources (files) required by the job as inputs be
> known
> > beforehand? So, if the map task is accessing a second file, how does it
> make
> > it different except that there are multiple files. The JobTracker would
> know
> > beforehand that multiple files would be accessed. Right?
> >
> > I am slightly confused why you have mentioned this case separately... Can
> > you elaborate on it a little bit?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> >> Typically the data flow is like this:1) Client submits a job description
> >> to
> >> the JobTracker.
> >> 2) JobTracker figures out block locations for the input file(s) by
> talking
> >> to HDFS NameNode.
> >> 3) JobTracker creates a job description file in HDFS which will be read
> by
> >> the nodes to copy over the job's code etc.
> >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> >> appropriate data blocks.
> >> 5) After running, maps create intermediate output files on those slaves.
> >> These are not in HDFS, they're in some temporary storage used by
> >> MapReduce.
> >> 6) JobTracker starts reduces on a series of slaves, which copy over the
> >> appropriate map outputs, apply the reduce function, and write the
> outputs
> >> to
> >> HDFS (one output file per reducer).
> >> 7) Some logs for the job may also be put into HDFS by the JobTracker.
> >>
> >> However, there is a big caveat, which is that the map and reduce tasks
> run
> >> arbitrary code. It is not unusual to have a map that opens a second HDFS
> >> file to read some information (e.g. for doing a join of a small table
> >> against a big file). If you use Hadoop Streaming or Pipes to write a job
> >> in
> >> Python, Ruby, C, etc, then you are launching arbitrary processes which
> may
> >> also access external resources in this manner. Some people also
> read/write
> >> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
> >> would ideally deal with these cases too.
> >>
> >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> >> wrote:
> >>
> >> > A quick question here. How does a typical hadoop job work at the
> system
> >> > level? What are the various interactions and how does the data flow?
> >> >
> >> > Amandeep
> >> >
> >> >
> >> > Amandeep Khurana
> >> > Computer Science Graduate Student
> >> > University of California, Santa Cruz
> >> >
> >> >
> >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> >> > wrote:
> >> >
> >> > > Thanks Matei. If the basic architecture is similar to the Google
> >> stuff, I
> >> > > can safely just work on the project using the information from the
> >> > papers.
> >> > >
> >> > > I am aware of the 4487 jira and the current status of the
> permissions
> >> > > mechanism. I had a look at them earlier.
> >> > >
> >> > > Cheers
> >> > > Amandeep
> >> > >
> >> > >
> >> > > Amandeep Khurana
> >> > > Computer Science Graduate Student
> >> > > University of California, Santa Cruz
> >> > >
> >> > >
> >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> >> > wrote:
> >> > >
> >> > >> Forgot to add, this JIRA details the latest security features that
> >> are
> >> > >> being
> >> > >> worked on in Hadoop trunk:
> >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> >> > >> This document describes the current status and limitations of the
> >> > >> permissions mechanism:
> >> > >>
> >> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> >> > >>
> >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <matei@cloudera.com
> >
> >> > >> wrote:
> >> > >>
> >> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS
> at
> >> > the
> >> > >> > level described in those papers. In particular, in HDFS, there is
> a
> >> > >> master
> >> > >> > node containing metadata and a number of slave nodes (datanodes)
> >> > >> containing
> >> > >> > blocks, as in GFS. Clients start by talking to the master to list
> >> > >> > directories, etc. When they want to read a region of some file,
> >> they
> >> > >> tell
> >> > >> > the master the filename and offset, and they receive a list of
> >> block
> >> > >> > locations (datanodes). They then contact the individual datanodes
> >> to
> >> > >> read
> >> > >> > the blocks. When clients write a file, they first obtain a new
> >> block
> >> > ID
> >> > >> and
> >> > >> > list of nodes to write it to from the master, then contact the
> >> > datanodes
> >> > >> to
> >> > >> > write it (actually, the datanodes pipeline the write as in GFS)
> and
> >> > >> report
> >> > >> > when the write is complete. HDFS actually has some security
> >> mechanisms
> >> > >> built
> >> > >> > in, authenticating users based on their Unix ID and providing
> >> > Unix-like
> >> > >> file
> >> > >> > permissions. I don't know much about how these are implemented,
> but
> >> > they
> >> > >> > would be a good place to start looking.
> >> > >> >
> >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> >> amansk@gmail.com
> >> > >> >wrote:
> >> > >> >
> >> > >> >> Thanks Matie
> >> > >> >>
> >> > >> >> I had gone through the architecture document online. I am
> >> currently
> >> > >> >> working
> >> > >> >> on a project towards Security in Hadoop. I do know how the data
> >> moves
> >> > >> >> around
> >> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and
> >> how
> >> > >> >> different it is from GFS. Can you throw some light on that?
> >> > >> >>
> >> > >> >> Security would also involve the Map Reduce jobs following the
> same
> >> > >> >> protocols. Thats why the question about how does the Hadoop
> >> framework
> >> > >> >> integrate with the HDFS, and how different is it from Map Reduce
> >> and
> >> > >> GFS.
> >> > >> >> The GFS and Map Reduce papers give a good information on how
> those
> >> > >> systems
> >> > >> >> are designed but there is nothing that concrete for Hadoop that
> I
> >> > have
> >> > >> >> been
> >> > >> >> able to find.
> >> > >> >>
> >> > >> >> Amandeep
> >> > >> >>
> >> > >> >>
> >> > >> >> Amandeep Khurana
> >> > >> >> Computer Science Graduate Student
> >> > >> >> University of California, Santa Cruz
> >> > >> >>
> >> > >> >>
> >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> >> matei@cloudera.com>
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >> > Hi Amandeep,
> >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> >> provide
> >> > >> those
> >> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
> >> > (large
> >> > >> >> > blocks, replication, etc); some notable things missing are
> >> > read-write
> >> > >> >> > support in the middle of a file (unlikely to be provided
> because
> >> > few
> >> > >> >> Hadoop
> >> > >> >> > applications require it) and multiple appenders (the record
> >> append
> >> > >> >> > operation). You can read about HDFS architecture at
> >> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html.
> >> The
> >> > >> >> MapReduce
> >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> Google's
> >> > >> >> MapReduce
> >> > >> >> > interacts with GFS (shipping computation to the data),
> although
> >> > >> Hadoop
> >> > >> >> > MapReduce also supports running over other distributed
> >> filesystems.
> >> > >> >> >
> >> > >> >> > Matei
> >> > >> >> >
> >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> >> > amansk@gmail.com
> >> > >> >
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > Hi
> >> > >> >> > >
> >> > >> >> > > Is the HDFS architecture completely based on the Google
> >> > Filesystem?
> >> > >> If
> >> > >> >> it
> >> > >> >> > > isnt, what are the differences between the two?
> >> > >> >> > >
> >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as
> how
> >> it
> >> > is
> >> > >> >> > between
> >> > >> >> > > the Google's version of Map Reduce and GFS?
> >> > >> >> > >
> >> > >> >> > > Amandeep
> >> > >> >> > >
> >> > >> >> > >
> >> > >> >> > > Amandeep Khurana
> >> > >> >> > > Computer Science Graduate Student
> >> > >> >> > > University of California, Santa Cruz
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
I mentioned this case because even jobs written in Java can use the HDFS API
to talk to the NameNode and access the filesystem. People often do this
because their job needs to read a config file, some small data table, etc
and use this information in its map or reduce functions. In this case, you
open the second file separately in your mapper's init function and read
whatever you need from it. In general I wanted to point out that you can't
know which files a job will access unless you look at its source code or
monitor the calls it makes; the input file(s) you provide in the job
description are a hint to the MapReduce framework to place your job on
certain nodes, but it's reasonable for the job to access other files as
well.
On Sun, Feb 15, 2009 at 6:14 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Another question that I have here - When the jobs run arbitrary code and
> access data from the HDFS, do they go to the namenode to get the block
> information?
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Assuming that the job is purely in Java and not involving streaming or
> > pipes, wouldnt the resources (files) required by the job as inputs be
> known
> > beforehand? So, if the map task is accessing a second file, how does it
> make
> > it different except that there are multiple files. The JobTracker would
> know
> > beforehand that multiple files would be accessed. Right?
> >
> > I am slightly confused why you have mentioned this case separately... Can
> > you elaborate on it a little bit?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> >> Typically the data flow is like this:1) Client submits a job description
> >> to
> >> the JobTracker.
> >> 2) JobTracker figures out block locations for the input file(s) by
> talking
> >> to HDFS NameNode.
> >> 3) JobTracker creates a job description file in HDFS which will be read
> by
> >> the nodes to copy over the job's code etc.
> >> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> >> appropriate data blocks.
> >> 5) After running, maps create intermediate output files on those slaves.
> >> These are not in HDFS, they're in some temporary storage used by
> >> MapReduce.
> >> 6) JobTracker starts reduces on a series of slaves, which copy over the
> >> appropriate map outputs, apply the reduce function, and write the
> outputs
> >> to
> >> HDFS (one output file per reducer).
> >> 7) Some logs for the job may also be put into HDFS by the JobTracker.
> >>
> >> However, there is a big caveat, which is that the map and reduce tasks
> run
> >> arbitrary code. It is not unusual to have a map that opens a second HDFS
> >> file to read some information (e.g. for doing a join of a small table
> >> against a big file). If you use Hadoop Streaming or Pipes to write a job
> >> in
> >> Python, Ruby, C, etc, then you are launching arbitrary processes which
> may
> >> also access external resources in this manner. Some people also
> read/write
> >> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
> >> would ideally deal with these cases too.
> >>
> >> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> >> wrote:
> >>
> >> > A quick question here. How does a typical hadoop job work at the
> system
> >> > level? What are the various interactions and how does the data flow?
> >> >
> >> > Amandeep
> >> >
> >> >
> >> > Amandeep Khurana
> >> > Computer Science Graduate Student
> >> > University of California, Santa Cruz
> >> >
> >> >
> >> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> >> > wrote:
> >> >
> >> > > Thanks Matei. If the basic architecture is similar to the Google
> >> stuff, I
> >> > > can safely just work on the project using the information from the
> >> > papers.
> >> > >
> >> > > I am aware of the 4487 jira and the current status of the
> permissions
> >> > > mechanism. I had a look at them earlier.
> >> > >
> >> > > Cheers
> >> > > Amandeep
> >> > >
> >> > >
> >> > > Amandeep Khurana
> >> > > Computer Science Graduate Student
> >> > > University of California, Santa Cruz
> >> > >
> >> > >
> >> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> >> > wrote:
> >> > >
> >> > >> Forgot to add, this JIRA details the latest security features that
> >> are
> >> > >> being
> >> > >> worked on in Hadoop trunk:
> >> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> >> > >> This document describes the current status and limitations of the
> >> > >> permissions mechanism:
> >> > >>
> >> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> >> > >>
> >> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <matei@cloudera.com
> >
> >> > >> wrote:
> >> > >>
> >> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS
> at
> >> > the
> >> > >> > level described in those papers. In particular, in HDFS, there is
> a
> >> > >> master
> >> > >> > node containing metadata and a number of slave nodes (datanodes)
> >> > >> containing
> >> > >> > blocks, as in GFS. Clients start by talking to the master to list
> >> > >> > directories, etc. When they want to read a region of some file,
> >> they
> >> > >> tell
> >> > >> > the master the filename and offset, and they receive a list of
> >> block
> >> > >> > locations (datanodes). They then contact the individual datanodes
> >> to
> >> > >> read
> >> > >> > the blocks. When clients write a file, they first obtain a new
> >> block
> >> > ID
> >> > >> and
> >> > >> > list of nodes to write it to from the master, then contact the
> >> > datanodes
> >> > >> to
> >> > >> > write it (actually, the datanodes pipeline the write as in GFS)
> and
> >> > >> report
> >> > >> > when the write is complete. HDFS actually has some security
> >> mechanisms
> >> > >> built
> >> > >> > in, authenticating users based on their Unix ID and providing
> >> > Unix-like
> >> > >> file
> >> > >> > permissions. I don't know much about how these are implemented,
> but
> >> > they
> >> > >> > would be a good place to start looking.
> >> > >> >
> >> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
> >> amansk@gmail.com
> >> > >> >wrote:
> >> > >> >
> >> > >> >> Thanks Matie
> >> > >> >>
> >> > >> >> I had gone through the architecture document online. I am
> >> currently
> >> > >> >> working
> >> > >> >> on a project towards Security in Hadoop. I do know how the data
> >> moves
> >> > >> >> around
> >> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and
> >> how
> >> > >> >> different it is from GFS. Can you throw some light on that?
> >> > >> >>
> >> > >> >> Security would also involve the Map Reduce jobs following the
> same
> >> > >> >> protocols. Thats why the question about how does the Hadoop
> >> framework
> >> > >> >> integrate with the HDFS, and how different is it from Map Reduce
> >> and
> >> > >> GFS.
> >> > >> >> The GFS and Map Reduce papers give a good information on how
> those
> >> > >> systems
> >> > >> >> are designed but there is nothing that concrete for Hadoop that
> I
> >> > have
> >> > >> >> been
> >> > >> >> able to find.
> >> > >> >>
> >> > >> >> Amandeep
> >> > >> >>
> >> > >> >>
> >> > >> >> Amandeep Khurana
> >> > >> >> Computer Science Graduate Student
> >> > >> >> University of California, Santa Cruz
> >> > >> >>
> >> > >> >>
> >> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> >> matei@cloudera.com>
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >> > Hi Amandeep,
> >> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> >> provide
> >> > >> those
> >> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
> >> > (large
> >> > >> >> > blocks, replication, etc); some notable things missing are
> >> > read-write
> >> > >> >> > support in the middle of a file (unlikely to be provided
> because
> >> > few
> >> > >> >> Hadoop
> >> > >> >> > applications require it) and multiple appenders (the record
> >> append
> >> > >> >> > operation). You can read about HDFS architecture at
> >> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html.
> >> The
> >> > >> >> MapReduce
> >> > >> >> > part of Hadoop interacts with HDFS in the same way that
> Google's
> >> > >> >> MapReduce
> >> > >> >> > interacts with GFS (shipping computation to the data),
> although
> >> > >> Hadoop
> >> > >> >> > MapReduce also supports running over other distributed
> >> filesystems.
> >> > >> >> >
> >> > >> >> > Matei
> >> > >> >> >
> >> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> >> > amansk@gmail.com
> >> > >> >
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > Hi
> >> > >> >> > >
> >> > >> >> > > Is the HDFS architecture completely based on the Google
> >> > Filesystem?
> >> > >> If
> >> > >> >> it
> >> > >> >> > > isnt, what are the differences between the two?
> >> > >> >> > >
> >> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as
> how
> >> it
> >> > is
> >> > >> >> > between
> >> > >> >> > > the Google's version of Map Reduce and GFS?
> >> > >> >> > >
> >> > >> >> > > Amandeep
> >> > >> >> > >
> >> > >> >> > >
> >> > >> >> > > Amandeep Khurana
> >> > >> >> > > Computer Science Graduate Student
> >> > >> >> > > University of California, Santa Cruz
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Another question that I have here - When the jobs run arbitrary code and
access data from the HDFS, do they go to the namenode to get the block
information?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Assuming that the job is purely in Java and not involving streaming or
> pipes, wouldnt the resources (files) required by the job as inputs be known
> beforehand? So, if the map task is accessing a second file, how does it make
> it different except that there are multiple files. The JobTracker would know
> beforehand that multiple files would be accessed. Right?
>
> I am slightly confused why you have mentioned this case separately... Can
> you elaborate on it a little bit?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
>> Typically the data flow is like this:1) Client submits a job description
>> to
>> the JobTracker.
>> 2) JobTracker figures out block locations for the input file(s) by talking
>> to HDFS NameNode.
>> 3) JobTracker creates a job description file in HDFS which will be read by
>> the nodes to copy over the job's code etc.
>> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
>> appropriate data blocks.
>> 5) After running, maps create intermediate output files on those slaves.
>> These are not in HDFS, they're in some temporary storage used by
>> MapReduce.
>> 6) JobTracker starts reduces on a series of slaves, which copy over the
>> appropriate map outputs, apply the reduce function, and write the outputs
>> to
>> HDFS (one output file per reducer).
>> 7) Some logs for the job may also be put into HDFS by the JobTracker.
>>
>> However, there is a big caveat, which is that the map and reduce tasks run
>> arbitrary code. It is not unusual to have a map that opens a second HDFS
>> file to read some information (e.g. for doing a join of a small table
>> against a big file). If you use Hadoop Streaming or Pipes to write a job
>> in
>> Python, Ruby, C, etc, then you are launching arbitrary processes which may
>> also access external resources in this manner. Some people also read/write
>> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
>> would ideally deal with these cases too.
>>
>> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
>> wrote:
>>
>> > A quick question here. How does a typical hadoop job work at the system
>> > level? What are the various interactions and how does the data flow?
>> >
>> > Amandeep
>> >
>> >
>> > Amandeep Khurana
>> > Computer Science Graduate Student
>> > University of California, Santa Cruz
>> >
>> >
>> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Matei. If the basic architecture is similar to the Google
>> stuff, I
>> > > can safely just work on the project using the information from the
>> > papers.
>> > >
>> > > I am aware of the 4487 jira and the current status of the permissions
>> > > mechanism. I had a look at them earlier.
>> > >
>> > > Cheers
>> > > Amandeep
>> > >
>> > >
>> > > Amandeep Khurana
>> > > Computer Science Graduate Student
>> > > University of California, Santa Cruz
>> > >
>> > >
>> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
>> > wrote:
>> > >
>> > >> Forgot to add, this JIRA details the latest security features that
>> are
>> > >> being
>> > >> worked on in Hadoop trunk:
>> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
>> > >> This document describes the current status and limitations of the
>> > >> permissions mechanism:
>> > >>
>> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
>> > >>
>> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
>> > >> wrote:
>> > >>
>> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
>> > the
>> > >> > level described in those papers. In particular, in HDFS, there is a
>> > >> master
>> > >> > node containing metadata and a number of slave nodes (datanodes)
>> > >> containing
>> > >> > blocks, as in GFS. Clients start by talking to the master to list
>> > >> > directories, etc. When they want to read a region of some file,
>> they
>> > >> tell
>> > >> > the master the filename and offset, and they receive a list of
>> block
>> > >> > locations (datanodes). They then contact the individual datanodes
>> to
>> > >> read
>> > >> > the blocks. When clients write a file, they first obtain a new
>> block
>> > ID
>> > >> and
>> > >> > list of nodes to write it to from the master, then contact the
>> > datanodes
>> > >> to
>> > >> > write it (actually, the datanodes pipeline the write as in GFS) and
>> > >> report
>> > >> > when the write is complete. HDFS actually has some security
>> mechanisms
>> > >> built
>> > >> > in, authenticating users based on their Unix ID and providing
>> > Unix-like
>> > >> file
>> > >> > permissions. I don't know much about how these are implemented, but
>> > they
>> > >> > would be a good place to start looking.
>> > >> >
>> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
>> amansk@gmail.com
>> > >> >wrote:
>> > >> >
>> > >> >> Thanks Matie
>> > >> >>
>> > >> >> I had gone through the architecture document online. I am
>> currently
>> > >> >> working
>> > >> >> on a project towards Security in Hadoop. I do know how the data
>> moves
>> > >> >> around
>> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and
>> how
>> > >> >> different it is from GFS. Can you throw some light on that?
>> > >> >>
>> > >> >> Security would also involve the Map Reduce jobs following the same
>> > >> >> protocols. Thats why the question about how does the Hadoop
>> framework
>> > >> >> integrate with the HDFS, and how different is it from Map Reduce
>> and
>> > >> GFS.
>> > >> >> The GFS and Map Reduce papers give a good information on how those
>> > >> systems
>> > >> >> are designed but there is nothing that concrete for Hadoop that I
>> > have
>> > >> >> been
>> > >> >> able to find.
>> > >> >>
>> > >> >> Amandeep
>> > >> >>
>> > >> >>
>> > >> >> Amandeep Khurana
>> > >> >> Computer Science Graduate Student
>> > >> >> University of California, Santa Cruz
>> > >> >>
>> > >> >>
>> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
>> matei@cloudera.com>
>> > >> >> wrote:
>> > >> >>
>> > >> >> > Hi Amandeep,
>> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
>> provide
>> > >> those
>> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
>> > (large
>> > >> >> > blocks, replication, etc); some notable things missing are
>> > read-write
>> > >> >> > support in the middle of a file (unlikely to be provided because
>> > few
>> > >> >> Hadoop
>> > >> >> > applications require it) and multiple appenders (the record
>> append
>> > >> >> > operation). You can read about HDFS architecture at
>> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html.
>> The
>> > >> >> MapReduce
>> > >> >> > part of Hadoop interacts with HDFS in the same way that Google's
>> > >> >> MapReduce
>> > >> >> > interacts with GFS (shipping computation to the data), although
>> > >> Hadoop
>> > >> >> > MapReduce also supports running over other distributed
>> filesystems.
>> > >> >> >
>> > >> >> > Matei
>> > >> >> >
>> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
>> > amansk@gmail.com
>> > >> >
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > Hi
>> > >> >> > >
>> > >> >> > > Is the HDFS architecture completely based on the Google
>> > Filesystem?
>> > >> If
>> > >> >> it
>> > >> >> > > isnt, what are the differences between the two?
>> > >> >> > >
>> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how
>> it
>> > is
>> > >> >> > between
>> > >> >> > > the Google's version of Map Reduce and GFS?
>> > >> >> > >
>> > >> >> > > Amandeep
>> > >> >> > >
>> > >> >> > >
>> > >> >> > > Amandeep Khurana
>> > >> >> > > Computer Science Graduate Student
>> > >> >> > > University of California, Santa Cruz
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Another question that I have here - When the jobs run arbitrary code and
access data from the HDFS, do they go to the namenode to get the block
information?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 6:00 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Assuming that the job is purely in Java and not involving streaming or
> pipes, wouldnt the resources (files) required by the job as inputs be known
> beforehand? So, if the map task is accessing a second file, how does it make
> it different except that there are multiple files. The JobTracker would know
> beforehand that multiple files would be accessed. Right?
>
> I am slightly confused why you have mentioned this case separately... Can
> you elaborate on it a little bit?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
>> Typically the data flow is like this:1) Client submits a job description
>> to
>> the JobTracker.
>> 2) JobTracker figures out block locations for the input file(s) by talking
>> to HDFS NameNode.
>> 3) JobTracker creates a job description file in HDFS which will be read by
>> the nodes to copy over the job's code etc.
>> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
>> appropriate data blocks.
>> 5) After running, maps create intermediate output files on those slaves.
>> These are not in HDFS, they're in some temporary storage used by
>> MapReduce.
>> 6) JobTracker starts reduces on a series of slaves, which copy over the
>> appropriate map outputs, apply the reduce function, and write the outputs
>> to
>> HDFS (one output file per reducer).
>> 7) Some logs for the job may also be put into HDFS by the JobTracker.
>>
>> However, there is a big caveat, which is that the map and reduce tasks run
>> arbitrary code. It is not unusual to have a map that opens a second HDFS
>> file to read some information (e.g. for doing a join of a small table
>> against a big file). If you use Hadoop Streaming or Pipes to write a job
>> in
>> Python, Ruby, C, etc, then you are launching arbitrary processes which may
>> also access external resources in this manner. Some people also read/write
>> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
>> would ideally deal with these cases too.
>>
>> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
>> wrote:
>>
>> > A quick question here. How does a typical hadoop job work at the system
>> > level? What are the various interactions and how does the data flow?
>> >
>> > Amandeep
>> >
>> >
>> > Amandeep Khurana
>> > Computer Science Graduate Student
>> > University of California, Santa Cruz
>> >
>> >
>> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Matei. If the basic architecture is similar to the Google
>> stuff, I
>> > > can safely just work on the project using the information from the
>> > papers.
>> > >
>> > > I am aware of the 4487 jira and the current status of the permissions
>> > > mechanism. I had a look at them earlier.
>> > >
>> > > Cheers
>> > > Amandeep
>> > >
>> > >
>> > > Amandeep Khurana
>> > > Computer Science Graduate Student
>> > > University of California, Santa Cruz
>> > >
>> > >
>> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
>> > wrote:
>> > >
>> > >> Forgot to add, this JIRA details the latest security features that
>> are
>> > >> being
>> > >> worked on in Hadoop trunk:
>> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
>> > >> This document describes the current status and limitations of the
>> > >> permissions mechanism:
>> > >>
>> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
>> > >>
>> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
>> > >> wrote:
>> > >>
>> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
>> > the
>> > >> > level described in those papers. In particular, in HDFS, there is a
>> > >> master
>> > >> > node containing metadata and a number of slave nodes (datanodes)
>> > >> containing
>> > >> > blocks, as in GFS. Clients start by talking to the master to list
>> > >> > directories, etc. When they want to read a region of some file,
>> they
>> > >> tell
>> > >> > the master the filename and offset, and they receive a list of
>> block
>> > >> > locations (datanodes). They then contact the individual datanodes
>> to
>> > >> read
>> > >> > the blocks. When clients write a file, they first obtain a new
>> block
>> > ID
>> > >> and
>> > >> > list of nodes to write it to from the master, then contact the
>> > datanodes
>> > >> to
>> > >> > write it (actually, the datanodes pipeline the write as in GFS) and
>> > >> report
>> > >> > when the write is complete. HDFS actually has some security
>> mechanisms
>> > >> built
>> > >> > in, authenticating users based on their Unix ID and providing
>> > Unix-like
>> > >> file
>> > >> > permissions. I don't know much about how these are implemented, but
>> > they
>> > >> > would be a good place to start looking.
>> > >> >
>> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <
>> amansk@gmail.com
>> > >> >wrote:
>> > >> >
>> > >> >> Thanks Matie
>> > >> >>
>> > >> >> I had gone through the architecture document online. I am
>> currently
>> > >> >> working
>> > >> >> on a project towards Security in Hadoop. I do know how the data
>> moves
>> > >> >> around
>> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and
>> how
>> > >> >> different it is from GFS. Can you throw some light on that?
>> > >> >>
>> > >> >> Security would also involve the Map Reduce jobs following the same
>> > >> >> protocols. Thats why the question about how does the Hadoop
>> framework
>> > >> >> integrate with the HDFS, and how different is it from Map Reduce
>> and
>> > >> GFS.
>> > >> >> The GFS and Map Reduce papers give a good information on how those
>> > >> systems
>> > >> >> are designed but there is nothing that concrete for Hadoop that I
>> > have
>> > >> >> been
>> > >> >> able to find.
>> > >> >>
>> > >> >> Amandeep
>> > >> >>
>> > >> >>
>> > >> >> Amandeep Khurana
>> > >> >> Computer Science Graduate Student
>> > >> >> University of California, Santa Cruz
>> > >> >>
>> > >> >>
>> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
>> matei@cloudera.com>
>> > >> >> wrote:
>> > >> >>
>> > >> >> > Hi Amandeep,
>> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
>> provide
>> > >> those
>> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
>> > (large
>> > >> >> > blocks, replication, etc); some notable things missing are
>> > read-write
>> > >> >> > support in the middle of a file (unlikely to be provided because
>> > few
>> > >> >> Hadoop
>> > >> >> > applications require it) and multiple appenders (the record
>> append
>> > >> >> > operation). You can read about HDFS architecture at
>> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html.
>> The
>> > >> >> MapReduce
>> > >> >> > part of Hadoop interacts with HDFS in the same way that Google's
>> > >> >> MapReduce
>> > >> >> > interacts with GFS (shipping computation to the data), although
>> > >> Hadoop
>> > >> >> > MapReduce also supports running over other distributed
>> filesystems.
>> > >> >> >
>> > >> >> > Matei
>> > >> >> >
>> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
>> > amansk@gmail.com
>> > >> >
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > Hi
>> > >> >> > >
>> > >> >> > > Is the HDFS architecture completely based on the Google
>> > Filesystem?
>> > >> If
>> > >> >> it
>> > >> >> > > isnt, what are the differences between the two?
>> > >> >> > >
>> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how
>> it
>> > is
>> > >> >> > between
>> > >> >> > > the Google's version of Map Reduce and GFS?
>> > >> >> > >
>> > >> >> > > Amandeep
>> > >> >> > >
>> > >> >> > >
>> > >> >> > > Amandeep Khurana
>> > >> >> > > Computer Science Graduate Student
>> > >> >> > > University of California, Santa Cruz
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Assuming that the job is purely in Java and not involving streaming or
pipes, wouldnt the resources (files) required by the job as inputs be known
beforehand? So, if the map task is accessing a second file, how does it make
it different except that there are multiple files. The JobTracker would know
beforehand that multiple files would be accessed. Right?
I am slightly confused why you have mentioned this case separately... Can
you elaborate on it a little bit?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Typically the data flow is like this:1) Client submits a job description to
> the JobTracker.
> 2) JobTracker figures out block locations for the input file(s) by talking
> to HDFS NameNode.
> 3) JobTracker creates a job description file in HDFS which will be read by
> the nodes to copy over the job's code etc.
> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> appropriate data blocks.
> 5) After running, maps create intermediate output files on those slaves.
> These are not in HDFS, they're in some temporary storage used by MapReduce.
> 6) JobTracker starts reduces on a series of slaves, which copy over the
> appropriate map outputs, apply the reduce function, and write the outputs
> to
> HDFS (one output file per reducer).
> 7) Some logs for the job may also be put into HDFS by the JobTracker.
>
> However, there is a big caveat, which is that the map and reduce tasks run
> arbitrary code. It is not unusual to have a map that opens a second HDFS
> file to read some information (e.g. for doing a join of a small table
> against a big file). If you use Hadoop Streaming or Pipes to write a job in
> Python, Ruby, C, etc, then you are launching arbitrary processes which may
> also access external resources in this manner. Some people also read/write
> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
> would ideally deal with these cases too.
>
> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > A quick question here. How does a typical hadoop job work at the system
> > level? What are the various interactions and how does the data flow?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Thanks Matei. If the basic architecture is similar to the Google stuff,
> I
> > > can safely just work on the project using the information from the
> > papers.
> > >
> > > I am aware of the 4487 jira and the current status of the permissions
> > > mechanism. I had a look at them earlier.
> > >
> > > Cheers
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > >> Forgot to add, this JIRA details the latest security features that are
> > >> being
> > >> worked on in Hadoop trunk:
> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > >> This document describes the current status and limitations of the
> > >> permissions mechanism:
> > >>
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > >>
> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
> > >> wrote:
> > >>
> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
> > the
> > >> > level described in those papers. In particular, in HDFS, there is a
> > >> master
> > >> > node containing metadata and a number of slave nodes (datanodes)
> > >> containing
> > >> > blocks, as in GFS. Clients start by talking to the master to list
> > >> > directories, etc. When they want to read a region of some file, they
> > >> tell
> > >> > the master the filename and offset, and they receive a list of block
> > >> > locations (datanodes). They then contact the individual datanodes to
> > >> read
> > >> > the blocks. When clients write a file, they first obtain a new block
> > ID
> > >> and
> > >> > list of nodes to write it to from the master, then contact the
> > datanodes
> > >> to
> > >> > write it (actually, the datanodes pipeline the write as in GFS) and
> > >> report
> > >> > when the write is complete. HDFS actually has some security
> mechanisms
> > >> built
> > >> > in, authenticating users based on their Unix ID and providing
> > Unix-like
> > >> file
> > >> > permissions. I don't know much about how these are implemented, but
> > they
> > >> > would be a good place to start looking.
> > >> >
> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Thanks Matie
> > >> >>
> > >> >> I had gone through the architecture document online. I am currently
> > >> >> working
> > >> >> on a project towards Security in Hadoop. I do know how the data
> moves
> > >> >> around
> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> > >> >> different it is from GFS. Can you throw some light on that?
> > >> >>
> > >> >> Security would also involve the Map Reduce jobs following the same
> > >> >> protocols. Thats why the question about how does the Hadoop
> framework
> > >> >> integrate with the HDFS, and how different is it from Map Reduce
> and
> > >> GFS.
> > >> >> The GFS and Map Reduce papers give a good information on how those
> > >> systems
> > >> >> are designed but there is nothing that concrete for Hadoop that I
> > have
> > >> >> been
> > >> >> able to find.
> > >> >>
> > >> >> Amandeep
> > >> >>
> > >> >>
> > >> >> Amandeep Khurana
> > >> >> Computer Science Graduate Student
> > >> >> University of California, Santa Cruz
> > >> >>
> > >> >>
> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> matei@cloudera.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Amandeep,
> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> provide
> > >> those
> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
> > (large
> > >> >> > blocks, replication, etc); some notable things missing are
> > read-write
> > >> >> > support in the middle of a file (unlikely to be provided because
> > few
> > >> >> Hadoop
> > >> >> > applications require it) and multiple appenders (the record
> append
> > >> >> > operation). You can read about HDFS architecture at
> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> > >> >> MapReduce
> > >> >> > part of Hadoop interacts with HDFS in the same way that Google's
> > >> >> MapReduce
> > >> >> > interacts with GFS (shipping computation to the data), although
> > >> Hadoop
> > >> >> > MapReduce also supports running over other distributed
> filesystems.
> > >> >> >
> > >> >> > Matei
> > >> >> >
> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > amansk@gmail.com
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Hi
> > >> >> > >
> > >> >> > > Is the HDFS architecture completely based on the Google
> > Filesystem?
> > >> If
> > >> >> it
> > >> >> > > isnt, what are the differences between the two?
> > >> >> > >
> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how
> it
> > is
> > >> >> > between
> > >> >> > > the Google's version of Map Reduce and GFS?
> > >> >> > >
> > >> >> > > Amandeep
> > >> >> > >
> > >> >> > >
> > >> >> > > Amandeep Khurana
> > >> >> > > Computer Science Graduate Student
> > >> >> > > University of California, Santa Cruz
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
This is good information! Thanks a ton. I'll take all this into account.
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Typically the data flow is like this:1) Client submits a job description to
> the JobTracker.
> 2) JobTracker figures out block locations for the input file(s) by talking
> to HDFS NameNode.
> 3) JobTracker creates a job description file in HDFS which will be read by
> the nodes to copy over the job's code etc.
> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> appropriate data blocks.
> 5) After running, maps create intermediate output files on those slaves.
> These are not in HDFS, they're in some temporary storage used by MapReduce.
> 6) JobTracker starts reduces on a series of slaves, which copy over the
> appropriate map outputs, apply the reduce function, and write the outputs
> to
> HDFS (one output file per reducer).
> 7) Some logs for the job may also be put into HDFS by the JobTracker.
>
> However, there is a big caveat, which is that the map and reduce tasks run
> arbitrary code. It is not unusual to have a map that opens a second HDFS
> file to read some information (e.g. for doing a join of a small table
> against a big file). If you use Hadoop Streaming or Pipes to write a job in
> Python, Ruby, C, etc, then you are launching arbitrary processes which may
> also access external resources in this manner. Some people also read/write
> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
> would ideally deal with these cases too.
>
> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > A quick question here. How does a typical hadoop job work at the system
> > level? What are the various interactions and how does the data flow?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Thanks Matei. If the basic architecture is similar to the Google stuff,
> I
> > > can safely just work on the project using the information from the
> > papers.
> > >
> > > I am aware of the 4487 jira and the current status of the permissions
> > > mechanism. I had a look at them earlier.
> > >
> > > Cheers
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > >> Forgot to add, this JIRA details the latest security features that are
> > >> being
> > >> worked on in Hadoop trunk:
> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > >> This document describes the current status and limitations of the
> > >> permissions mechanism:
> > >>
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > >>
> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
> > >> wrote:
> > >>
> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
> > the
> > >> > level described in those papers. In particular, in HDFS, there is a
> > >> master
> > >> > node containing metadata and a number of slave nodes (datanodes)
> > >> containing
> > >> > blocks, as in GFS. Clients start by talking to the master to list
> > >> > directories, etc. When they want to read a region of some file, they
> > >> tell
> > >> > the master the filename and offset, and they receive a list of block
> > >> > locations (datanodes). They then contact the individual datanodes to
> > >> read
> > >> > the blocks. When clients write a file, they first obtain a new block
> > ID
> > >> and
> > >> > list of nodes to write it to from the master, then contact the
> > datanodes
> > >> to
> > >> > write it (actually, the datanodes pipeline the write as in GFS) and
> > >> report
> > >> > when the write is complete. HDFS actually has some security
> mechanisms
> > >> built
> > >> > in, authenticating users based on their Unix ID and providing
> > Unix-like
> > >> file
> > >> > permissions. I don't know much about how these are implemented, but
> > they
> > >> > would be a good place to start looking.
> > >> >
> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Thanks Matie
> > >> >>
> > >> >> I had gone through the architecture document online. I am currently
> > >> >> working
> > >> >> on a project towards Security in Hadoop. I do know how the data
> moves
> > >> >> around
> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> > >> >> different it is from GFS. Can you throw some light on that?
> > >> >>
> > >> >> Security would also involve the Map Reduce jobs following the same
> > >> >> protocols. Thats why the question about how does the Hadoop
> framework
> > >> >> integrate with the HDFS, and how different is it from Map Reduce
> and
> > >> GFS.
> > >> >> The GFS and Map Reduce papers give a good information on how those
> > >> systems
> > >> >> are designed but there is nothing that concrete for Hadoop that I
> > have
> > >> >> been
> > >> >> able to find.
> > >> >>
> > >> >> Amandeep
> > >> >>
> > >> >>
> > >> >> Amandeep Khurana
> > >> >> Computer Science Graduate Student
> > >> >> University of California, Santa Cruz
> > >> >>
> > >> >>
> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> matei@cloudera.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Amandeep,
> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> provide
> > >> those
> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
> > (large
> > >> >> > blocks, replication, etc); some notable things missing are
> > read-write
> > >> >> > support in the middle of a file (unlikely to be provided because
> > few
> > >> >> Hadoop
> > >> >> > applications require it) and multiple appenders (the record
> append
> > >> >> > operation). You can read about HDFS architecture at
> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> > >> >> MapReduce
> > >> >> > part of Hadoop interacts with HDFS in the same way that Google's
> > >> >> MapReduce
> > >> >> > interacts with GFS (shipping computation to the data), although
> > >> Hadoop
> > >> >> > MapReduce also supports running over other distributed
> filesystems.
> > >> >> >
> > >> >> > Matei
> > >> >> >
> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > amansk@gmail.com
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Hi
> > >> >> > >
> > >> >> > > Is the HDFS architecture completely based on the Google
> > Filesystem?
> > >> If
> > >> >> it
> > >> >> > > isnt, what are the differences between the two?
> > >> >> > >
> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how
> it
> > is
> > >> >> > between
> > >> >> > > the Google's version of Map Reduce and GFS?
> > >> >> > >
> > >> >> > > Amandeep
> > >> >> > >
> > >> >> > >
> > >> >> > > Amandeep Khurana
> > >> >> > > Computer Science Graduate Student
> > >> >> > > University of California, Santa Cruz
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Assuming that the job is purely in Java and not involving streaming or
pipes, wouldnt the resources (files) required by the job as inputs be known
beforehand? So, if the map task is accessing a second file, how does it make
it different except that there are multiple files. The JobTracker would know
beforehand that multiple files would be accessed. Right?
I am slightly confused why you have mentioned this case separately... Can
you elaborate on it a little bit?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Typically the data flow is like this:1) Client submits a job description to
> the JobTracker.
> 2) JobTracker figures out block locations for the input file(s) by talking
> to HDFS NameNode.
> 3) JobTracker creates a job description file in HDFS which will be read by
> the nodes to copy over the job's code etc.
> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> appropriate data blocks.
> 5) After running, maps create intermediate output files on those slaves.
> These are not in HDFS, they're in some temporary storage used by MapReduce.
> 6) JobTracker starts reduces on a series of slaves, which copy over the
> appropriate map outputs, apply the reduce function, and write the outputs
> to
> HDFS (one output file per reducer).
> 7) Some logs for the job may also be put into HDFS by the JobTracker.
>
> However, there is a big caveat, which is that the map and reduce tasks run
> arbitrary code. It is not unusual to have a map that opens a second HDFS
> file to read some information (e.g. for doing a join of a small table
> against a big file). If you use Hadoop Streaming or Pipes to write a job in
> Python, Ruby, C, etc, then you are launching arbitrary processes which may
> also access external resources in this manner. Some people also read/write
> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
> would ideally deal with these cases too.
>
> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > A quick question here. How does a typical hadoop job work at the system
> > level? What are the various interactions and how does the data flow?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Thanks Matei. If the basic architecture is similar to the Google stuff,
> I
> > > can safely just work on the project using the information from the
> > papers.
> > >
> > > I am aware of the 4487 jira and the current status of the permissions
> > > mechanism. I had a look at them earlier.
> > >
> > > Cheers
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > >> Forgot to add, this JIRA details the latest security features that are
> > >> being
> > >> worked on in Hadoop trunk:
> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > >> This document describes the current status and limitations of the
> > >> permissions mechanism:
> > >>
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > >>
> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
> > >> wrote:
> > >>
> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
> > the
> > >> > level described in those papers. In particular, in HDFS, there is a
> > >> master
> > >> > node containing metadata and a number of slave nodes (datanodes)
> > >> containing
> > >> > blocks, as in GFS. Clients start by talking to the master to list
> > >> > directories, etc. When they want to read a region of some file, they
> > >> tell
> > >> > the master the filename and offset, and they receive a list of block
> > >> > locations (datanodes). They then contact the individual datanodes to
> > >> read
> > >> > the blocks. When clients write a file, they first obtain a new block
> > ID
> > >> and
> > >> > list of nodes to write it to from the master, then contact the
> > datanodes
> > >> to
> > >> > write it (actually, the datanodes pipeline the write as in GFS) and
> > >> report
> > >> > when the write is complete. HDFS actually has some security
> mechanisms
> > >> built
> > >> > in, authenticating users based on their Unix ID and providing
> > Unix-like
> > >> file
> > >> > permissions. I don't know much about how these are implemented, but
> > they
> > >> > would be a good place to start looking.
> > >> >
> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Thanks Matie
> > >> >>
> > >> >> I had gone through the architecture document online. I am currently
> > >> >> working
> > >> >> on a project towards Security in Hadoop. I do know how the data
> moves
> > >> >> around
> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> > >> >> different it is from GFS. Can you throw some light on that?
> > >> >>
> > >> >> Security would also involve the Map Reduce jobs following the same
> > >> >> protocols. Thats why the question about how does the Hadoop
> framework
> > >> >> integrate with the HDFS, and how different is it from Map Reduce
> and
> > >> GFS.
> > >> >> The GFS and Map Reduce papers give a good information on how those
> > >> systems
> > >> >> are designed but there is nothing that concrete for Hadoop that I
> > have
> > >> >> been
> > >> >> able to find.
> > >> >>
> > >> >> Amandeep
> > >> >>
> > >> >>
> > >> >> Amandeep Khurana
> > >> >> Computer Science Graduate Student
> > >> >> University of California, Santa Cruz
> > >> >>
> > >> >>
> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> matei@cloudera.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Amandeep,
> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> provide
> > >> those
> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
> > (large
> > >> >> > blocks, replication, etc); some notable things missing are
> > read-write
> > >> >> > support in the middle of a file (unlikely to be provided because
> > few
> > >> >> Hadoop
> > >> >> > applications require it) and multiple appenders (the record
> append
> > >> >> > operation). You can read about HDFS architecture at
> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> > >> >> MapReduce
> > >> >> > part of Hadoop interacts with HDFS in the same way that Google's
> > >> >> MapReduce
> > >> >> > interacts with GFS (shipping computation to the data), although
> > >> Hadoop
> > >> >> > MapReduce also supports running over other distributed
> filesystems.
> > >> >> >
> > >> >> > Matei
> > >> >> >
> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > amansk@gmail.com
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Hi
> > >> >> > >
> > >> >> > > Is the HDFS architecture completely based on the Google
> > Filesystem?
> > >> If
> > >> >> it
> > >> >> > > isnt, what are the differences between the two?
> > >> >> > >
> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how
> it
> > is
> > >> >> > between
> > >> >> > > the Google's version of Map Reduce and GFS?
> > >> >> > >
> > >> >> > > Amandeep
> > >> >> > >
> > >> >> > >
> > >> >> > > Amandeep Khurana
> > >> >> > > Computer Science Graduate Student
> > >> >> > > University of California, Santa Cruz
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
This is good information! Thanks a ton. I'll take all this into account.
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 4:47 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Typically the data flow is like this:1) Client submits a job description to
> the JobTracker.
> 2) JobTracker figures out block locations for the input file(s) by talking
> to HDFS NameNode.
> 3) JobTracker creates a job description file in HDFS which will be read by
> the nodes to copy over the job's code etc.
> 4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
> appropriate data blocks.
> 5) After running, maps create intermediate output files on those slaves.
> These are not in HDFS, they're in some temporary storage used by MapReduce.
> 6) JobTracker starts reduces on a series of slaves, which copy over the
> appropriate map outputs, apply the reduce function, and write the outputs
> to
> HDFS (one output file per reducer).
> 7) Some logs for the job may also be put into HDFS by the JobTracker.
>
> However, there is a big caveat, which is that the map and reduce tasks run
> arbitrary code. It is not unusual to have a map that opens a second HDFS
> file to read some information (e.g. for doing a join of a small table
> against a big file). If you use Hadoop Streaming or Pipes to write a job in
> Python, Ruby, C, etc, then you are launching arbitrary processes which may
> also access external resources in this manner. Some people also read/write
> to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
> would ideally deal with these cases too.
>
> On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > A quick question here. How does a typical hadoop job work at the system
> > level? What are the various interactions and how does the data flow?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Thanks Matei. If the basic architecture is similar to the Google stuff,
> I
> > > can safely just work on the project using the information from the
> > papers.
> > >
> > > I am aware of the 4487 jira and the current status of the permissions
> > > mechanism. I had a look at them earlier.
> > >
> > > Cheers
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> > >
> > > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> > wrote:
> > >
> > >> Forgot to add, this JIRA details the latest security features that are
> > >> being
> > >> worked on in Hadoop trunk:
> > >> https://issues.apache.org/jira/browse/HADOOP-4487.
> > >> This document describes the current status and limitations of the
> > >> permissions mechanism:
> > >>
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> > >>
> > >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
> > >> wrote:
> > >>
> > >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
> > the
> > >> > level described in those papers. In particular, in HDFS, there is a
> > >> master
> > >> > node containing metadata and a number of slave nodes (datanodes)
> > >> containing
> > >> > blocks, as in GFS. Clients start by talking to the master to list
> > >> > directories, etc. When they want to read a region of some file, they
> > >> tell
> > >> > the master the filename and offset, and they receive a list of block
> > >> > locations (datanodes). They then contact the individual datanodes to
> > >> read
> > >> > the blocks. When clients write a file, they first obtain a new block
> > ID
> > >> and
> > >> > list of nodes to write it to from the master, then contact the
> > datanodes
> > >> to
> > >> > write it (actually, the datanodes pipeline the write as in GFS) and
> > >> report
> > >> > when the write is complete. HDFS actually has some security
> mechanisms
> > >> built
> > >> > in, authenticating users based on their Unix ID and providing
> > Unix-like
> > >> file
> > >> > permissions. I don't know much about how these are implemented, but
> > they
> > >> > would be a good place to start looking.
> > >> >
> > >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> > >> >wrote:
> > >> >
> > >> >> Thanks Matie
> > >> >>
> > >> >> I had gone through the architecture document online. I am currently
> > >> >> working
> > >> >> on a project towards Security in Hadoop. I do know how the data
> moves
> > >> >> around
> > >> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> > >> >> different it is from GFS. Can you throw some light on that?
> > >> >>
> > >> >> Security would also involve the Map Reduce jobs following the same
> > >> >> protocols. Thats why the question about how does the Hadoop
> framework
> > >> >> integrate with the HDFS, and how different is it from Map Reduce
> and
> > >> GFS.
> > >> >> The GFS and Map Reduce papers give a good information on how those
> > >> systems
> > >> >> are designed but there is nothing that concrete for Hadoop that I
> > have
> > >> >> been
> > >> >> able to find.
> > >> >>
> > >> >> Amandeep
> > >> >>
> > >> >>
> > >> >> Amandeep Khurana
> > >> >> Computer Science Graduate Student
> > >> >> University of California, Santa Cruz
> > >> >>
> > >> >>
> > >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <
> matei@cloudera.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Amandeep,
> > >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to
> provide
> > >> those
> > >> >> > capabilities as an open-source project. HDFS is similar to GFS
> > (large
> > >> >> > blocks, replication, etc); some notable things missing are
> > read-write
> > >> >> > support in the middle of a file (unlikely to be provided because
> > few
> > >> >> Hadoop
> > >> >> > applications require it) and multiple appenders (the record
> append
> > >> >> > operation). You can read about HDFS architecture at
> > >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> > >> >> MapReduce
> > >> >> > part of Hadoop interacts with HDFS in the same way that Google's
> > >> >> MapReduce
> > >> >> > interacts with GFS (shipping computation to the data), although
> > >> Hadoop
> > >> >> > MapReduce also supports running over other distributed
> filesystems.
> > >> >> >
> > >> >> > Matei
> > >> >> >
> > >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> > amansk@gmail.com
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Hi
> > >> >> > >
> > >> >> > > Is the HDFS architecture completely based on the Google
> > Filesystem?
> > >> If
> > >> >> it
> > >> >> > > isnt, what are the differences between the two?
> > >> >> > >
> > >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how
> it
> > is
> > >> >> > between
> > >> >> > > the Google's version of Map Reduce and GFS?
> > >> >> > >
> > >> >> > > Amandeep
> > >> >> > >
> > >> >> > >
> > >> >> > > Amandeep Khurana
> > >> >> > > Computer Science Graduate Student
> > >> >> > > University of California, Santa Cruz
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Typically the data flow is like this:1) Client submits a job description to
the JobTracker.
2) JobTracker figures out block locations for the input file(s) by talking
to HDFS NameNode.
3) JobTracker creates a job description file in HDFS which will be read by
the nodes to copy over the job's code etc.
4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
appropriate data blocks.
5) After running, maps create intermediate output files on those slaves.
These are not in HDFS, they're in some temporary storage used by MapReduce.
6) JobTracker starts reduces on a series of slaves, which copy over the
appropriate map outputs, apply the reduce function, and write the outputs to
HDFS (one output file per reducer).
7) Some logs for the job may also be put into HDFS by the JobTracker.
However, there is a big caveat, which is that the map and reduce tasks run
arbitrary code. It is not unusual to have a map that opens a second HDFS
file to read some information (e.g. for doing a join of a small table
against a big file). If you use Hadoop Streaming or Pipes to write a job in
Python, Ruby, C, etc, then you are launching arbitrary processes which may
also access external resources in this manner. Some people also read/write
to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
would ideally deal with these cases too.
On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com> wrote:
> A quick question here. How does a typical hadoop job work at the system
> level? What are the various interactions and how does the data flow?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Thanks Matei. If the basic architecture is similar to the Google stuff, I
> > can safely just work on the project using the information from the
> papers.
> >
> > I am aware of the 4487 jira and the current status of the permissions
> > mechanism. I had a look at them earlier.
> >
> > Cheers
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> >> Forgot to add, this JIRA details the latest security features that are
> >> being
> >> worked on in Hadoop trunk:
> >> https://issues.apache.org/jira/browse/HADOOP-4487.
> >> This document describes the current status and limitations of the
> >> permissions mechanism:
> >> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> >>
> >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
> >> wrote:
> >>
> >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
> the
> >> > level described in those papers. In particular, in HDFS, there is a
> >> master
> >> > node containing metadata and a number of slave nodes (datanodes)
> >> containing
> >> > blocks, as in GFS. Clients start by talking to the master to list
> >> > directories, etc. When they want to read a region of some file, they
> >> tell
> >> > the master the filename and offset, and they receive a list of block
> >> > locations (datanodes). They then contact the individual datanodes to
> >> read
> >> > the blocks. When clients write a file, they first obtain a new block
> ID
> >> and
> >> > list of nodes to write it to from the master, then contact the
> datanodes
> >> to
> >> > write it (actually, the datanodes pipeline the write as in GFS) and
> >> report
> >> > when the write is complete. HDFS actually has some security mechanisms
> >> built
> >> > in, authenticating users based on their Unix ID and providing
> Unix-like
> >> file
> >> > permissions. I don't know much about how these are implemented, but
> they
> >> > would be a good place to start looking.
> >> >
> >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> >> >wrote:
> >> >
> >> >> Thanks Matie
> >> >>
> >> >> I had gone through the architecture document online. I am currently
> >> >> working
> >> >> on a project towards Security in Hadoop. I do know how the data moves
> >> >> around
> >> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> >> >> different it is from GFS. Can you throw some light on that?
> >> >>
> >> >> Security would also involve the Map Reduce jobs following the same
> >> >> protocols. Thats why the question about how does the Hadoop framework
> >> >> integrate with the HDFS, and how different is it from Map Reduce and
> >> GFS.
> >> >> The GFS and Map Reduce papers give a good information on how those
> >> systems
> >> >> are designed but there is nothing that concrete for Hadoop that I
> have
> >> >> been
> >> >> able to find.
> >> >>
> >> >> Amandeep
> >> >>
> >> >>
> >> >> Amandeep Khurana
> >> >> Computer Science Graduate Student
> >> >> University of California, Santa Cruz
> >> >>
> >> >>
> >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Amandeep,
> >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide
> >> those
> >> >> > capabilities as an open-source project. HDFS is similar to GFS
> (large
> >> >> > blocks, replication, etc); some notable things missing are
> read-write
> >> >> > support in the middle of a file (unlikely to be provided because
> few
> >> >> Hadoop
> >> >> > applications require it) and multiple appenders (the record append
> >> >> > operation). You can read about HDFS architecture at
> >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> >> >> MapReduce
> >> >> > part of Hadoop interacts with HDFS in the same way that Google's
> >> >> MapReduce
> >> >> > interacts with GFS (shipping computation to the data), although
> >> Hadoop
> >> >> > MapReduce also supports running over other distributed filesystems.
> >> >> >
> >> >> > Matei
> >> >> >
> >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> amansk@gmail.com
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi
> >> >> > >
> >> >> > > Is the HDFS architecture completely based on the Google
> Filesystem?
> >> If
> >> >> it
> >> >> > > isnt, what are the differences between the two?
> >> >> > >
> >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how it
> is
> >> >> > between
> >> >> > > the Google's version of Map Reduce and GFS?
> >> >> > >
> >> >> > > Amandeep
> >> >> > >
> >> >> > >
> >> >> > > Amandeep Khurana
> >> >> > > Computer Science Graduate Student
> >> >> > > University of California, Santa Cruz
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Typically the data flow is like this:1) Client submits a job description to
the JobTracker.
2) JobTracker figures out block locations for the input file(s) by talking
to HDFS NameNode.
3) JobTracker creates a job description file in HDFS which will be read by
the nodes to copy over the job's code etc.
4) JobTracker starts map tasks on the slaves (TaskTrackers) with the
appropriate data blocks.
5) After running, maps create intermediate output files on those slaves.
These are not in HDFS, they're in some temporary storage used by MapReduce.
6) JobTracker starts reduces on a series of slaves, which copy over the
appropriate map outputs, apply the reduce function, and write the outputs to
HDFS (one output file per reducer).
7) Some logs for the job may also be put into HDFS by the JobTracker.
However, there is a big caveat, which is that the map and reduce tasks run
arbitrary code. It is not unusual to have a map that opens a second HDFS
file to read some information (e.g. for doing a join of a small table
against a big file). If you use Hadoop Streaming or Pipes to write a job in
Python, Ruby, C, etc, then you are launching arbitrary processes which may
also access external resources in this manner. Some people also read/write
to DBs (e.g. MySQL) from their tasks. A comprehensive security solution
would ideally deal with these cases too.
On Sun, Feb 15, 2009 at 3:22 PM, Amandeep Khurana <am...@gmail.com> wrote:
> A quick question here. How does a typical hadoop job work at the system
> level? What are the various interactions and how does the data flow?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Thanks Matei. If the basic architecture is similar to the Google stuff, I
> > can safely just work on the project using the information from the
> papers.
> >
> > I am aware of the 4487 jira and the current status of the permissions
> > mechanism. I had a look at them earlier.
> >
> > Cheers
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
> >
> >> Forgot to add, this JIRA details the latest security features that are
> >> being
> >> worked on in Hadoop trunk:
> >> https://issues.apache.org/jira/browse/HADOOP-4487.
> >> This document describes the current status and limitations of the
> >> permissions mechanism:
> >> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
> >>
> >> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
> >> wrote:
> >>
> >> > I think it's safe to assume that Hadoop works like MapReduce/GFS at
> the
> >> > level described in those papers. In particular, in HDFS, there is a
> >> master
> >> > node containing metadata and a number of slave nodes (datanodes)
> >> containing
> >> > blocks, as in GFS. Clients start by talking to the master to list
> >> > directories, etc. When they want to read a region of some file, they
> >> tell
> >> > the master the filename and offset, and they receive a list of block
> >> > locations (datanodes). They then contact the individual datanodes to
> >> read
> >> > the blocks. When clients write a file, they first obtain a new block
> ID
> >> and
> >> > list of nodes to write it to from the master, then contact the
> datanodes
> >> to
> >> > write it (actually, the datanodes pipeline the write as in GFS) and
> >> report
> >> > when the write is complete. HDFS actually has some security mechanisms
> >> built
> >> > in, authenticating users based on their Unix ID and providing
> Unix-like
> >> file
> >> > permissions. I don't know much about how these are implemented, but
> they
> >> > would be a good place to start looking.
> >> >
> >> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> >> >wrote:
> >> >
> >> >> Thanks Matie
> >> >>
> >> >> I had gone through the architecture document online. I am currently
> >> >> working
> >> >> on a project towards Security in Hadoop. I do know how the data moves
> >> >> around
> >> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> >> >> different it is from GFS. Can you throw some light on that?
> >> >>
> >> >> Security would also involve the Map Reduce jobs following the same
> >> >> protocols. Thats why the question about how does the Hadoop framework
> >> >> integrate with the HDFS, and how different is it from Map Reduce and
> >> GFS.
> >> >> The GFS and Map Reduce papers give a good information on how those
> >> systems
> >> >> are designed but there is nothing that concrete for Hadoop that I
> have
> >> >> been
> >> >> able to find.
> >> >>
> >> >> Amandeep
> >> >>
> >> >>
> >> >> Amandeep Khurana
> >> >> Computer Science Graduate Student
> >> >> University of California, Santa Cruz
> >> >>
> >> >>
> >> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Amandeep,
> >> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide
> >> those
> >> >> > capabilities as an open-source project. HDFS is similar to GFS
> (large
> >> >> > blocks, replication, etc); some notable things missing are
> read-write
> >> >> > support in the middle of a file (unlikely to be provided because
> few
> >> >> Hadoop
> >> >> > applications require it) and multiple appenders (the record append
> >> >> > operation). You can read about HDFS architecture at
> >> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> >> >> MapReduce
> >> >> > part of Hadoop interacts with HDFS in the same way that Google's
> >> >> MapReduce
> >> >> > interacts with GFS (shipping computation to the data), although
> >> Hadoop
> >> >> > MapReduce also supports running over other distributed filesystems.
> >> >> >
> >> >> > Matei
> >> >> >
> >> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <
> amansk@gmail.com
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi
> >> >> > >
> >> >> > > Is the HDFS architecture completely based on the Google
> Filesystem?
> >> If
> >> >> it
> >> >> > > isnt, what are the differences between the two?
> >> >> > >
> >> >> > > Secondly, is the coupling between Hadoop and HDFS same as how it
> is
> >> >> > between
> >> >> > > the Google's version of Map Reduce and GFS?
> >> >> > >
> >> >> > > Amandeep
> >> >> > >
> >> >> > >
> >> >> > > Amandeep Khurana
> >> >> > > Computer Science Graduate Student
> >> >> > > University of California, Santa Cruz
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
A quick question here. How does a typical hadoop job work at the system
level? What are the various interactions and how does the data flow?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Thanks Matei. If the basic architecture is similar to the Google stuff, I
> can safely just work on the project using the information from the papers.
>
> I am aware of the 4487 jira and the current status of the permissions
> mechanism. I had a look at them earlier.
>
> Cheers
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
>> Forgot to add, this JIRA details the latest security features that are
>> being
>> worked on in Hadoop trunk:
>> https://issues.apache.org/jira/browse/HADOOP-4487.
>> This document describes the current status and limitations of the
>> permissions mechanism:
>> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
>>
>> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
>> wrote:
>>
>> > I think it's safe to assume that Hadoop works like MapReduce/GFS at the
>> > level described in those papers. In particular, in HDFS, there is a
>> master
>> > node containing metadata and a number of slave nodes (datanodes)
>> containing
>> > blocks, as in GFS. Clients start by talking to the master to list
>> > directories, etc. When they want to read a region of some file, they
>> tell
>> > the master the filename and offset, and they receive a list of block
>> > locations (datanodes). They then contact the individual datanodes to
>> read
>> > the blocks. When clients write a file, they first obtain a new block ID
>> and
>> > list of nodes to write it to from the master, then contact the datanodes
>> to
>> > write it (actually, the datanodes pipeline the write as in GFS) and
>> report
>> > when the write is complete. HDFS actually has some security mechanisms
>> built
>> > in, authenticating users based on their Unix ID and providing Unix-like
>> file
>> > permissions. I don't know much about how these are implemented, but they
>> > would be a good place to start looking.
>> >
>> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
>> >wrote:
>> >
>> >> Thanks Matie
>> >>
>> >> I had gone through the architecture document online. I am currently
>> >> working
>> >> on a project towards Security in Hadoop. I do know how the data moves
>> >> around
>> >> in the GFS but wasnt sure how much of that does HDFS follow and how
>> >> different it is from GFS. Can you throw some light on that?
>> >>
>> >> Security would also involve the Map Reduce jobs following the same
>> >> protocols. Thats why the question about how does the Hadoop framework
>> >> integrate with the HDFS, and how different is it from Map Reduce and
>> GFS.
>> >> The GFS and Map Reduce papers give a good information on how those
>> systems
>> >> are designed but there is nothing that concrete for Hadoop that I have
>> >> been
>> >> able to find.
>> >>
>> >> Amandeep
>> >>
>> >>
>> >> Amandeep Khurana
>> >> Computer Science Graduate Student
>> >> University of California, Santa Cruz
>> >>
>> >>
>> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
>> >> wrote:
>> >>
>> >> > Hi Amandeep,
>> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide
>> those
>> >> > capabilities as an open-source project. HDFS is similar to GFS (large
>> >> > blocks, replication, etc); some notable things missing are read-write
>> >> > support in the middle of a file (unlikely to be provided because few
>> >> Hadoop
>> >> > applications require it) and multiple appenders (the record append
>> >> > operation). You can read about HDFS architecture at
>> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
>> >> MapReduce
>> >> > part of Hadoop interacts with HDFS in the same way that Google's
>> >> MapReduce
>> >> > interacts with GFS (shipping computation to the data), although
>> Hadoop
>> >> > MapReduce also supports running over other distributed filesystems.
>> >> >
>> >> > Matei
>> >> >
>> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <amansk@gmail.com
>> >
>> >> > wrote:
>> >> >
>> >> > > Hi
>> >> > >
>> >> > > Is the HDFS architecture completely based on the Google Filesystem?
>> If
>> >> it
>> >> > > isnt, what are the differences between the two?
>> >> > >
>> >> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
>> >> > between
>> >> > > the Google's version of Map Reduce and GFS?
>> >> > >
>> >> > > Amandeep
>> >> > >
>> >> > >
>> >> > > Amandeep Khurana
>> >> > > Computer Science Graduate Student
>> >> > > University of California, Santa Cruz
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
A quick question here. How does a typical hadoop job work at the system
level? What are the various interactions and how does the data flow?
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 3:20 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Thanks Matei. If the basic architecture is similar to the Google stuff, I
> can safely just work on the project using the information from the papers.
>
> I am aware of the 4487 jira and the current status of the permissions
> mechanism. I had a look at them earlier.
>
> Cheers
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
>> Forgot to add, this JIRA details the latest security features that are
>> being
>> worked on in Hadoop trunk:
>> https://issues.apache.org/jira/browse/HADOOP-4487.
>> This document describes the current status and limitations of the
>> permissions mechanism:
>> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
>>
>> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com>
>> wrote:
>>
>> > I think it's safe to assume that Hadoop works like MapReduce/GFS at the
>> > level described in those papers. In particular, in HDFS, there is a
>> master
>> > node containing metadata and a number of slave nodes (datanodes)
>> containing
>> > blocks, as in GFS. Clients start by talking to the master to list
>> > directories, etc. When they want to read a region of some file, they
>> tell
>> > the master the filename and offset, and they receive a list of block
>> > locations (datanodes). They then contact the individual datanodes to
>> read
>> > the blocks. When clients write a file, they first obtain a new block ID
>> and
>> > list of nodes to write it to from the master, then contact the datanodes
>> to
>> > write it (actually, the datanodes pipeline the write as in GFS) and
>> report
>> > when the write is complete. HDFS actually has some security mechanisms
>> built
>> > in, authenticating users based on their Unix ID and providing Unix-like
>> file
>> > permissions. I don't know much about how these are implemented, but they
>> > would be a good place to start looking.
>> >
>> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
>> >wrote:
>> >
>> >> Thanks Matie
>> >>
>> >> I had gone through the architecture document online. I am currently
>> >> working
>> >> on a project towards Security in Hadoop. I do know how the data moves
>> >> around
>> >> in the GFS but wasnt sure how much of that does HDFS follow and how
>> >> different it is from GFS. Can you throw some light on that?
>> >>
>> >> Security would also involve the Map Reduce jobs following the same
>> >> protocols. Thats why the question about how does the Hadoop framework
>> >> integrate with the HDFS, and how different is it from Map Reduce and
>> GFS.
>> >> The GFS and Map Reduce papers give a good information on how those
>> systems
>> >> are designed but there is nothing that concrete for Hadoop that I have
>> >> been
>> >> able to find.
>> >>
>> >> Amandeep
>> >>
>> >>
>> >> Amandeep Khurana
>> >> Computer Science Graduate Student
>> >> University of California, Santa Cruz
>> >>
>> >>
>> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
>> >> wrote:
>> >>
>> >> > Hi Amandeep,
>> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide
>> those
>> >> > capabilities as an open-source project. HDFS is similar to GFS (large
>> >> > blocks, replication, etc); some notable things missing are read-write
>> >> > support in the middle of a file (unlikely to be provided because few
>> >> Hadoop
>> >> > applications require it) and multiple appenders (the record append
>> >> > operation). You can read about HDFS architecture at
>> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
>> >> MapReduce
>> >> > part of Hadoop interacts with HDFS in the same way that Google's
>> >> MapReduce
>> >> > interacts with GFS (shipping computation to the data), although
>> Hadoop
>> >> > MapReduce also supports running over other distributed filesystems.
>> >> >
>> >> > Matei
>> >> >
>> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <amansk@gmail.com
>> >
>> >> > wrote:
>> >> >
>> >> > > Hi
>> >> > >
>> >> > > Is the HDFS architecture completely based on the Google Filesystem?
>> If
>> >> it
>> >> > > isnt, what are the differences between the two?
>> >> > >
>> >> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
>> >> > between
>> >> > > the Google's version of Map Reduce and GFS?
>> >> > >
>> >> > > Amandeep
>> >> > >
>> >> > >
>> >> > > Amandeep Khurana
>> >> > > Computer Science Graduate Student
>> >> > > University of California, Santa Cruz
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Thanks Matei. If the basic architecture is similar to the Google stuff, I
can safely just work on the project using the information from the papers.
I am aware of the 4487 jira and the current status of the permissions
mechanism. I had a look at them earlier.
Cheers
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Forgot to add, this JIRA details the latest security features that are
> being
> worked on in Hadoop trunk:
> https://issues.apache.org/jira/browse/HADOOP-4487.
> This document describes the current status and limitations of the
> permissions mechanism:
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
>
> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > I think it's safe to assume that Hadoop works like MapReduce/GFS at the
> > level described in those papers. In particular, in HDFS, there is a
> master
> > node containing metadata and a number of slave nodes (datanodes)
> containing
> > blocks, as in GFS. Clients start by talking to the master to list
> > directories, etc. When they want to read a region of some file, they tell
> > the master the filename and offset, and they receive a list of block
> > locations (datanodes). They then contact the individual datanodes to read
> > the blocks. When clients write a file, they first obtain a new block ID
> and
> > list of nodes to write it to from the master, then contact the datanodes
> to
> > write it (actually, the datanodes pipeline the write as in GFS) and
> report
> > when the write is complete. HDFS actually has some security mechanisms
> built
> > in, authenticating users based on their Unix ID and providing Unix-like
> file
> > permissions. I don't know much about how these are implemented, but they
> > would be a good place to start looking.
> >
> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> >wrote:
> >
> >> Thanks Matie
> >>
> >> I had gone through the architecture document online. I am currently
> >> working
> >> on a project towards Security in Hadoop. I do know how the data moves
> >> around
> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> >> different it is from GFS. Can you throw some light on that?
> >>
> >> Security would also involve the Map Reduce jobs following the same
> >> protocols. Thats why the question about how does the Hadoop framework
> >> integrate with the HDFS, and how different is it from Map Reduce and
> GFS.
> >> The GFS and Map Reduce papers give a good information on how those
> systems
> >> are designed but there is nothing that concrete for Hadoop that I have
> >> been
> >> able to find.
> >>
> >> Amandeep
> >>
> >>
> >> Amandeep Khurana
> >> Computer Science Graduate Student
> >> University of California, Santa Cruz
> >>
> >>
> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
> >> wrote:
> >>
> >> > Hi Amandeep,
> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide
> those
> >> > capabilities as an open-source project. HDFS is similar to GFS (large
> >> > blocks, replication, etc); some notable things missing are read-write
> >> > support in the middle of a file (unlikely to be provided because few
> >> Hadoop
> >> > applications require it) and multiple appenders (the record append
> >> > operation). You can read about HDFS architecture at
> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> >> MapReduce
> >> > part of Hadoop interacts with HDFS in the same way that Google's
> >> MapReduce
> >> > interacts with GFS (shipping computation to the data), although Hadoop
> >> > MapReduce also supports running over other distributed filesystems.
> >> >
> >> > Matei
> >> >
> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi
> >> > >
> >> > > Is the HDFS architecture completely based on the Google Filesystem?
> If
> >> it
> >> > > isnt, what are the differences between the two?
> >> > >
> >> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
> >> > between
> >> > > the Google's version of Map Reduce and GFS?
> >> > >
> >> > > Amandeep
> >> > >
> >> > >
> >> > > Amandeep Khurana
> >> > > Computer Science Graduate Student
> >> > > University of California, Santa Cruz
> >> > >
> >> >
> >>
> >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Thanks Matei. If the basic architecture is similar to the Google stuff, I
can safely just work on the project using the information from the papers.
I am aware of the 4487 jira and the current status of the permissions
mechanism. I had a look at them earlier.
Cheers
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 2:40 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Forgot to add, this JIRA details the latest security features that are
> being
> worked on in Hadoop trunk:
> https://issues.apache.org/jira/browse/HADOOP-4487.
> This document describes the current status and limitations of the
> permissions mechanism:
> http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
>
> On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com> wrote:
>
> > I think it's safe to assume that Hadoop works like MapReduce/GFS at the
> > level described in those papers. In particular, in HDFS, there is a
> master
> > node containing metadata and a number of slave nodes (datanodes)
> containing
> > blocks, as in GFS. Clients start by talking to the master to list
> > directories, etc. When they want to read a region of some file, they tell
> > the master the filename and offset, and they receive a list of block
> > locations (datanodes). They then contact the individual datanodes to read
> > the blocks. When clients write a file, they first obtain a new block ID
> and
> > list of nodes to write it to from the master, then contact the datanodes
> to
> > write it (actually, the datanodes pipeline the write as in GFS) and
> report
> > when the write is complete. HDFS actually has some security mechanisms
> built
> > in, authenticating users based on their Unix ID and providing Unix-like
> file
> > permissions. I don't know much about how these are implemented, but they
> > would be a good place to start looking.
> >
> > On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <amansk@gmail.com
> >wrote:
> >
> >> Thanks Matie
> >>
> >> I had gone through the architecture document online. I am currently
> >> working
> >> on a project towards Security in Hadoop. I do know how the data moves
> >> around
> >> in the GFS but wasnt sure how much of that does HDFS follow and how
> >> different it is from GFS. Can you throw some light on that?
> >>
> >> Security would also involve the Map Reduce jobs following the same
> >> protocols. Thats why the question about how does the Hadoop framework
> >> integrate with the HDFS, and how different is it from Map Reduce and
> GFS.
> >> The GFS and Map Reduce papers give a good information on how those
> systems
> >> are designed but there is nothing that concrete for Hadoop that I have
> >> been
> >> able to find.
> >>
> >> Amandeep
> >>
> >>
> >> Amandeep Khurana
> >> Computer Science Graduate Student
> >> University of California, Santa Cruz
> >>
> >>
> >> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
> >> wrote:
> >>
> >> > Hi Amandeep,
> >> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide
> those
> >> > capabilities as an open-source project. HDFS is similar to GFS (large
> >> > blocks, replication, etc); some notable things missing are read-write
> >> > support in the middle of a file (unlikely to be provided because few
> >> Hadoop
> >> > applications require it) and multiple appenders (the record append
> >> > operation). You can read about HDFS architecture at
> >> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> >> MapReduce
> >> > part of Hadoop interacts with HDFS in the same way that Google's
> >> MapReduce
> >> > interacts with GFS (shipping computation to the data), although Hadoop
> >> > MapReduce also supports running over other distributed filesystems.
> >> >
> >> > Matei
> >> >
> >> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi
> >> > >
> >> > > Is the HDFS architecture completely based on the Google Filesystem?
> If
> >> it
> >> > > isnt, what are the differences between the two?
> >> > >
> >> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
> >> > between
> >> > > the Google's version of Map Reduce and GFS?
> >> > >
> >> > > Amandeep
> >> > >
> >> > >
> >> > > Amandeep Khurana
> >> > > Computer Science Graduate Student
> >> > > University of California, Santa Cruz
> >> > >
> >> >
> >>
> >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Forgot to add, this JIRA details the latest security features that are being
worked on in Hadoop trunk: https://issues.apache.org/jira/browse/HADOOP-4487.
This document describes the current status and limitations of the
permissions mechanism:
http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> I think it's safe to assume that Hadoop works like MapReduce/GFS at the
> level described in those papers. In particular, in HDFS, there is a master
> node containing metadata and a number of slave nodes (datanodes) containing
> blocks, as in GFS. Clients start by talking to the master to list
> directories, etc. When they want to read a region of some file, they tell
> the master the filename and offset, and they receive a list of block
> locations (datanodes). They then contact the individual datanodes to read
> the blocks. When clients write a file, they first obtain a new block ID and
> list of nodes to write it to from the master, then contact the datanodes to
> write it (actually, the datanodes pipeline the write as in GFS) and report
> when the write is complete. HDFS actually has some security mechanisms built
> in, authenticating users based on their Unix ID and providing Unix-like file
> permissions. I don't know much about how these are implemented, but they
> would be a good place to start looking.
>
> On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <am...@gmail.com>wrote:
>
>> Thanks Matie
>>
>> I had gone through the architecture document online. I am currently
>> working
>> on a project towards Security in Hadoop. I do know how the data moves
>> around
>> in the GFS but wasnt sure how much of that does HDFS follow and how
>> different it is from GFS. Can you throw some light on that?
>>
>> Security would also involve the Map Reduce jobs following the same
>> protocols. Thats why the question about how does the Hadoop framework
>> integrate with the HDFS, and how different is it from Map Reduce and GFS.
>> The GFS and Map Reduce papers give a good information on how those systems
>> are designed but there is nothing that concrete for Hadoop that I have
>> been
>> able to find.
>>
>> Amandeep
>>
>>
>> Amandeep Khurana
>> Computer Science Graduate Student
>> University of California, Santa Cruz
>>
>>
>> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
>> wrote:
>>
>> > Hi Amandeep,
>> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
>> > capabilities as an open-source project. HDFS is similar to GFS (large
>> > blocks, replication, etc); some notable things missing are read-write
>> > support in the middle of a file (unlikely to be provided because few
>> Hadoop
>> > applications require it) and multiple appenders (the record append
>> > operation). You can read about HDFS architecture at
>> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
>> MapReduce
>> > part of Hadoop interacts with HDFS in the same way that Google's
>> MapReduce
>> > interacts with GFS (shipping computation to the data), although Hadoop
>> > MapReduce also supports running over other distributed filesystems.
>> >
>> > Matei
>> >
>> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
>> > wrote:
>> >
>> > > Hi
>> > >
>> > > Is the HDFS architecture completely based on the Google Filesystem? If
>> it
>> > > isnt, what are the differences between the two?
>> > >
>> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
>> > between
>> > > the Google's version of Map Reduce and GFS?
>> > >
>> > > Amandeep
>> > >
>> > >
>> > > Amandeep Khurana
>> > > Computer Science Graduate Student
>> > > University of California, Santa Cruz
>> > >
>> >
>>
>
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Forgot to add, this JIRA details the latest security features that are being
worked on in Hadoop trunk: https://issues.apache.org/jira/browse/HADOOP-4487.
This document describes the current status and limitations of the
permissions mechanism:
http://hadoop.apache.org/core/docs/current/hdfs_permissions_guide.html.
On Sun, Feb 15, 2009 at 2:35 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> I think it's safe to assume that Hadoop works like MapReduce/GFS at the
> level described in those papers. In particular, in HDFS, there is a master
> node containing metadata and a number of slave nodes (datanodes) containing
> blocks, as in GFS. Clients start by talking to the master to list
> directories, etc. When they want to read a region of some file, they tell
> the master the filename and offset, and they receive a list of block
> locations (datanodes). They then contact the individual datanodes to read
> the blocks. When clients write a file, they first obtain a new block ID and
> list of nodes to write it to from the master, then contact the datanodes to
> write it (actually, the datanodes pipeline the write as in GFS) and report
> when the write is complete. HDFS actually has some security mechanisms built
> in, authenticating users based on their Unix ID and providing Unix-like file
> permissions. I don't know much about how these are implemented, but they
> would be a good place to start looking.
>
> On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <am...@gmail.com>wrote:
>
>> Thanks Matie
>>
>> I had gone through the architecture document online. I am currently
>> working
>> on a project towards Security in Hadoop. I do know how the data moves
>> around
>> in the GFS but wasnt sure how much of that does HDFS follow and how
>> different it is from GFS. Can you throw some light on that?
>>
>> Security would also involve the Map Reduce jobs following the same
>> protocols. Thats why the question about how does the Hadoop framework
>> integrate with the HDFS, and how different is it from Map Reduce and GFS.
>> The GFS and Map Reduce papers give a good information on how those systems
>> are designed but there is nothing that concrete for Hadoop that I have
>> been
>> able to find.
>>
>> Amandeep
>>
>>
>> Amandeep Khurana
>> Computer Science Graduate Student
>> University of California, Santa Cruz
>>
>>
>> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
>> wrote:
>>
>> > Hi Amandeep,
>> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
>> > capabilities as an open-source project. HDFS is similar to GFS (large
>> > blocks, replication, etc); some notable things missing are read-write
>> > support in the middle of a file (unlikely to be provided because few
>> Hadoop
>> > applications require it) and multiple appenders (the record append
>> > operation). You can read about HDFS architecture at
>> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
>> MapReduce
>> > part of Hadoop interacts with HDFS in the same way that Google's
>> MapReduce
>> > interacts with GFS (shipping computation to the data), although Hadoop
>> > MapReduce also supports running over other distributed filesystems.
>> >
>> > Matei
>> >
>> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
>> > wrote:
>> >
>> > > Hi
>> > >
>> > > Is the HDFS architecture completely based on the Google Filesystem? If
>> it
>> > > isnt, what are the differences between the two?
>> > >
>> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
>> > between
>> > > the Google's version of Map Reduce and GFS?
>> > >
>> > > Amandeep
>> > >
>> > >
>> > > Amandeep Khurana
>> > > Computer Science Graduate Student
>> > > University of California, Santa Cruz
>> > >
>> >
>>
>
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
I think it's safe to assume that Hadoop works like MapReduce/GFS at the
level described in those papers. In particular, in HDFS, there is a master
node containing metadata and a number of slave nodes (datanodes) containing
blocks, as in GFS. Clients start by talking to the master to list
directories, etc. When they want to read a region of some file, they tell
the master the filename and offset, and they receive a list of block
locations (datanodes). They then contact the individual datanodes to read
the blocks. When clients write a file, they first obtain a new block ID and
list of nodes to write it to from the master, then contact the datanodes to
write it (actually, the datanodes pipeline the write as in GFS) and report
when the write is complete. HDFS actually has some security mechanisms built
in, authenticating users based on their Unix ID and providing Unix-like file
permissions. I don't know much about how these are implemented, but they
would be a good place to start looking.
On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Thanks Matie
>
> I had gone through the architecture document online. I am currently working
> on a project towards Security in Hadoop. I do know how the data moves
> around
> in the GFS but wasnt sure how much of that does HDFS follow and how
> different it is from GFS. Can you throw some light on that?
>
> Security would also involve the Map Reduce jobs following the same
> protocols. Thats why the question about how does the Hadoop framework
> integrate with the HDFS, and how different is it from Map Reduce and GFS.
> The GFS and Map Reduce papers give a good information on how those systems
> are designed but there is nothing that concrete for Hadoop that I have been
> able to find.
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
>
> > Hi Amandeep,
> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
> > capabilities as an open-source project. HDFS is similar to GFS (large
> > blocks, replication, etc); some notable things missing are read-write
> > support in the middle of a file (unlikely to be provided because few
> Hadoop
> > applications require it) and multiple appenders (the record append
> > operation). You can read about HDFS architecture at
> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> MapReduce
> > part of Hadoop interacts with HDFS in the same way that Google's
> MapReduce
> > interacts with GFS (shipping computation to the data), although Hadoop
> > MapReduce also supports running over other distributed filesystems.
> >
> > Matei
> >
> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > Is the HDFS architecture completely based on the Google Filesystem? If
> it
> > > isnt, what are the differences between the two?
> > >
> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
> > between
> > > the Google's version of Map Reduce and GFS?
> > >
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
I think it's safe to assume that Hadoop works like MapReduce/GFS at the
level described in those papers. In particular, in HDFS, there is a master
node containing metadata and a number of slave nodes (datanodes) containing
blocks, as in GFS. Clients start by talking to the master to list
directories, etc. When they want to read a region of some file, they tell
the master the filename and offset, and they receive a list of block
locations (datanodes). They then contact the individual datanodes to read
the blocks. When clients write a file, they first obtain a new block ID and
list of nodes to write it to from the master, then contact the datanodes to
write it (actually, the datanodes pipeline the write as in GFS) and report
when the write is complete. HDFS actually has some security mechanisms built
in, authenticating users based on their Unix ID and providing Unix-like file
permissions. I don't know much about how these are implemented, but they
would be a good place to start looking.
On Sun, Feb 15, 2009 at 1:36 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Thanks Matie
>
> I had gone through the architecture document online. I am currently working
> on a project towards Security in Hadoop. I do know how the data moves
> around
> in the GFS but wasnt sure how much of that does HDFS follow and how
> different it is from GFS. Can you throw some light on that?
>
> Security would also involve the Map Reduce jobs following the same
> protocols. Thats why the question about how does the Hadoop framework
> integrate with the HDFS, and how different is it from Map Reduce and GFS.
> The GFS and Map Reduce papers give a good information on how those systems
> are designed but there is nothing that concrete for Hadoop that I have been
> able to find.
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com>
> wrote:
>
> > Hi Amandeep,
> > Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
> > capabilities as an open-source project. HDFS is similar to GFS (large
> > blocks, replication, etc); some notable things missing are read-write
> > support in the middle of a file (unlikely to be provided because few
> Hadoop
> > applications require it) and multiple appenders (the record append
> > operation). You can read about HDFS architecture at
> > http://hadoop.apache.org/core/docs/current/hdfs_design.html. The
> MapReduce
> > part of Hadoop interacts with HDFS in the same way that Google's
> MapReduce
> > interacts with GFS (shipping computation to the data), although Hadoop
> > MapReduce also supports running over other distributed filesystems.
> >
> > Matei
> >
> > On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > Is the HDFS architecture completely based on the Google Filesystem? If
> it
> > > isnt, what are the differences between the two?
> > >
> > > Secondly, is the coupling between Hadoop and HDFS same as how it is
> > between
> > > the Google's version of Map Reduce and GFS?
> > >
> > > Amandeep
> > >
> > >
> > > Amandeep Khurana
> > > Computer Science Graduate Student
> > > University of California, Santa Cruz
> > >
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Thanks Matie
I had gone through the architecture document online. I am currently working
on a project towards Security in Hadoop. I do know how the data moves around
in the GFS but wasnt sure how much of that does HDFS follow and how
different it is from GFS. Can you throw some light on that?
Security would also involve the Map Reduce jobs following the same
protocols. Thats why the question about how does the Hadoop framework
integrate with the HDFS, and how different is it from Map Reduce and GFS.
The GFS and Map Reduce papers give a good information on how those systems
are designed but there is nothing that concrete for Hadoop that I have been
able to find.
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Hi Amandeep,
> Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
> capabilities as an open-source project. HDFS is similar to GFS (large
> blocks, replication, etc); some notable things missing are read-write
> support in the middle of a file (unlikely to be provided because few Hadoop
> applications require it) and multiple appenders (the record append
> operation). You can read about HDFS architecture at
> http://hadoop.apache.org/core/docs/current/hdfs_design.html. The MapReduce
> part of Hadoop interacts with HDFS in the same way that Google's MapReduce
> interacts with GFS (shipping computation to the data), although Hadoop
> MapReduce also supports running over other distributed filesystems.
>
> Matei
>
> On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Hi
> >
> > Is the HDFS architecture completely based on the Google Filesystem? If it
> > isnt, what are the differences between the two?
> >
> > Secondly, is the coupling between Hadoop and HDFS same as how it is
> between
> > the Google's version of Map Reduce and GFS?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
>
Re: HDFS architecture based on GFS?
Posted by Amandeep Khurana <am...@gmail.com>.
Thanks Matie
I had gone through the architecture document online. I am currently working
on a project towards Security in Hadoop. I do know how the data moves around
in the GFS but wasnt sure how much of that does HDFS follow and how
different it is from GFS. Can you throw some light on that?
Security would also involve the Map Reduce jobs following the same
protocols. Thats why the question about how does the Hadoop framework
integrate with the HDFS, and how different is it from Map Reduce and GFS.
The GFS and Map Reduce papers give a good information on how those systems
are designed but there is nothing that concrete for Hadoop that I have been
able to find.
Amandeep
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Feb 15, 2009 at 12:07 PM, Matei Zaharia <ma...@cloudera.com> wrote:
> Hi Amandeep,
> Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
> capabilities as an open-source project. HDFS is similar to GFS (large
> blocks, replication, etc); some notable things missing are read-write
> support in the middle of a file (unlikely to be provided because few Hadoop
> applications require it) and multiple appenders (the record append
> operation). You can read about HDFS architecture at
> http://hadoop.apache.org/core/docs/current/hdfs_design.html. The MapReduce
> part of Hadoop interacts with HDFS in the same way that Google's MapReduce
> interacts with GFS (shipping computation to the data), although Hadoop
> MapReduce also supports running over other distributed filesystems.
>
> Matei
>
> On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > Hi
> >
> > Is the HDFS architecture completely based on the Google Filesystem? If it
> > isnt, what are the differences between the two?
> >
> > Secondly, is the coupling between Hadoop and HDFS same as how it is
> between
> > the Google's version of Map Reduce and GFS?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Hi Amandeep,
Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
capabilities as an open-source project. HDFS is similar to GFS (large
blocks, replication, etc); some notable things missing are read-write
support in the middle of a file (unlikely to be provided because few Hadoop
applications require it) and multiple appenders (the record append
operation). You can read about HDFS architecture at
http://hadoop.apache.org/core/docs/current/hdfs_design.html. The MapReduce
part of Hadoop interacts with HDFS in the same way that Google's MapReduce
interacts with GFS (shipping computation to the data), although Hadoop
MapReduce also supports running over other distributed filesystems.
Matei
On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com> wrote:
> Hi
>
> Is the HDFS architecture completely based on the Google Filesystem? If it
> isnt, what are the differences between the two?
>
> Secondly, is the coupling between Hadoop and HDFS same as how it is between
> the Google's version of Map Reduce and GFS?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
Re: HDFS architecture based on GFS?
Posted by Matei Zaharia <ma...@cloudera.com>.
Hi Amandeep,
Hadoop is definitely inspired by MapReduce/GFS and aims to provide those
capabilities as an open-source project. HDFS is similar to GFS (large
blocks, replication, etc); some notable things missing are read-write
support in the middle of a file (unlikely to be provided because few Hadoop
applications require it) and multiple appenders (the record append
operation). You can read about HDFS architecture at
http://hadoop.apache.org/core/docs/current/hdfs_design.html. The MapReduce
part of Hadoop interacts with HDFS in the same way that Google's MapReduce
interacts with GFS (shipping computation to the data), although Hadoop
MapReduce also supports running over other distributed filesystems.
Matei
On Sun, Feb 15, 2009 at 11:57 AM, Amandeep Khurana <am...@gmail.com> wrote:
> Hi
>
> Is the HDFS architecture completely based on the Google Filesystem? If it
> isnt, what are the differences between the two?
>
> Secondly, is the coupling between Hadoop and HDFS same as how it is between
> the Google's version of Map Reduce and GFS?
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>