You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Arijit Mukherjee <ar...@gmail.com> on 2009/02/26 12:30:33 UTC
first tests with hive
Hi All
I'm a newbie to hadoop and hive and am trying to set it up on a cluster. I
am trying to find out more about the partitioning as done in Hive. If I use
a create table statement with a "partitioned by" clause, which as per the
documentation is a virtual column, is the data physically partitioned on
multiple nodes (meaning would the different nodes have different subsets of
the actual data)? Is it possible to check the content of each partition?
Actually, I'm trying to compare the concepts of Hive with some other
frameworks such as Greenplum where the data is distributed across nodes.
Any help/pointers is appreciated. Thanx in advance.
Cheers
Arijit
--
"And when the night is cloudy,
There is still a light that shines on me,
Shine on until tomorrow, let it be."
Re: first tests with hive
Posted by Arijit Mukherjee <ar...@gmail.com>.
Thanx mate - I'll go through these and get back.
Arijit
2009/2/26 Raghu Murthy <ra...@facebook.com>
> Hi Arijit,
>
> Hive uses HDFS as its underlying storage. When a table is partitioned by
> some column, say 'ds', hive stores partitions in separate HDFS directories.
> So, if there are two partitions with ds values '2009-02-20' and
> '2009-02-21', their corresponding data is stored in the HDFS directories:
> /user/hive/warehouse/<tablename>/ds=2009-02-20/*
> /user/hive/warehouse/<tablename>/ds=2009-02-21/*
> This allows Hive to scan only the HDFS files necessary for a given query
> (via input pruning). For more information about the hive data model see
>
> http://wiki.apache.org/hadoop/Hive/Design#head-507fb621ecdf8a9c2dfcfc5db2dda
> 52a724534fe<http://wiki.apache.org/hadoop/Hive/Design#head-507fb621ecdf8a9c2dfcfc5db2dda%0A52a724534fe>
>
> The actual distribution of data in files across nodes is determined by HDFS
> which supports different levels of replication. See
> http://hadoop.apache.org/core/docs/current/hdfs_design.html for more
> details.
>
> Hope this helps.
> raghu
>
> On 2/26/09 3:30 AM, "Arijit Mukherjee" <ar...@gmail.com> wrote:
>
> > Hi All
> >
> > I'm a newbie to hadoop and hive and am trying to set it up on a cluster.
> I am
> > trying to find out more about the partitioning as done in Hive. If I use
> a
> > create table statement with a "partitioned by" clause, which as per the
> > documentation is a virtual column, is the data physically partitioned on
> > multiple nodes (meaning would the different nodes have different subsets
> of
> > the actual data)? Is it possible to check the content of each partition?
> >
> > Actually, I'm trying to compare the concepts of Hive with some other
> > frameworks such as Greenplum where the data is distributed across nodes.
> >
> > Any help/pointers is appreciated. Thanx in advance.
> >
> > Cheers
> > Arijit
>
>
--
"And when the night is cloudy,
There is still a light that shines on me,
Shine on until tomorrow, let it be."
Re: first tests with hive
Posted by Raghu Murthy <ra...@facebook.com>.
Hi Arijit,
Hive uses HDFS as its underlying storage. When a table is partitioned by
some column, say 'ds', hive stores partitions in separate HDFS directories.
So, if there are two partitions with ds values '2009-02-20' and
'2009-02-21', their corresponding data is stored in the HDFS directories:
/user/hive/warehouse/<tablename>/ds=2009-02-20/*
/user/hive/warehouse/<tablename>/ds=2009-02-21/*
This allows Hive to scan only the HDFS files necessary for a given query
(via input pruning). For more information about the hive data model see
http://wiki.apache.org/hadoop/Hive/Design#head-507fb621ecdf8a9c2dfcfc5db2dda
52a724534fe
The actual distribution of data in files across nodes is determined by HDFS
which supports different levels of replication. See
http://hadoop.apache.org/core/docs/current/hdfs_design.html for more
details.
Hope this helps.
raghu
On 2/26/09 3:30 AM, "Arijit Mukherjee" <ar...@gmail.com> wrote:
> Hi All
>
> I'm a newbie to hadoop and hive and am trying to set it up on a cluster. I am
> trying to find out more about the partitioning as done in Hive. If I use a
> create table statement with a "partitioned by" clause, which as per the
> documentation is a virtual column, is the data physically partitioned on
> multiple nodes (meaning would the different nodes have different subsets of
> the actual data)? Is it possible to check the content of each partition?
>
> Actually, I'm trying to compare the concepts of Hive with some other
> frameworks such as Greenplum where the data is distributed across nodes.
>
> Any help/pointers is appreciated. Thanx in advance.
>
> Cheers
> Arijit