You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Jay Vee <jv...@gmail.com> on 2013/12/19 21:35:29 UTC
from relational to bigger data
We have a large relational database ( ~ 500 GB, hundreds of tables ).
We have summary tables that we rebuild from scratch each night that takes
about 10 hours.
>From these summary tables, we have a web interface that accesses the
summary tables to build reports.
There is a business reason for doing a complete rebuild of the summary
tables each night, and using
views (as in the sense of Oracle views) is not an option at this time.
If I wanted to leverage Big Data technologies to speed up the summary table
rebuild, what would be the first step into getting all data into some big
data storage technology?
Ideally in the end, we want to retain the summary tables in a relational
database and have reporting work the same without modifications.
It's just the crunching of the data and building these relational summary
tables where we need a significant performance increase.
Re: from relational to bigger data
Posted by Andre Kelpe <ak...@concurrentinc.com>.
You could also give cascading lingual a try:
http://www.cascading.org/lingual/ http://docs.cascading.org/lingual/1.0/
We have a connector for oracle (
https://github.com/Cascading/cascading-jdbc#oracle), so you could read the
data from oracle do the processing on a hadoop cluster and write it back
into oracle all via SQL or a combination of SQL and Java/Cascading (
https://github.com/Cascading/cascading-jdbc#in-lingual).
- André
On Thu, Dec 19, 2013 at 9:35 PM, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary
> table rebuild, what would be the first step into getting all data into some
> big data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
>
>
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: from relational to bigger data
Posted by Andre Kelpe <ak...@concurrentinc.com>.
You could also give cascading lingual a try:
http://www.cascading.org/lingual/ http://docs.cascading.org/lingual/1.0/
We have a connector for oracle (
https://github.com/Cascading/cascading-jdbc#oracle), so you could read the
data from oracle do the processing on a hadoop cluster and write it back
into oracle all via SQL or a combination of SQL and Java/Cascading (
https://github.com/Cascading/cascading-jdbc#in-lingual).
- André
On Thu, Dec 19, 2013 at 9:35 PM, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary
> table rebuild, what would be the first step into getting all data into some
> big data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
>
>
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: from relational to bigger data
Posted by Bertrand Dechoux <de...@gmail.com>.
And to make things a bit more explicit, Sqoop is the name of a project "SQL
to Hadopp".
http://sqoop.apache.org/
As for crunch, I guess Chris used the generic term.
You could use MapReduce jobs with the Java API, Pig, Hive, Cascading,
Crunch (indeed)...
Regards
Bertrand
On Thu, Dec 19, 2013 at 10:59 PM, Vinay Bagare <vb...@me.com> wrote:
> I would also look at current setup.
> I agree with Chris that 500 GB is fairly insignificant.
>
>
> Best,
> Vinay Bagare
>
>
>
> On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
>
> > In big data terms, 500G isn't big. But, moving that much data around
> > every night is not trivial either. I'm going to guess at a lot here,
> > but at a very high level.
> >
> > 1. Sqoop the data required to build the summary tables into Hadoop.
> > 2. Crunch the summaries into new tables (really just files on Hadoop)
> > 3. Sqoop the summarized data back out to Oracle
> > 4. Build Indices as needed.
> >
> > Depending on the size of the data being sqoop'd, this might help. It
> > might also take longer. A real solution would require more details
> > and analysis.
> >
> > Chris
> >
> > On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> >> We have a large relational database ( ~ 500 GB, hundreds of tables ).
> >>
> >> We have summary tables that we rebuild from scratch each night that
> takes
> >> about 10 hours.
> >> From these summary tables, we have a web interface that accesses the
> >> summary tables to build reports.
> >>
> >> There is a business reason for doing a complete rebuild of the summary
> >> tables each night, and using
> >> views (as in the sense of Oracle views) is not an option at this time.
> >>
> >> If I wanted to leverage Big Data technologies to speed up the summary
> table
> >> rebuild, what would be the first step into getting all data into some
> big
> >> data storage technology?
> >>
> >> Ideally in the end, we want to retain the summary tables in a relational
> >> database and have reporting work the same without modifications.
> >>
> >> It's just the crunching of the data and building these relational
> summary
> >> tables where we need a significant performance increase.
> >>
>
>
Re: from relational to bigger data
Posted by Bertrand Dechoux <de...@gmail.com>.
And to make things a bit more explicit, Sqoop is the name of a project "SQL
to Hadopp".
http://sqoop.apache.org/
As for crunch, I guess Chris used the generic term.
You could use MapReduce jobs with the Java API, Pig, Hive, Cascading,
Crunch (indeed)...
Regards
Bertrand
On Thu, Dec 19, 2013 at 10:59 PM, Vinay Bagare <vb...@me.com> wrote:
> I would also look at current setup.
> I agree with Chris that 500 GB is fairly insignificant.
>
>
> Best,
> Vinay Bagare
>
>
>
> On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
>
> > In big data terms, 500G isn't big. But, moving that much data around
> > every night is not trivial either. I'm going to guess at a lot here,
> > but at a very high level.
> >
> > 1. Sqoop the data required to build the summary tables into Hadoop.
> > 2. Crunch the summaries into new tables (really just files on Hadoop)
> > 3. Sqoop the summarized data back out to Oracle
> > 4. Build Indices as needed.
> >
> > Depending on the size of the data being sqoop'd, this might help. It
> > might also take longer. A real solution would require more details
> > and analysis.
> >
> > Chris
> >
> > On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> >> We have a large relational database ( ~ 500 GB, hundreds of tables ).
> >>
> >> We have summary tables that we rebuild from scratch each night that
> takes
> >> about 10 hours.
> >> From these summary tables, we have a web interface that accesses the
> >> summary tables to build reports.
> >>
> >> There is a business reason for doing a complete rebuild of the summary
> >> tables each night, and using
> >> views (as in the sense of Oracle views) is not an option at this time.
> >>
> >> If I wanted to leverage Big Data technologies to speed up the summary
> table
> >> rebuild, what would be the first step into getting all data into some
> big
> >> data storage technology?
> >>
> >> Ideally in the end, we want to retain the summary tables in a relational
> >> database and have reporting work the same without modifications.
> >>
> >> It's just the crunching of the data and building these relational
> summary
> >> tables where we need a significant performance increase.
> >>
>
>
Re: from relational to bigger data
Posted by Bertrand Dechoux <de...@gmail.com>.
And to make things a bit more explicit, Sqoop is the name of a project "SQL
to Hadopp".
http://sqoop.apache.org/
As for crunch, I guess Chris used the generic term.
You could use MapReduce jobs with the Java API, Pig, Hive, Cascading,
Crunch (indeed)...
Regards
Bertrand
On Thu, Dec 19, 2013 at 10:59 PM, Vinay Bagare <vb...@me.com> wrote:
> I would also look at current setup.
> I agree with Chris that 500 GB is fairly insignificant.
>
>
> Best,
> Vinay Bagare
>
>
>
> On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
>
> > In big data terms, 500G isn't big. But, moving that much data around
> > every night is not trivial either. I'm going to guess at a lot here,
> > but at a very high level.
> >
> > 1. Sqoop the data required to build the summary tables into Hadoop.
> > 2. Crunch the summaries into new tables (really just files on Hadoop)
> > 3. Sqoop the summarized data back out to Oracle
> > 4. Build Indices as needed.
> >
> > Depending on the size of the data being sqoop'd, this might help. It
> > might also take longer. A real solution would require more details
> > and analysis.
> >
> > Chris
> >
> > On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> >> We have a large relational database ( ~ 500 GB, hundreds of tables ).
> >>
> >> We have summary tables that we rebuild from scratch each night that
> takes
> >> about 10 hours.
> >> From these summary tables, we have a web interface that accesses the
> >> summary tables to build reports.
> >>
> >> There is a business reason for doing a complete rebuild of the summary
> >> tables each night, and using
> >> views (as in the sense of Oracle views) is not an option at this time.
> >>
> >> If I wanted to leverage Big Data technologies to speed up the summary
> table
> >> rebuild, what would be the first step into getting all data into some
> big
> >> data storage technology?
> >>
> >> Ideally in the end, we want to retain the summary tables in a relational
> >> database and have reporting work the same without modifications.
> >>
> >> It's just the crunching of the data and building these relational
> summary
> >> tables where we need a significant performance increase.
> >>
>
>
Re: from relational to bigger data
Posted by Bertrand Dechoux <de...@gmail.com>.
And to make things a bit more explicit, Sqoop is the name of a project "SQL
to Hadopp".
http://sqoop.apache.org/
As for crunch, I guess Chris used the generic term.
You could use MapReduce jobs with the Java API, Pig, Hive, Cascading,
Crunch (indeed)...
Regards
Bertrand
On Thu, Dec 19, 2013 at 10:59 PM, Vinay Bagare <vb...@me.com> wrote:
> I would also look at current setup.
> I agree with Chris that 500 GB is fairly insignificant.
>
>
> Best,
> Vinay Bagare
>
>
>
> On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
>
> > In big data terms, 500G isn't big. But, moving that much data around
> > every night is not trivial either. I'm going to guess at a lot here,
> > but at a very high level.
> >
> > 1. Sqoop the data required to build the summary tables into Hadoop.
> > 2. Crunch the summaries into new tables (really just files on Hadoop)
> > 3. Sqoop the summarized data back out to Oracle
> > 4. Build Indices as needed.
> >
> > Depending on the size of the data being sqoop'd, this might help. It
> > might also take longer. A real solution would require more details
> > and analysis.
> >
> > Chris
> >
> > On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> >> We have a large relational database ( ~ 500 GB, hundreds of tables ).
> >>
> >> We have summary tables that we rebuild from scratch each night that
> takes
> >> about 10 hours.
> >> From these summary tables, we have a web interface that accesses the
> >> summary tables to build reports.
> >>
> >> There is a business reason for doing a complete rebuild of the summary
> >> tables each night, and using
> >> views (as in the sense of Oracle views) is not an option at this time.
> >>
> >> If I wanted to leverage Big Data technologies to speed up the summary
> table
> >> rebuild, what would be the first step into getting all data into some
> big
> >> data storage technology?
> >>
> >> Ideally in the end, we want to retain the summary tables in a relational
> >> database and have reporting work the same without modifications.
> >>
> >> It's just the crunching of the data and building these relational
> summary
> >> tables where we need a significant performance increase.
> >>
>
>
Re: from relational to bigger data
Posted by Vinay Bagare <vb...@me.com>.
I would also look at current setup.
I agree with Chris that 500 GB is fairly insignificant.
Best,
Vinay Bagare
On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
> In big data terms, 500G isn't big. But, moving that much data around
> every night is not trivial either. I'm going to guess at a lot here,
> but at a very high level.
>
> 1. Sqoop the data required to build the summary tables into Hadoop.
> 2. Crunch the summaries into new tables (really just files on Hadoop)
> 3. Sqoop the summarized data back out to Oracle
> 4. Build Indices as needed.
>
> Depending on the size of the data being sqoop'd, this might help. It
> might also take longer. A real solution would require more details
> and analysis.
>
> Chris
>
> On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
>> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>>
>> We have summary tables that we rebuild from scratch each night that takes
>> about 10 hours.
>> From these summary tables, we have a web interface that accesses the
>> summary tables to build reports.
>>
>> There is a business reason for doing a complete rebuild of the summary
>> tables each night, and using
>> views (as in the sense of Oracle views) is not an option at this time.
>>
>> If I wanted to leverage Big Data technologies to speed up the summary table
>> rebuild, what would be the first step into getting all data into some big
>> data storage technology?
>>
>> Ideally in the end, we want to retain the summary tables in a relational
>> database and have reporting work the same without modifications.
>>
>> It's just the crunching of the data and building these relational summary
>> tables where we need a significant performance increase.
>>
Re: from relational to bigger data
Posted by Vinay Bagare <vb...@me.com>.
I would also look at current setup.
I agree with Chris that 500 GB is fairly insignificant.
Best,
Vinay Bagare
On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
> In big data terms, 500G isn't big. But, moving that much data around
> every night is not trivial either. I'm going to guess at a lot here,
> but at a very high level.
>
> 1. Sqoop the data required to build the summary tables into Hadoop.
> 2. Crunch the summaries into new tables (really just files on Hadoop)
> 3. Sqoop the summarized data back out to Oracle
> 4. Build Indices as needed.
>
> Depending on the size of the data being sqoop'd, this might help. It
> might also take longer. A real solution would require more details
> and analysis.
>
> Chris
>
> On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
>> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>>
>> We have summary tables that we rebuild from scratch each night that takes
>> about 10 hours.
>> From these summary tables, we have a web interface that accesses the
>> summary tables to build reports.
>>
>> There is a business reason for doing a complete rebuild of the summary
>> tables each night, and using
>> views (as in the sense of Oracle views) is not an option at this time.
>>
>> If I wanted to leverage Big Data technologies to speed up the summary table
>> rebuild, what would be the first step into getting all data into some big
>> data storage technology?
>>
>> Ideally in the end, we want to retain the summary tables in a relational
>> database and have reporting work the same without modifications.
>>
>> It's just the crunching of the data and building these relational summary
>> tables where we need a significant performance increase.
>>
Re: from relational to bigger data
Posted by Vinay Bagare <vb...@me.com>.
I would also look at current setup.
I agree with Chris that 500 GB is fairly insignificant.
Best,
Vinay Bagare
On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
> In big data terms, 500G isn't big. But, moving that much data around
> every night is not trivial either. I'm going to guess at a lot here,
> but at a very high level.
>
> 1. Sqoop the data required to build the summary tables into Hadoop.
> 2. Crunch the summaries into new tables (really just files on Hadoop)
> 3. Sqoop the summarized data back out to Oracle
> 4. Build Indices as needed.
>
> Depending on the size of the data being sqoop'd, this might help. It
> might also take longer. A real solution would require more details
> and analysis.
>
> Chris
>
> On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
>> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>>
>> We have summary tables that we rebuild from scratch each night that takes
>> about 10 hours.
>> From these summary tables, we have a web interface that accesses the
>> summary tables to build reports.
>>
>> There is a business reason for doing a complete rebuild of the summary
>> tables each night, and using
>> views (as in the sense of Oracle views) is not an option at this time.
>>
>> If I wanted to leverage Big Data technologies to speed up the summary table
>> rebuild, what would be the first step into getting all data into some big
>> data storage technology?
>>
>> Ideally in the end, we want to retain the summary tables in a relational
>> database and have reporting work the same without modifications.
>>
>> It's just the crunching of the data and building these relational summary
>> tables where we need a significant performance increase.
>>
Re: from relational to bigger data
Posted by Vinay Bagare <vb...@me.com>.
I would also look at current setup.
I agree with Chris that 500 GB is fairly insignificant.
Best,
Vinay Bagare
On Dec 19, 2013, at 12:51 PM, Chris Embree <ce...@gmail.com> wrote:
> In big data terms, 500G isn't big. But, moving that much data around
> every night is not trivial either. I'm going to guess at a lot here,
> but at a very high level.
>
> 1. Sqoop the data required to build the summary tables into Hadoop.
> 2. Crunch the summaries into new tables (really just files on Hadoop)
> 3. Sqoop the summarized data back out to Oracle
> 4. Build Indices as needed.
>
> Depending on the size of the data being sqoop'd, this might help. It
> might also take longer. A real solution would require more details
> and analysis.
>
> Chris
>
> On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
>> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>>
>> We have summary tables that we rebuild from scratch each night that takes
>> about 10 hours.
>> From these summary tables, we have a web interface that accesses the
>> summary tables to build reports.
>>
>> There is a business reason for doing a complete rebuild of the summary
>> tables each night, and using
>> views (as in the sense of Oracle views) is not an option at this time.
>>
>> If I wanted to leverage Big Data technologies to speed up the summary table
>> rebuild, what would be the first step into getting all data into some big
>> data storage technology?
>>
>> Ideally in the end, we want to retain the summary tables in a relational
>> database and have reporting work the same without modifications.
>>
>> It's just the crunching of the data and building these relational summary
>> tables where we need a significant performance increase.
>>
Re: from relational to bigger data
Posted by Chris Embree <ce...@gmail.com>.
In big data terms, 500G isn't big. But, moving that much data around
every night is not trivial either. I'm going to guess at a lot here,
but at a very high level.
1. Sqoop the data required to build the summary tables into Hadoop.
2. Crunch the summaries into new tables (really just files on Hadoop)
3. Sqoop the summarized data back out to Oracle
4. Build Indices as needed.
Depending on the size of the data being sqoop'd, this might help. It
might also take longer. A real solution would require more details
and analysis.
Chris
On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary table
> rebuild, what would be the first step into getting all data into some big
> data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
Re:from relational to bigger data
Posted by 小网客 <sm...@foxmail.com>.
Writing an inputformat then use mapreduce job to deal with your relational data
------------------
---------------------------------------------
BestWishes！
小网客
Blog:http://snv.iteye.com/
Email:1134687142@qq.com
------------------ Original ------------------
From: "Jay Vee";<jv...@gmail.com>;
Date: Fri, Dec 20, 2013 04:35 AM
To: "user"<us...@hadoop.apache.org>;
Subject: from relational to bigger data
We have a large relational database ( ~ 500 GB, hundreds of tables ).
We have summary tables that we rebuild from scratch each night that takes about 10 hours.
From these summary tables, we have a web interface that accesses the summary tables to build reports.
There is a business reason for doing a complete rebuild of the summary tables each night, and using
views (as in the sense of Oracle views) is not an option at this time.
If I wanted to leverage Big Data technologies to speed up the summary table rebuild, what would be the first step into getting all data into some big data storage technology?
Ideally in the end, we want to retain the summary tables in a relational database and have reporting work the same without modifications.
It's just the crunching of the data and building these relational summary tables where we need a significant performance increase.
Re:from relational to bigger data
Posted by 小网客 <sm...@foxmail.com>.
Writing an inputformat then use mapreduce job to deal with your relational data
------------------
---------------------------------------------
BestWishes！
小网客
Blog:http://snv.iteye.com/
Email:1134687142@qq.com
------------------ Original ------------------
From: "Jay Vee";<jv...@gmail.com>;
Date: Fri, Dec 20, 2013 04:35 AM
To: "user"<us...@hadoop.apache.org>;
Subject: from relational to bigger data
We have a large relational database ( ~ 500 GB, hundreds of tables ).
We have summary tables that we rebuild from scratch each night that takes about 10 hours.
From these summary tables, we have a web interface that accesses the summary tables to build reports.
There is a business reason for doing a complete rebuild of the summary tables each night, and using
views (as in the sense of Oracle views) is not an option at this time.
If I wanted to leverage Big Data technologies to speed up the summary table rebuild, what would be the first step into getting all data into some big data storage technology?
Ideally in the end, we want to retain the summary tables in a relational database and have reporting work the same without modifications.
It's just the crunching of the data and building these relational summary tables where we need a significant performance increase.
Re: from relational to bigger data
Posted by Chris Embree <ce...@gmail.com>.
In big data terms, 500G isn't big. But, moving that much data around
every night is not trivial either. I'm going to guess at a lot here,
but at a very high level.
1. Sqoop the data required to build the summary tables into Hadoop.
2. Crunch the summaries into new tables (really just files on Hadoop)
3. Sqoop the summarized data back out to Oracle
4. Build Indices as needed.
Depending on the size of the data being sqoop'd, this might help. It
might also take longer. A real solution would require more details
and analysis.
Chris
On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary table
> rebuild, what would be the first step into getting all data into some big
> data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
Re: from relational to bigger data
Posted by Andre Kelpe <ak...@concurrentinc.com>.
You could also give cascading lingual a try:
http://www.cascading.org/lingual/ http://docs.cascading.org/lingual/1.0/
We have a connector for oracle (
https://github.com/Cascading/cascading-jdbc#oracle), so you could read the
data from oracle do the processing on a hadoop cluster and write it back
into oracle all via SQL or a combination of SQL and Java/Cascading (
https://github.com/Cascading/cascading-jdbc#in-lingual).
- André
On Thu, Dec 19, 2013 at 9:35 PM, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary
> table rebuild, what would be the first step into getting all data into some
> big data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
>
>
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re: from relational to bigger data
Posted by Chris Embree <ce...@gmail.com>.
In big data terms, 500G isn't big. But, moving that much data around
every night is not trivial either. I'm going to guess at a lot here,
but at a very high level.
1. Sqoop the data required to build the summary tables into Hadoop.
2. Crunch the summaries into new tables (really just files on Hadoop)
3. Sqoop the summarized data back out to Oracle
4. Build Indices as needed.
Depending on the size of the data being sqoop'd, this might help. It
might also take longer. A real solution would require more details
and analysis.
Chris
On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary table
> rebuild, what would be the first step into getting all data into some big
> data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
Re: from relational to bigger data
Posted by Chris Embree <ce...@gmail.com>.
In big data terms, 500G isn't big. But, moving that much data around
every night is not trivial either. I'm going to guess at a lot here,
but at a very high level.
1. Sqoop the data required to build the summary tables into Hadoop.
2. Crunch the summaries into new tables (really just files on Hadoop)
3. Sqoop the summarized data back out to Oracle
4. Build Indices as needed.
Depending on the size of the data being sqoop'd, this might help. It
might also take longer. A real solution would require more details
and analysis.
Chris
On 12/19/13, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary table
> rebuild, what would be the first step into getting all data into some big
> data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
Re:from relational to bigger data
Posted by 小网客 <sm...@foxmail.com>.
Writing an inputformat then use mapreduce job to deal with your relational data
------------------
---------------------------------------------
BestWishes！
小网客
Blog:http://snv.iteye.com/
Email:1134687142@qq.com
------------------ Original ------------------
From: "Jay Vee";<jv...@gmail.com>;
Date: Fri, Dec 20, 2013 04:35 AM
To: "user"<us...@hadoop.apache.org>;
Subject: from relational to bigger data
We have a large relational database ( ~ 500 GB, hundreds of tables ).
We have summary tables that we rebuild from scratch each night that takes about 10 hours.
From these summary tables, we have a web interface that accesses the summary tables to build reports.
There is a business reason for doing a complete rebuild of the summary tables each night, and using
views (as in the sense of Oracle views) is not an option at this time.
If I wanted to leverage Big Data technologies to speed up the summary table rebuild, what would be the first step into getting all data into some big data storage technology?
Ideally in the end, we want to retain the summary tables in a relational database and have reporting work the same without modifications.
It's just the crunching of the data and building these relational summary tables where we need a significant performance increase.
Re: from relational to bigger data
Posted by Andre Kelpe <ak...@concurrentinc.com>.
You could also give cascading lingual a try:
http://www.cascading.org/lingual/ http://docs.cascading.org/lingual/1.0/
We have a connector for oracle (
https://github.com/Cascading/cascading-jdbc#oracle), so you could read the
data from oracle do the processing on a hadoop cluster and write it back
into oracle all via SQL or a combination of SQL and Java/Cascading (
https://github.com/Cascading/cascading-jdbc#in-lingual).
- André
On Thu, Dec 19, 2013 at 9:35 PM, Jay Vee <jv...@gmail.com> wrote:
> We have a large relational database ( ~ 500 GB, hundreds of tables ).
>
> We have summary tables that we rebuild from scratch each night that takes
> about 10 hours.
> From these summary tables, we have a web interface that accesses the
> summary tables to build reports.
>
> There is a business reason for doing a complete rebuild of the summary
> tables each night, and using
> views (as in the sense of Oracle views) is not an option at this time.
>
> If I wanted to leverage Big Data technologies to speed up the summary
> table rebuild, what would be the first step into getting all data into some
> big data storage technology?
>
> Ideally in the end, we want to retain the summary tables in a relational
> database and have reporting work the same without modifications.
>
> It's just the crunching of the data and building these relational summary
> tables where we need a significant performance increase.
>
>
>
--
André Kelpe
andre@concurrentinc.com
http://concurrentinc.com
Re:from relational to bigger data
Posted by 小网客 <sm...@foxmail.com>.
Writing an inputformat then use mapreduce job to deal with your relational data
------------------
---------------------------------------------
BestWishes！
小网客
Blog:http://snv.iteye.com/
Email:1134687142@qq.com
------------------ Original ------------------
From: "Jay Vee";<jv...@gmail.com>;
Date: Fri, Dec 20, 2013 04:35 AM
To: "user"<us...@hadoop.apache.org>;
Subject: from relational to bigger data
We have a large relational database ( ~ 500 GB, hundreds of tables ).
We have summary tables that we rebuild from scratch each night that takes about 10 hours.
From these summary tables, we have a web interface that accesses the summary tables to build reports.
There is a business reason for doing a complete rebuild of the summary tables each night, and using
views (as in the sense of Oracle views) is not an option at this time.
If I wanted to leverage Big Data technologies to speed up the summary table rebuild, what would be the first step into getting all data into some big data storage technology?
Ideally in the end, we want to retain the summary tables in a relational database and have reporting work the same without modifications.
It's just the crunching of the data and building these relational summary tables where we need a significant performance increase.