Storing data into hive table having columns from multiple files

[va...@polarisFT.com]

Hi All,

I have a query on a scenario which is below:

There are 2 files "file1", "file2". And a hive table "test" which is created with 15 columns with "," delimiter.
"file1" contains 25 columns and "file2" contains 15 columns having seperator as ",":
 
But, out of 25 columns, 10 columns of data from file1 are inserted to hive table "test"
and out of 15 columns, 5 columns of data from file2 are inserted to the same hive table "test"
(lets forget about the remaining 15 columns from file1 and 10 columns from file2)
 
First, "file1" is processed and then "file2" is processed.
When "file1" is processed, 10 columns gets inserted into "test" table for a row.
When "file2" is processed, 5 columns have to be inserted into "test" table of the same row.
 
Could you please help me on how to insert the 5 columns of file2 to the same row in "test" table?
(which has 10 columns of data inserted from file1 for that row)


Thanks & Regards,
Varaprasada Reddy


This e-Mail may contain proprietary and confidential information and is sent for the intended recipient(s) only.  If by an addressing or transmission error this mail has been misdirected to you, you are requested to delete this mail immediately. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message, contents or its attachment other than by its intended recipient/s is strictly prohibited.

Visit us at http://www.polarisFT.com

Re: Storing data into hive table having columns from multiple files

[pandeeswaran bhoopathy <pa...@me.com>]

Do you mean the merge operation?

Sent from my iPad

On 20-Jun-2013, at 2:51 PM, varaprasad.bheri@polarisFT.com wrote:

> Hi All,
> 
> I have a query on a scenario which is below:
> 
> There are 2 files "file1", "file2". And a hive table "test" which is created with 15 columns with "," delimiter.
> "file1" contains 25 columns and "file2" contains 15 columns having seperator as ",":
> 
> But, out of 25 columns, 10 columns of data from file1 are inserted to hive table "test"
> and out of 15 columns, 5 columns of data from file2 are inserted to the same hive table "test"
> (lets forget about the remaining 15 columns from file1 and 10 columns from file2)
> 
> First, "file1" is processed and then "file2" is processed.
> When "file1" is processed, 10 columns gets inserted into "test" table for a row.
> When "file2" is processed, 5 columns have to be inserted into "test" table of the same row.
> 
> Could you please help me on how to insert the 5 columns of file2 to the same row in "test" table?
> (which has 10 columns of data inserted from file1 for that row)
> 
> 
> Thanks & Regards,
> Varaprasada Reddy
> 
> 
> This e-Mail may contain proprietary and confidential information and is sent for the intended recipient(s) only.  If by an addressing or transmission error this mail has been misdirected to you, you are requested to delete this mail immediately. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail message, contents or its attachment other than by its intended recipient/s is strictly prohibited.
> 
> Visit us at http://www.polarisFT.com