You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Jarek Jarcec Cecho <ja...@apache.org> on 2014/05/04 22:55:22 UTC
Re: Custom output format / Avro-Parquet Support
To my best knowledge the argument --as-avrodatafile is not deprecated - why do you think so Ananth?
Jarcec
On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote:
> Thanks Venkat, I gave it a try with HCatalog today. Since --as-avrodatafile
> is depricated, I'm not able generate Avro object and give it to Parquet
> Avro Output format. So the job not able to initialized.
>
> This is my Hive table
>
> create external table hcat_test (id BIGINT, name BIGINT)
> ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
> STORED AS
> INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
> OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
> LOCATION '/tmp/data/hcat_test';
>
> my sqoop command
>
> sqoop import --table test_db --username username --connect
> "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
> --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
> --hcatalog-table hcat_employee
>
>
> and I'm getting
>
> 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
> Exception in thread "main" java.lang.InstantiationError:
> org.apache.hadoop.mapreduce.JobContext
> at
> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
> at
> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
> at
> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>
>
>
>
> I suspect the absence of Avro Object causing the issue. Any suggestions pls.
>
>
>
>
>
>
>
> Regards,
> Ananth.P,
>
>
>
>
>
>
>
> On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
> > Sqoop already is integrated with HCatalog so many different storage types
> > should be possible to be used with Sqoop as long as there is a hive serde
> > for it. Check out the sqoop user guide on HCatalog integration.
> >
> >
> > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <va...@gmail.com> wrote:
> >
> >> How can I extend Sqoop client to support custom output format or provide
> >> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
> >> output format.
> >>
> >> Regards,
> >> Ananth.P,
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <va...@gmail.com>wrote:
> >>
> >>> How can I extend Sqoop client to support custom output format or provide
> >>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
> >>> output format.
> >>>
> >>> Regards,
> >>> Ananth.P,
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <va...@gmail.com>wrote:
> >>>
> >>>> How can I extend Sqoop client to support custom output format or
> >>>> provide Avro-Parquet support? Right now Sqoop support only Text and
> >>>> Sequence file output format.
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>> Ananth.P,
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> > to which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender immediately
> > and delete it from your system. Thank You.
Re: Custom output format / Avro-Parquet Support
Posted by Ananth Durai <va...@gmail.com>.
will it be added in to Sqoop road map anytime soon?
Regards,
Ananth.P,
On Mon, May 5, 2014 at 12:41 PM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:
> yes, you can't use --as-avrodatafile with --hcatalog-table option. (In
> fact you can't use --hive-import also with hcatalog options). I have
> not myself tried paraquet format with hcatalog-table option though
>
> Thanks
>
> Venkat
>
>
> On Mon, May 5, 2014 at 11:33 AM, Ananth Durai <va...@gmail.com> wrote:
>
>> Jarek, Surprisingly --as-avrodatafile is not supported for HCat export.
>> When I tried I got error like,
>>
>> HCatalog job is not compatible with AVRO format option as-avrodatafile
>> option.
>>
>> Regards,
>> Ananth.P,
>>
>>
>>
>>
>>
>>
>>
>> On Sun, May 4, 2014 at 1:55 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>>
>>> To my best knowledge the argument --as-avrodatafile is not deprecated -
>>> why do you think so Ananth?
>>>
>>> Jarcec
>>>
>>> On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote:
>>> > Thanks Venkat, I gave it a try with HCatalog today. Since
>>> --as-avrodatafile
>>> > is depricated, I'm not able generate Avro object and give it to Parquet
>>> > Avro Output format. So the job not able to initialized.
>>> >
>>> > This is my Hive table
>>> >
>>> > create external table hcat_test (id BIGINT, name BIGINT)
>>> > ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
>>> > STORED AS
>>> > INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
>>> > OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
>>> > LOCATION '/tmp/data/hcat_test';
>>> >
>>> > my sqoop command
>>> >
>>> > sqoop import --table test_db --username username --connect
>>> > "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
>>> > --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
>>> > --hcatalog-table hcat_employee
>>> >
>>> >
>>> > and I'm getting
>>> >
>>> > 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
>>> >
>>> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
>>> > Exception in thread "main" java.lang.InstantiationError:
>>> > org.apache.hadoop.mapreduce.JobContext
>>> > at
>>> >
>>> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
>>> > at
>>> >
>>> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
>>> > at
>>> >
>>> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
>>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>>> > at
>>> >
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> > at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>> > at
>>> >
>>> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
>>> > at
>>> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
>>> > at
>>> >
>>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
>>> > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
>>> > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
>>> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
>>> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>>> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>>> >
>>> >
>>> >
>>> >
>>> > I suspect the absence of Avro Object causing the issue. Any
>>> suggestions pls.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Regards,
>>> > Ananth.P,
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
>>> > vranganathan@hortonworks.com> wrote:
>>> >
>>> > > Sqoop already is integrated with HCatalog so many different storage
>>> types
>>> > > should be possible to be used with Sqoop as long as there is a hive
>>> serde
>>> > > for it. Check out the sqoop user guide on HCatalog integration.
>>> > >
>>> > >
>>> > > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <va...@gmail.com>
>>> wrote:
>>> > >
>>> > >> How can I extend Sqoop client to support custom output format or
>>> provide
>>> > >> Avro-Parquet support? Right now Sqoop support only Text and
>>> Sequence file
>>> > >> output format.
>>> > >>
>>> > >> Regards,
>>> > >> Ananth.P,
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <vananth22@gmail.com
>>> >wrote:
>>> > >>
>>> > >>> How can I extend Sqoop client to support custom output format or
>>> provide
>>> > >>> Avro-Parquet support? Right now Sqoop support only Text and
>>> Sequence file
>>> > >>> output format.
>>> > >>>
>>> > >>> Regards,
>>> > >>> Ananth.P,
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <vananth22@gmail.com
>>> >wrote:
>>> > >>>
>>> > >>>> How can I extend Sqoop client to support custom output format or
>>> > >>>> provide Avro-Parquet support? Right now Sqoop support only Text
>>> and
>>> > >>>> Sequence file output format.
>>> > >>>>
>>> > >>>>
>>> > >>>>
>>> > >>>> Regards,
>>> > >>>> Ananth.P,
>>> > >>>>
>>> > >>>>
>>> > >>>>
>>> > >>>>
>>> > >>>>
>>> > >>>>
>>> > >>>
>>> > >>
>>> > >
>>> > > CONFIDENTIALITY NOTICE
>>> > > NOTICE: This message is intended for the use of the individual or
>>> entity
>>> > > to which it is addressed and may contain information that is
>>> confidential,
>>> > > privileged and exempt from disclosure under applicable law. If the
>>> reader
>>> > > of this message is not the intended recipient, you are hereby
>>> notified that
>>> > > any printing, copying, dissemination, distribution, disclosure or
>>> > > forwarding of this communication is strictly prohibited. If you have
>>> > > received this communication in error, please contact the sender
>>> immediately
>>> > > and delete it from your system. Thank You.
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Custom output format / Avro-Parquet Support
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
yes, you can't use --as-avrodatafile with --hcatalog-table option. (In
fact you can't use --hive-import also with hcatalog options). I have
not myself tried paraquet format with hcatalog-table option though
Thanks
Venkat
On Mon, May 5, 2014 at 11:33 AM, Ananth Durai <va...@gmail.com> wrote:
> Jarek, Surprisingly --as-avrodatafile is not supported for HCat export.
> When I tried I got error like,
>
> HCatalog job is not compatible with AVRO format option as-avrodatafile
> option.
>
> Regards,
> Ananth.P,
>
>
>
>
>
>
>
> On Sun, May 4, 2014 at 1:55 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
>> To my best knowledge the argument --as-avrodatafile is not deprecated -
>> why do you think so Ananth?
>>
>> Jarcec
>>
>> On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote:
>> > Thanks Venkat, I gave it a try with HCatalog today. Since
>> --as-avrodatafile
>> > is depricated, I'm not able generate Avro object and give it to Parquet
>> > Avro Output format. So the job not able to initialized.
>> >
>> > This is my Hive table
>> >
>> > create external table hcat_test (id BIGINT, name BIGINT)
>> > ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
>> > STORED AS
>> > INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
>> > OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
>> > LOCATION '/tmp/data/hcat_test';
>> >
>> > my sqoop command
>> >
>> > sqoop import --table test_db --username username --connect
>> > "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
>> > --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
>> > --hcatalog-table hcat_employee
>> >
>> >
>> > and I'm getting
>> >
>> > 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
>> >
>> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
>> > Exception in thread "main" java.lang.InstantiationError:
>> > org.apache.hadoop.mapreduce.JobContext
>> > at
>> >
>> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
>> > at
>> >
>> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
>> > at
>> >
>> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> > at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>> > at
>> >
>> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
>> > at
>> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
>> > at
>> >
>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
>> > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
>> > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
>> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
>> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>> >
>> >
>> >
>> >
>> > I suspect the absence of Avro Object causing the issue. Any suggestions
>> pls.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Regards,
>> > Ananth.P,
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
>> > vranganathan@hortonworks.com> wrote:
>> >
>> > > Sqoop already is integrated with HCatalog so many different storage
>> types
>> > > should be possible to be used with Sqoop as long as there is a hive
>> serde
>> > > for it. Check out the sqoop user guide on HCatalog integration.
>> > >
>> > >
>> > > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <va...@gmail.com>
>> wrote:
>> > >
>> > >> How can I extend Sqoop client to support custom output format or
>> provide
>> > >> Avro-Parquet support? Right now Sqoop support only Text and Sequence
>> file
>> > >> output format.
>> > >>
>> > >> Regards,
>> > >> Ananth.P,
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <vananth22@gmail.com
>> >wrote:
>> > >>
>> > >>> How can I extend Sqoop client to support custom output format or
>> provide
>> > >>> Avro-Parquet support? Right now Sqoop support only Text and
>> Sequence file
>> > >>> output format.
>> > >>>
>> > >>> Regards,
>> > >>> Ananth.P,
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <vananth22@gmail.com
>> >wrote:
>> > >>>
>> > >>>> How can I extend Sqoop client to support custom output format or
>> > >>>> provide Avro-Parquet support? Right now Sqoop support only Text and
>> > >>>> Sequence file output format.
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> Regards,
>> > >>>> Ananth.P,
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>
>> > >>
>> > >
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > > to which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby
>> notified that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> immediately
>> > > and delete it from your system. Thank You.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Custom output format / Avro-Parquet Support
Posted by Ananth Durai <va...@gmail.com>.
Jarek, Surprisingly --as-avrodatafile is not supported for HCat export.
When I tried I got error like,
HCatalog job is not compatible with AVRO format option as-avrodatafile
option.
Regards,
Ananth.P,
On Sun, May 4, 2014 at 1:55 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> To my best knowledge the argument --as-avrodatafile is not deprecated -
> why do you think so Ananth?
>
> Jarcec
>
> On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote:
> > Thanks Venkat, I gave it a try with HCatalog today. Since
> --as-avrodatafile
> > is depricated, I'm not able generate Avro object and give it to Parquet
> > Avro Output format. So the job not able to initialized.
> >
> > This is my Hive table
> >
> > create external table hcat_test (id BIGINT, name BIGINT)
> > ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
> > STORED AS
> > INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
> > OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
> > LOCATION '/tmp/data/hcat_test';
> >
> > my sqoop command
> >
> > sqoop import --table test_db --username username --connect
> > "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
> > --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
> > --hcatalog-table hcat_employee
> >
> >
> > and I'm getting
> >
> > 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
> >
> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
> > Exception in thread "main" java.lang.InstantiationError:
> > org.apache.hadoop.mapreduce.JobContext
> > at
> >
> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
> > at
> >
> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
> > at
> >
> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
> > at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> > at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> > at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
> > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> >
> >
> >
> >
> > I suspect the absence of Avro Object causing the issue. Any suggestions
> pls.
> >
> >
> >
> >
> >
> >
> >
> > Regards,
> > Ananth.P,
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
> > vranganathan@hortonworks.com> wrote:
> >
> > > Sqoop already is integrated with HCatalog so many different storage
> types
> > > should be possible to be used with Sqoop as long as there is a hive
> serde
> > > for it. Check out the sqoop user guide on HCatalog integration.
> > >
> > >
> > > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <va...@gmail.com>
> wrote:
> > >
> > >> How can I extend Sqoop client to support custom output format or
> provide
> > >> Avro-Parquet support? Right now Sqoop support only Text and Sequence
> file
> > >> output format.
> > >>
> > >> Regards,
> > >> Ananth.P,
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <vananth22@gmail.com
> >wrote:
> > >>
> > >>> How can I extend Sqoop client to support custom output format or
> provide
> > >>> Avro-Parquet support? Right now Sqoop support only Text and Sequence
> file
> > >>> output format.
> > >>>
> > >>> Regards,
> > >>> Ananth.P,
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <vananth22@gmail.com
> >wrote:
> > >>>
> > >>>> How can I extend Sqoop client to support custom output format or
> > >>>> provide Avro-Parquet support? Right now Sqoop support only Text and
> > >>>> Sequence file output format.
> > >>>>
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>> Ananth.P,
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > > to which it is addressed and may contain information that is
> confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> immediately
> > > and delete it from your system. Thank You.
>