You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Kunal Sachdeva <ku...@gmail.com> on 2013/10/09 15:01:22 UTC
How to invoke extractor during IMPORT Job Submition process- Sqoop
version 1.99.1 Connector
Hi,
*Subject: Want to invoke custom extractor class run method for printing
data- MySql table's columns values data in Import job*.
I am newby to Sqoop2 (*Currently working on connector version 1.99.1*) and
want some help from you. I have developed Sqoop2 connector similar to
generic connector 1.99.1 provided by apache, here
http://sqoop.apache.org/docs/1.99.1/
.
In other words my connector is 99% similar to generic connector 1.99.1 but
have different name for packages and classes.I have tested with my own
connector Import and Export both are working fine. But I want to print
data during Import job submission and before it writes data to
given/specific file, I mean when we import MySql Database table and submit
the Import job then whatever data Sqoop2 is importing from that table, I
want all rows,columns to be appended with my name or some other value 'abc'
through connector.
I know, Each connector in Sqoop2 extends class SqoopConnector by providing
classes that should be called. If I need to run custom extractor, I will
need to create custom connector that will have different extractor. On the
client side, I will have to create connection for your new connector
instead of the build-in Generic JDBC Connector.
I have already done these above steps and My connector is working fine for
both Import,Export jobs and I have seen result in desired location.
But, as said in Import job, to append anything in table rows data before
writing them to any file , we have to code in extractor. My extractor is
100% same as generic connector extractor. But when and how to invoke this
custom conector extractor is the main thing which I do not know and I want
to invoke this.
Please help me which class and method I need to add/modify so that I can
get invoke my connector extractor.
I have followed all steps mentioned here
http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard thing
for me. I do not know which value I have to give in command prompt and all.
Thanks in advance.
--
Regards,
Kunal
Re: How to invoke extractor during IMPORT Job Submition process-
Sqoop version 1.99.1 Connector
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Kunal,
the extractor is connector driver rather then user given. As a result you can't specify which extractor will be used as this is specified by the connector. You will have to create your own connector if you need to call custom extractor.
Jarcec
On Thu, Oct 17, 2013 at 03:14:16PM +0530, K Singh wrote:
> Hi ,
>
> Thanks for clarification.
>
> I have already followed below URL to make my own generic connector :
>
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
>
>
> But the requirement is, once I have my own connector which is working fine,
> I want to invoke Extractor during IMPORT job. The built in generic
> connector also not calling Extractor. So Is there any way to invoke
> extractor or how and when Extractor is being called?
>
> In simple words, If we already developed/built generic connector and
> working fine then How can Import job submission/I call or invoke extractor
> class ( e.q. GenericJdbcImportExtractor.class,) during IMPORT process. I
> want to just call this class default method.
>
> Please guide me for this. Is there any additional command line arguments or
> something like which we can use to invoke/call extractor because by
> default it is not being called in IMPORT job. Like we have for validation:
>
> *Validation arguments More
> Details<http://archive.cloudera.com/cdh4/cdh/4/sqoop-1.4.3-cdh4.4.0/SqoopUserGuide.html#validation>
> *
> Argument Description --validate Enable validation of data copied, supports
> single table copy only. --validator <class-name> Specify validator class to
> use. --validation-threshold <class-name> Specify validation threshold class
> to use. +--validation-failurehandler <class-name >+ Specify validation
> failure handler class to use.
>
>
> So is there anything for extractor as well?
>
> Thanks.
>
>
>
>
> On Fri, Oct 11, 2013 at 9:13 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
>
> > Hi Kunal,
> > Sqoop2 is currently under heavy development, so we do not have a "How to
> > create connector" guide available at the moment. We will definitely have
> > one once the interfaces will settle down.
> >
> > For the time being, I'm afraid that you will need to follow existing code.
> > If you need to call custom import extractor in your new connector, you will
> > need to specify it in the Importer structure that is exposed by your
> > connector. Please take a look on the Generic JDBC Connector example:
> >
> >
> > https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
> >
> > Jarcec
> >
> > On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> > > Hi,
> > >
> > > *Subject: Want to invoke custom extractor class run method for printing
> > > data- MySql table's columns values data in Import job*.
> > >
> > > I am newby to Sqoop2 (*Currently working on connector version 1.99.1*)
> > and
> > > want some help from you. I have developed Sqoop2 connector similar to
> > > generic connector 1.99.1 provided by apache, here
> > > http://sqoop.apache.org/docs/1.99.1/
> > > .
> > >
> > > In other words my connector is 99% similar to generic connector 1.99.1
> > but
> > > have different name for packages and classes.I have tested with my own
> > > connector Import and Export both are working fine. But I want to print
> > > data during Import job submission and before it writes data to
> > > given/specific file, I mean when we import MySql Database table and
> > submit
> > > the Import job then whatever data Sqoop2 is importing from that table, I
> > > want all rows,columns to be appended with my name or some other value
> > 'abc'
> > > through connector.
> > >
> > > I know, Each connector in Sqoop2 extends class SqoopConnector by
> > providing
> > > classes that should be called. If I need to run custom extractor, I will
> > > need to create custom connector that will have different extractor. On
> > the
> > > client side, I will have to create connection for your new connector
> > > instead of the build-in Generic JDBC Connector.
> > >
> > > I have already done these above steps and My connector is working fine
> > for
> > > both Import,Export jobs and I have seen result in desired location.
> > >
> > > But, as said in Import job, to append anything in table rows data before
> > > writing them to any file , we have to code in extractor. My extractor is
> > > 100% same as generic connector extractor. But when and how to invoke this
> > > custom conector extractor is the main thing which I do not know and I
> > want
> > > to invoke this.
> > >
> > > Please help me which class and method I need to add/modify so that I can
> > > get invoke my connector extractor.
> > >
> > > I have followed all steps mentioned here
> > > http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard
> > thing
> > > for me. I do not know which value I have to give in command prompt and
> > all.
> > >
> > > Thanks in advance.
> > >
> > > --
> > > Regards,
> > > Kunal
> >
Re: How to invoke extractor during IMPORT Job Submition process-
Sqoop version 1.99.1 Connector
Posted by K Singh <k....@gmail.com>.
Hi ,
Thanks for clarification.
I have already followed below URL to make my own generic connector :
https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
But the requirement is, once I have my own connector which is working fine,
I want to invoke Extractor during IMPORT job. The built in generic
connector also not calling Extractor. So Is there any way to invoke
extractor or how and when Extractor is being called?
In simple words, If we already developed/built generic connector and
working fine then How can Import job submission/I call or invoke extractor
class ( e.q. GenericJdbcImportExtractor.class,) during IMPORT process. I
want to just call this class default method.
Please guide me for this. Is there any additional command line arguments or
something like which we can use to invoke/call extractor because by
default it is not being called in IMPORT job. Like we have for validation:
*Validation arguments More
Details<http://archive.cloudera.com/cdh4/cdh/4/sqoop-1.4.3-cdh4.4.0/SqoopUserGuide.html#validation>
*
Argument Description --validate Enable validation of data copied, supports
single table copy only. --validator <class-name> Specify validator class to
use. --validation-threshold <class-name> Specify validation threshold class
to use. +--validation-failurehandler <class-name >+ Specify validation
failure handler class to use.
So is there anything for extractor as well?
Thanks.
On Fri, Oct 11, 2013 at 9:13 PM, Jarek Jarcec Cecho <ja...@apache.org>wrote:
> Hi Kunal,
> Sqoop2 is currently under heavy development, so we do not have a "How to
> create connector" guide available at the moment. We will definitely have
> one once the interfaces will settle down.
>
> For the time being, I'm afraid that you will need to follow existing code.
> If you need to call custom import extractor in your new connector, you will
> need to specify it in the Importer structure that is exposed by your
> connector. Please take a look on the Generic JDBC Connector example:
>
>
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
>
> Jarcec
>
> On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> > Hi,
> >
> > *Subject: Want to invoke custom extractor class run method for printing
> > data- MySql table's columns values data in Import job*.
> >
> > I am newby to Sqoop2 (*Currently working on connector version 1.99.1*)
> and
> > want some help from you. I have developed Sqoop2 connector similar to
> > generic connector 1.99.1 provided by apache, here
> > http://sqoop.apache.org/docs/1.99.1/
> > .
> >
> > In other words my connector is 99% similar to generic connector 1.99.1
> but
> > have different name for packages and classes.I have tested with my own
> > connector Import and Export both are working fine. But I want to print
> > data during Import job submission and before it writes data to
> > given/specific file, I mean when we import MySql Database table and
> submit
> > the Import job then whatever data Sqoop2 is importing from that table, I
> > want all rows,columns to be appended with my name or some other value
> 'abc'
> > through connector.
> >
> > I know, Each connector in Sqoop2 extends class SqoopConnector by
> providing
> > classes that should be called. If I need to run custom extractor, I will
> > need to create custom connector that will have different extractor. On
> the
> > client side, I will have to create connection for your new connector
> > instead of the build-in Generic JDBC Connector.
> >
> > I have already done these above steps and My connector is working fine
> for
> > both Import,Export jobs and I have seen result in desired location.
> >
> > But, as said in Import job, to append anything in table rows data before
> > writing them to any file , we have to code in extractor. My extractor is
> > 100% same as generic connector extractor. But when and how to invoke this
> > custom conector extractor is the main thing which I do not know and I
> want
> > to invoke this.
> >
> > Please help me which class and method I need to add/modify so that I can
> > get invoke my connector extractor.
> >
> > I have followed all steps mentioned here
> > http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard
> thing
> > for me. I do not know which value I have to give in command prompt and
> all.
> >
> > Thanks in advance.
> >
> > --
> > Regards,
> > Kunal
>
Re: How to invoke extractor during IMPORT Job Submition process-
Sqoop version 1.99.1 Connector
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hi Kunal,
Sqoop2 is currently under heavy development, so we do not have a "How to create connector" guide available at the moment. We will definitely have one once the interfaces will settle down.
For the time being, I'm afraid that you will need to follow existing code. If you need to call custom import extractor in your new connector, you will need to specify it in the Importer structure that is exposed by your connector. Please take a look on the Generic JDBC Connector example:
https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
Jarcec
On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> Hi,
>
> *Subject: Want to invoke custom extractor class run method for printing
> data- MySql table's columns values data in Import job*.
>
> I am newby to Sqoop2 (*Currently working on connector version 1.99.1*) and
> want some help from you. I have developed Sqoop2 connector similar to
> generic connector 1.99.1 provided by apache, here
> http://sqoop.apache.org/docs/1.99.1/
> .
>
> In other words my connector is 99% similar to generic connector 1.99.1 but
> have different name for packages and classes.I have tested with my own
> connector Import and Export both are working fine. But I want to print
> data during Import job submission and before it writes data to
> given/specific file, I mean when we import MySql Database table and submit
> the Import job then whatever data Sqoop2 is importing from that table, I
> want all rows,columns to be appended with my name or some other value 'abc'
> through connector.
>
> I know, Each connector in Sqoop2 extends class SqoopConnector by providing
> classes that should be called. If I need to run custom extractor, I will
> need to create custom connector that will have different extractor. On the
> client side, I will have to create connection for your new connector
> instead of the build-in Generic JDBC Connector.
>
> I have already done these above steps and My connector is working fine for
> both Import,Export jobs and I have seen result in desired location.
>
> But, as said in Import job, to append anything in table rows data before
> writing them to any file , we have to code in extractor. My extractor is
> 100% same as generic connector extractor. But when and how to invoke this
> custom conector extractor is the main thing which I do not know and I want
> to invoke this.
>
> Please help me which class and method I need to add/modify so that I can
> get invoke my connector extractor.
>
> I have followed all steps mentioned here
> http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard thing
> for me. I do not know which value I have to give in command prompt and all.
>
> Thanks in advance.
>
> --
> Regards,
> Kunal