You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Veena Basavaraj <vb...@cloudera.com> on 2014/11/03 23:31:56 UTC
Review Request 27543: DOC: Connector SDK docs + validation to be
updated
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/
-----------------------------------------------------------
Review request for Sqoop.
Repository: sqoop-sqoop2
Description
-------
I have taken liberty to write some stuff, happy to amend it!
Diffs
-----
docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
Diff: https://reviews.apache.org/r/27543/diff/
Testing
-------
Thanks,
Veena Basavaraj
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Jarek Cecho <ja...@apache.org>.
> On Nov. 4, 2014, 3:08 a.m., Jarek Cecho wrote:
> > docs/src/site/sphinx/ConnectorDevelopment.rst, line 129
> > <https://reviews.apache.org/r/27543/diff/2/?file=748012#file748012line129>
> >
> > We should not be talking about a Mapreduce job here - that is specific to the execution engine. I would say that initializer is executed first before any other action.
>
> Veena Basavaraj wrote:
> agree. you are ok wit removing it, it existed before.
Yes, I'm OK with removing it.
> On Nov. 4, 2014, 3:08 a.m., Jarek Cecho wrote:
> > docs/src/site/sphinx/ConnectorDevelopment.rst, line 165
> > <https://reviews.apache.org/r/27543/diff/2/?file=748012#file748012line165>
> >
> > Let's not confuse the connector developer with Driver here. Let's just state that they are suppose to create 0..N partitions where N is driven by user.
> >
> > Similarly I don't think that we should talk about mappers and reducers here as that is a specific to one single execution engine (albeit the only one right now). Perhaps we can add some explanation block to say "hey in MR exec engine it's implemented this way" as an example?
>
> Veena Basavaraj wrote:
> sure I will use the example patterm. Since at tne end of this doc, we clearly show how MR is used
Yeah and that portion seems to be useful as an example :)
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/#review59718
-----------------------------------------------------------
On Nov. 4, 2014, 3:07 a.m., Veena Basavaraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27543/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2014, 3:07 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I have taken liberty to write some stuff, happy to amend it!
>
>
> I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
>
>
> Diffs
> -----
>
> docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
>
> Diff: https://reviews.apache.org/r/27543/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Veena Basavaraj
>
>
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Veena Basavaraj <vb...@cloudera.com>.
> On Nov. 3, 2014, 7:08 p.m., Jarek Cecho wrote:
> > docs/src/site/sphinx/ConnectorDevelopment.rst, line 129
> > <https://reviews.apache.org/r/27543/diff/2/?file=748012#file748012line129>
> >
> > We should not be talking about a Mapreduce job here - that is specific to the execution engine. I would say that initializer is executed first before any other action.
agree. you are ok wit removing it, it existed before.
> On Nov. 3, 2014, 7:08 p.m., Jarek Cecho wrote:
> > docs/src/site/sphinx/ConnectorDevelopment.rst, line 149
> > <https://reviews.apache.org/r/27543/diff/2/?file=748012#file748012line149>
> >
> > I believe that we do required empty Schema right? Null can actually throw a NullPointerException at some places (not 100% sure after recent code changes).
we have a ticket that addressed this, let me see if I patched it for 1.99.4
> On Nov. 3, 2014, 7:08 p.m., Jarek Cecho wrote:
> > docs/src/site/sphinx/ConnectorDevelopment.rst, line 165
> > <https://reviews.apache.org/r/27543/diff/2/?file=748012#file748012line165>
> >
> > Let's not confuse the connector developer with Driver here. Let's just state that they are suppose to create 0..N partitions where N is driven by user.
> >
> > Similarly I don't think that we should talk about mappers and reducers here as that is a specific to one single execution engine (albeit the only one right now). Perhaps we can add some explanation block to say "hey in MR exec engine it's implemented this way" as an example?
sure I will use the example patterm. Since at tne end of this doc, we clearly show how MR is used
- Veena
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/#review59718
-----------------------------------------------------------
On Nov. 3, 2014, 7:07 p.m., Veena Basavaraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27543/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2014, 7:07 p.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I have taken liberty to write some stuff, happy to amend it!
>
>
> I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
>
>
> Diffs
> -----
>
> docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
>
> Diff: https://reviews.apache.org/r/27543/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Veena Basavaraj
>
>
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/#review59718
-----------------------------------------------------------
Good material Veena! I have couple of notes below:
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101023>
I think that we should mention here that "from" and "to" job can have different inputs.
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101025>
We should not be talking about a Mapreduce job here - that is specific to the execution engine. I would say that initializer is executed first before any other action.
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101024>
I believe that we do required empty Schema right? Null can actually throw a NullPointerException at some places (not 100% sure after recent code changes).
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101026>
Let's not confuse the connector developer with Driver here. Let's just state that they are suppose to create 0..N partitions where N is driven by user.
Similarly I don't think that we should talk about mappers and reducers here as that is a specific to one single execution engine (albeit the only one right now). Perhaps we can add some explanation block to say "hey in MR exec engine it's implemented this way" as an example?
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101027>
... must implement...
Might be better wording (nit).
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101028>
The extractor needs to iterate through the entire partition, not data set. All partitions together will give entire dataset.
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101029>
Nit: Typo in the words "developers"
Also the loader technically runs in the output committer and not on the reducer side. That is important because we can run map-only job and still run the loader.
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101030>
Two @ signs in teh ConfigClass
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101031>
Nit: Typo in "Validatora"
docs/src/site/sphinx/ConnectorDevelopment.rst
<https://reviews.apache.org/r/27543/#comment101033>
Typo "wiork"
- Jarek Cecho
On Nov. 4, 2014, 3:07 a.m., Veena Basavaraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27543/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2014, 3:07 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I have taken liberty to write some stuff, happy to amend it!
>
>
> I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
>
>
> Diffs
> -----
>
> docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
>
> Diff: https://reviews.apache.org/r/27543/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Veena Basavaraj
>
>
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/#review59735
-----------------------------------------------------------
Ship it!
Please attach latest patch to the JIRA and I'll commit it.
- Jarek Cecho
On Nov. 4, 2014, 4:54 a.m., Veena Basavaraj wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27543/
> -----------------------------------------------------------
>
> (Updated Nov. 4, 2014, 4:54 a.m.)
>
>
> Review request for Sqoop.
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> I have taken liberty to write some stuff, happy to amend it!
>
>
> I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
>
>
> Diffs
> -----
>
> docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
>
> Diff: https://reviews.apache.org/r/27543/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Veena Basavaraj
>
>
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/
-----------------------------------------------------------
(Updated Nov. 3, 2014, 8:54 p.m.)
Review request for Sqoop.
Changes
-------
fixed WS
Repository: sqoop-sqoop2
Description
-------
I have taken liberty to write some stuff, happy to amend it!
I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
Diffs (updated)
-----
docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
Diff: https://reviews.apache.org/r/27543/diff/
Testing
-------
Thanks,
Veena Basavaraj
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/
-----------------------------------------------------------
(Updated Nov. 3, 2014, 8:47 p.m.)
Review request for Sqoop.
Repository: sqoop-sqoop2
Description
-------
I have taken liberty to write some stuff, happy to amend it!
I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
Diffs (updated)
-----
docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
Diff: https://reviews.apache.org/r/27543/diff/
Testing
-------
Thanks,
Veena Basavaraj
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/
-----------------------------------------------------------
(Updated Nov. 3, 2014, 7:07 p.m.)
Review request for Sqoop.
Changes
-------
added example of the empty configuration and the sample validator classes we already provide
Repository: sqoop-sqoop2
Description
-------
I have taken liberty to write some stuff, happy to amend it!
I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
Diffs (updated)
-----
docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
Diff: https://reviews.apache.org/r/27543/diff/
Testing
-------
Thanks,
Veena Basavaraj
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/
-----------------------------------------------------------
(Updated Nov. 3, 2014, 6:44 p.m.)
Review request for Sqoop.
Repository: sqoop-sqoop2
Description
-------
I have taken liberty to write some stuff, happy to amend it!
I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
Diffs (updated)
-----
docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
Diff: https://reviews.apache.org/r/27543/diff/
Testing
-------
Thanks,
Veena Basavaraj
Re: Review Request 27543: DOC: Connector SDK docs + validation to be
updated
Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27543/
-----------------------------------------------------------
(Updated Nov. 3, 2014, 2:44 p.m.)
Review request for Sqoop.
Repository: sqoop-sqoop2
Description (updated)
-------
I have taken liberty to write some stuff, happy to amend it!
I will remove the WS once the cotnents are approved. Typing some of this at the qcon:)
Diffs
-----
docs/src/site/sphinx/ConnectorDevelopment.rst e4b5402
Diff: https://reviews.apache.org/r/27543/diff/
Testing
-------
Thanks,
Veena Basavaraj