You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Qian Xu <qi...@intel.com> on 2015/01/13 16:14:17 UTC

Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/
-----------------------------------------------------------

Review request for Sqoop.


Bugs: SQOOP-1988
    https://issues.apache.org/jira/browse/SQOOP-1988


Repository: sqoop-sqoop2


Description
-------

see jira


Diffs
-----

  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 

Diff: https://reviews.apache.org/r/29849/diff/


Testing
-------

Existing tests passed 


Thanks,

Qian Xu


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 13, 2015, 7:47 a.m., Veena Basavaraj wrote:
> > Another place the comment needs to be modfied
> > 
> > /**
> >    *
> >    * @param fields
> >    * @return Return the data in "fields" converted from matching the fromSchema to matching the toSchema.
> >    * Right not "converted" means re-ordering if needed and handling nulls.
> >    */
> >   abstract public Object[] getMatchingData(Object[] fields);
> > 
> > 
> > Please search all places in code, I am surprised there are so many places and I keep missing it
> 
> Qian Xu wrote:
>     I removed the line is because the file is currently modified and IntelliJ warns the corrupt docstring. I cannot find any other similar issue in modified files. If you think the do-me-a-favor change is not relevant to the patch, I can revert it. 
>     
>     We have discussed some kind of things before. A static style check tool in maven will help. Please feel free to create a ticket.

we should remove the comment that says "handling nulls", that is what i referred to


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67863
-----------------------------------------------------------


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.

> On Jan. 13, 2015, 11:47 p.m., Veena Basavaraj wrote:
> > Another place the comment needs to be modfied
> > 
> > /**
> >    *
> >    * @param fields
> >    * @return Return the data in "fields" converted from matching the fromSchema to matching the toSchema.
> >    * Right not "converted" means re-ordering if needed and handling nulls.
> >    */
> >   abstract public Object[] getMatchingData(Object[] fields);
> > 
> > 
> > Please search all places in code, I am surprised there are so many places and I keep missing it

I removed the line is because the file is currently modified and IntelliJ warns the corrupt docstring. I cannot find any other similar issue in modified files. If you think the do-me-a-favor change is not relevant to the patch, I can revert it. 

We have discussed some kind of things before. A static style check tool in maven will help. Please feel free to create a ticket.


- Qian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67863
-----------------------------------------------------------


On Jan. 13, 2015, 11:14 p.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 11:14 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67863
-----------------------------------------------------------


Another place the comment needs to be modfied

/**
   *
   * @param fields
   * @return Return the data in "fields" converted from matching the fromSchema to matching the toSchema.
   * Right not "converted" means re-ordering if needed and handling nulls.
   */
  abstract public Object[] getMatchingData(Object[] fields);


Please search all places in code, I am surprised there are so many places and I keep missing it

- Veena Basavaraj


On Jan. 13, 2015, 7:14 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 7:14 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 13, 2015, 7:45 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java, line 47
> > <https://reviews.apache.org/r/29849/diff/1/?file=818950#file818950line47>
> >
> >     fix the comment in LocationMatcher
> >     
> >     
> >     /**
> >      * Convert data according to FROM schema to data according to TO schema. This is
> >      * done based on column location, Data in first column in FROM goes into first
> >      * column in TO, etc., if TO schema has more fields and they are "nullable",
> >      * their values will be set to null. If TO schema has extra non-null fields, we
> >      * will throw an exception.
> >      */
> 
> Qian Xu wrote:
>     No. See `tryFillNullInArrayForUnexpectedColumn()`

got it. for those extra fields


> On Jan. 13, 2015, 7:45 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java, line 47
> > <https://reviews.apache.org/r/29849/diff/1/?file=818951#file818951line47>
> >
> >     Overall I dont understand why even from schema is maintained in the Matcher since it never even used,
> >     
> >     Another ticket to clean up this. 
> >     
> >     At the end of the day what is important is to schema so we can write the data adapting to it in one of the many ways we want to do matching.
> 
> Qian Xu wrote:
>     FROM schema is required by name matching.

can you point me to the code where it is using it,


> On Jan. 13, 2015, 7:45 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java, line 16
> > <https://reviews.apache.org/r/29849/diff/1/?file=818952#file818952line16>
> >
> >     /**
> >      * Convert data according to FROM schema to data according to TO schema. This is
> >      * done based on column name, If TO schema has more fields and they are
> >      * "nullable", their values will be set to null. If TO schema has extra non-null
> >      * fields, we will throw an exception.
> >      */
> >      *
> >      
> >      
> >      fix the comment since null adding is now been removed
> 
> Qian Xu wrote:
>     No. See `tryFillNullInArrayForUnexpectedColumn()`

unrelated question, what if From schema has more columns than To schema? What do they do, this logic seems half baked to me.


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67859
-----------------------------------------------------------


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 13, 2015, 7:45 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java, line 47
> > <https://reviews.apache.org/r/29849/diff/1/?file=818951#file818951line47>
> >
> >     Overall I dont understand why even from schema is maintained in the Matcher since it never even used,
> >     
> >     Another ticket to clean up this. 
> >     
> >     At the end of the day what is important is to schema so we can write the data adapting to it in one of the many ways we want to do matching.
> 
> Qian Xu wrote:
>     FROM schema is required by name matching.
> 
> Veena Basavaraj wrote:
>     can you point me to the code where it is using it,

nm i found it, its here, we already do this in the schema now since we need it for avro and JSON, so another place to clean up this code.


 for (Column fromCol : getFromSchema().getColumnsList()) {
      fromColNameIndexMap.put(fromCol.getName(), fromIndex);
      fromIndex++;
    }


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67859
-----------------------------------------------------------


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.

> On Jan. 13, 2015, 11:45 p.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java, line 47
> > <https://reviews.apache.org/r/29849/diff/1/?file=818950#file818950line47>
> >
> >     fix the comment in LocationMatcher
> >     
> >     
> >     /**
> >      * Convert data according to FROM schema to data according to TO schema. This is
> >      * done based on column location, Data in first column in FROM goes into first
> >      * column in TO, etc., if TO schema has more fields and they are "nullable",
> >      * their values will be set to null. If TO schema has extra non-null fields, we
> >      * will throw an exception.
> >      */

No. See `tryFillNullInArrayForUnexpectedColumn()`


> On Jan. 13, 2015, 11:45 p.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java, line 16
> > <https://reviews.apache.org/r/29849/diff/1/?file=818952#file818952line16>
> >
> >     /**
> >      * Convert data according to FROM schema to data according to TO schema. This is
> >      * done based on column name, If TO schema has more fields and they are
> >      * "nullable", their values will be set to null. If TO schema has extra non-null
> >      * fields, we will throw an exception.
> >      */
> >      *
> >      
> >      
> >      fix the comment since null adding is now been removed

No. See `tryFillNullInArrayForUnexpectedColumn()`


- Qian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67859
-----------------------------------------------------------


On Jan. 13, 2015, 11:14 p.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 11:14 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.

> On Jan. 13, 2015, 11:45 p.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java, line 47
> > <https://reviews.apache.org/r/29849/diff/1/?file=818951#file818951line47>
> >
> >     Overall I dont understand why even from schema is maintained in the Matcher since it never even used,
> >     
> >     Another ticket to clean up this. 
> >     
> >     At the end of the day what is important is to schema so we can write the data adapting to it in one of the many ways we want to do matching.

FROM schema is required by name matching.


- Qian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67859
-----------------------------------------------------------


On Jan. 13, 2015, 11:14 p.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 11:14 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67859
-----------------------------------------------------------



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java
<https://reviews.apache.org/r/29849/#comment111951>

    fix the comment in LocationMatcher
    
    /**
     * Convert data according to FROM schema to data according to TO schema. This is
     * done based on column location, Data in first column in FROM goes into first
     * column in TO, etc., if TO schema has more fields and they are "nullable",
     * their values will be set to null. If TO schema has extra non-null fields, we
     * will throw an exception.
     */



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java
<https://reviews.apache.org/r/29849/#comment111955>

    Overall I dont understand why even from schema is maintained in the Matcher since it never even used,
    
    Another ticket to clean up this. 
    
    At the end of the day what is important is to schema so we can write the data adapting to it in one of the many ways we want to do matching.



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java
<https://reviews.apache.org/r/29849/#comment111953>

    /**
     * Convert data according to FROM schema to data according to TO schema. This is
     * done based on column name, If TO schema has more fields and they are
     * "nullable", their values will be set to null. If TO schema has extra non-null
     * fields, we will throw an exception.
     */
     *
     
     
     fix the comment since null adding is now been removed


- Veena Basavaraj


On Jan. 13, 2015, 7:14 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 7:14 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review69358
-----------------------------------------------------------

Ship it!


Ship It!

- Abraham Elmahrek


On Jan. 23, 2015, 6:28 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 23, 2015, 6:28 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java 10772d9 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 899f734 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 365cb87 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java f23694d 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 742a35c 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 22, 2015, 10:32 p.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java, line 63
> > <https://reviews.apache.org/r/29849/diff/4/?file=831586#file831586line63>
> >
> >     can we  remove fromColNameIndexMap  variable
> >     
> >     getFromSchema().getColumnNameIndex(toCol.getName()))

its ok we can remoe this another rb


    fromColNameIndexMap = new HashMap<String, Integer>();
    int fromIndex = 0;

    for (Column fromCol : getFromSchema().getColumnsList()) {
      fromColNameIndexMap.put(fromCol.getName(), fromIndex);
      fromIndex++;
    }


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review69353
-----------------------------------------------------------


On Jan. 22, 2015, 10:28 p.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 22, 2015, 10:28 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java 10772d9 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 899f734 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 365cb87 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java f23694d 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 742a35c 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review69353
-----------------------------------------------------------

Ship it!



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java
<https://reviews.apache.org/r/29849/#comment114031>

    can we  remove fromColNameIndexMap  variable
    
    getFromSchema().getColumnNameIndex(toCol.getName()))


- Veena Basavaraj


On Jan. 22, 2015, 10:28 p.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 22, 2015, 10:28 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java 10772d9 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 899f734 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 365cb87 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java f23694d 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 742a35c 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/
-----------------------------------------------------------

(Updated Jan. 23, 2015, 2:28 p.m.)


Review request for Sqoop.


Bugs: SQOOP-1988
    https://issues.apache.org/jira/browse/SQOOP-1988


Repository: sqoop-sqoop2


Description
-------

see jira


Diffs (updated)
-----

  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java 10772d9 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 899f734 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 365cb87 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java f23694d 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 742a35c 

Diff: https://reviews.apache.org/r/29849/diff/


Testing
-------

Existing tests passed 


Thanks,

Qian Xu


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67875
-----------------------------------------------------------



connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java
<https://reviews.apache.org/r/29849/#comment111967>

    you mean From Schema? since the comment is saying from schema is not specified.


- Veena Basavaraj


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 13, 2015, 8:34 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java, line 24
> > <https://reviews.apache.org/r/29849/diff/3/?file=819035#file819035line24>
> >
> >     pleae clean up 0000 if it is not used
> >     
> >     I am not sure why we need the 0001 checks since MatcherFactory instantiates these matchers after doing the right check
> 
> Veena Basavaraj wrote:
>     Let me add some more details on why I say the 0001 check is not required.
>     see the code in Matcher constructor, it is really confusing to have some code in MatcherFactroy and some code in Matcher Constructor, so if both are empty, then it is always set to some bytearray, so there is no need to do the extra check, it would have been nice if all this logic was in one place in MatcherFactroy.
>     
>         if (fromSchema.isEmpty() && toSchema.isEmpty()) {
>           this.fromSchema = ByteArraySchema.getInstance();
>           this.toSchema = ByteArraySchema.getInstance();
> 
> Qian Xu wrote:
>     I really appreciate your carefulness. I also tried to understand the existing code. I imagine there are three cases: (1) both sides have a schema (jdbc to jdbc), so a NameMatcher is required (2) only one side has schema (jdbc to kite), so we need direct data copy (3) the case uses ByteArraySchema, i dont know. 
>     
>     The Matcher class has been changed recently. I need some time to refresh the patch with some bold changes next week.

yes there are adhoc rules at this point, that if both are empty in case of hdfs-kafka, say they set ot byte array.

other cases, the from and to are always set to the same, so there is never a case where a schema can be null. but double check if there is possibility of empty, I guess that can be the case when someone gives NullSchema, i.e no fields at all.


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67879
-----------------------------------------------------------


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.

> On Jan. 14, 2015, 12:34 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java, line 24
> > <https://reviews.apache.org/r/29849/diff/3/?file=819035#file819035line24>
> >
> >     pleae clean up 0000 if it is not used
> >     
> >     I am not sure why we need the 0001 checks since MatcherFactory instantiates these matchers after doing the right check
> 
> Veena Basavaraj wrote:
>     Let me add some more details on why I say the 0001 check is not required.
>     see the code in Matcher constructor, it is really confusing to have some code in MatcherFactroy and some code in Matcher Constructor, so if both are empty, then it is always set to some bytearray, so there is no need to do the extra check, it would have been nice if all this logic was in one place in MatcherFactroy.
>     
>         if (fromSchema.isEmpty() && toSchema.isEmpty()) {
>           this.fromSchema = ByteArraySchema.getInstance();
>           this.toSchema = ByteArraySchema.getInstance();

I really appreciate your carefulness. I also tried to understand the existing code. I imagine there are three cases: (1) both sides have a schema (jdbc to jdbc), so a NameMatcher is required (2) only one side has schema (jdbc to kite), so we need direct data copy (3) the case uses ByteArraySchema, i dont know. 

The Matcher class has been changed recently. I need some time to refresh the patch with some bold changes next week.


- Qian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67879
-----------------------------------------------------------


On Jan. 14, 2015, 12:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 14, 2015, 12:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 13, 2015, 8:34 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java, line 24
> > <https://reviews.apache.org/r/29849/diff/3/?file=819035#file819035line24>
> >
> >     pleae clean up 0000 if it is not used
> >     
> >     I am not sure why we need the 0001 checks since MatcherFactory instantiates these matchers after doing the right check

Let me add some more details on why I say the 0001 check is not required.
see the code in Matcher constructor, it is really confusing to have some code in MatcherFactroy and some code in Matcher Constructor, so if both are empty, then it is always set to some bytearray, so there is no need to do the extra check, it would have been nice if all this logic was in one place in MatcherFactroy.

    if (fromSchema.isEmpty() && toSchema.isEmpty()) {
      this.fromSchema = ByteArraySchema.getInstance();
      this.toSchema = ByteArraySchema.getInstance();


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67879
-----------------------------------------------------------


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67879
-----------------------------------------------------------



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java
<https://reviews.apache.org/r/29849/#comment111972>

    pleae clean up 0000 if it is not used
    
    I am not sure why we need the 0001 checks since MatcherFactory instantiates these matchers after doing the right check



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java
<https://reviews.apache.org/r/29849/#comment111973>

    I am not sure you got the point I mentioned, if you see the MatcherFactory code, it already does the empty check, so why need this check here? in that case Name matcher needs both from and to, so why only to is checked, 
    
    As I said above this whole 0001 check is not even required


- Veena Basavaraj


On Jan. 13, 2015, 8:13 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 8:13 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/
-----------------------------------------------------------

(Updated Jan. 14, 2015, 12:13 a.m.)


Review request for Sqoop.


Bugs: SQOOP-1988
    https://issues.apache.org/jira/browse/SQOOP-1988


Repository: sqoop-sqoop2


Description
-------

see jira


Diffs (updated)
-----

  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 

Diff: https://reviews.apache.org/r/29849/diff/


Testing
-------

Existing tests passed 


Thanks,

Qian Xu


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/
-----------------------------------------------------------

(Updated Jan. 14, 2015, 12:10 a.m.)


Review request for Sqoop.


Bugs: SQOOP-1988
    https://issues.apache.org/jira/browse/SQOOP-1988


Repository: sqoop-sqoop2


Description
-------

see jira


Diffs (updated)
-----

  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/MatcherError.java 577b091 
  connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
  connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 

Diff: https://reviews.apache.org/r/29849/diff/


Testing
-------

Existing tests passed 


Thanks,

Qian Xu


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.

> On Jan. 13, 2015, 7:36 a.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java, line 38
> > <https://reviews.apache.org/r/29849/diff/1/?file=818952#file818952line38>
> >
> >     while youa re please fix the , it should be Too few? not sure I understand this 
> >     
> >     public enum MatcherError implements ErrorCode {
> >       MATCHER_0000("To few Schemas provided."),
> >     
> >       ;
> >       
> >       Second, why only change this to an exception in Name and not in LocationMatcher?

on second look at the code it is even unnecessary to check for empty, please drop it

public class MatcherFactory {
  public static Matcher getMatcher(Schema fromSchema, Schema toSchema) {
    if (toSchema.isEmpty() || fromSchema.isEmpty()) {
      return new LocationMatcher(fromSchema, toSchema);
    } else {
      return new NameMatcher(fromSchema, toSchema);
    }
  }
}


- Veena


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67856
-----------------------------------------------------------


On Jan. 13, 2015, 7:14 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 7:14 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Qian Xu <qi...@intel.com>.

> On Jan. 13, 2015, 11:36 p.m., Veena Basavaraj wrote:
> > connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java, line 54
> > <https://reviews.apache.org/r/29849/diff/1/?file=818954#file818954line54>
> >
> >     are ew sure we want to nuke this test case?

Yes. As the TO schema is never empty for NameMatcher


- Qian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67856
-----------------------------------------------------------


On Jan. 13, 2015, 11:14 p.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 11:14 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>


Re: Review Request 29849: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat

Posted by Veena Basavaraj <vb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29849/#review67856
-----------------------------------------------------------



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java
<https://reviews.apache.org/r/29849/#comment111946>

    while youa re please fix the , it should be Too few? not sure I understand this 
    
    public enum MatcherError implements ErrorCode {
      MATCHER_0000("To few Schemas provided."),
    
      ;
      
      Second, why only change this to an exception in Name and not in LocationMatcher?



connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java
<https://reviews.apache.org/r/29849/#comment111948>

    thanks for the boolean clean up, simple if/else does the shop.



connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java
<https://reviews.apache.org/r/29849/#comment111949>

    are ew sure we want to nuke this test case?


- Veena Basavaraj


On Jan. 13, 2015, 7:14 a.m., Qian Xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29849/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2015, 7:14 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-1988
>     https://issues.apache.org/jira/browse/SQOOP-1988
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> see jira
> 
> 
> Diffs
> -----
> 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/LocationMatcher.java d92723e 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/Matcher.java 36ac5a5 
>   connector/connector-sdk/src/main/java/org/apache/sqoop/connector/matcher/NameMatcher.java 7cbc39f 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestLocationMatcher.java 624fa7b 
>   connector/connector-sdk/src/test/java/org/apache/sqoop/connector/matcher/TestNameMatcher.java 76ff0da 
> 
> Diff: https://reviews.apache.org/r/29849/diff/
> 
> 
> Testing
> -------
> 
> Existing tests passed 
> 
> 
> Thanks,
> 
> Qian Xu
> 
>