Issue w/ Sqoop incremental import

[Anson Abraham <an...@gmail.com>]

I'm having issues of doing a sqoop incemental import to hdfs.  i get
failures from multiple ways:

--connect 'jdbc:postgresql://www.example/dbname'  --username blah --
query 'SELECT guidval::varchar(36), column2, datefield2,
lastmodifieddate from table foo WHERE $CONDITIONS' --target-dir /hdfs/
path --split-by datefield2 --check-column lastmodifieddata --last-
value '2011-11-03' --incremental lastmodified

The error I get is Incremental requires an import.  So I'm assuming
I'm doing this incorrectly w/ the incremental import, where it won't
work w/ a query.

So I tried this then:

--connect 'jdbc:postgresql://www.example/dbname'  --username blah --
query 'SELECT guidval::varchar(36), column2, datefield2,
lastmodifieddate from table foo WHERE $CONDITIONS or
(lastmodifieddate  > '2011-11-03')' --target-dir /hdfs/path --split-by
datefield2

But I get an error of:
ERROR: cannot cast type integer to timestamp without time zone

when i try to convert to timestamp, that doesn't work.

trying this this:

--connect 'jdbc:postgresql://www.example/dbname'  --username blah --
query 'SELECT guidval::varchar(36), column2, datefield2,
lastmodifieddate from table foo' --where "lastmodifieddate  >
'2011-11-03' " --target-dir /hdfs/path --split-by datefield2

But the error I get, requires that I put a $CONDITIONS clause, which
doesn't work either.
I can't do the --table option, b/c one field
is a UUID which apparently isn't supported by SQOOP as of yet.  If so,
it kept failing, until I did a full select converting the UUID column
to varchar(36).

Also, the timestamp entered is not current timestamp, it can be any
timestamp that will be passed as a parameter.

Thanks

Re: Issue w/ Sqoop incremental import

[Mark Roddy <ma...@gmail.com>]

On Thu, Nov 3, 2011 at 6:22 PM, Anson Abraham <an...@gmail.com> wrote:
> I'm having issues of doing a sqoop incemental import to hdfs.  i get
> failures from multiple ways:
>
> --connect 'jdbc:postgresql://www.example/dbname'  --username blah --
> query 'SELECT guidval::varchar(36), column2, datefield2,
> lastmodifieddate from table foo WHERE $CONDITIONS' --target-dir /hdfs/
> path --split-by datefield2 --check-column lastmodifieddata --last-
> value '2011-11-03' --incremental lastmodified
>
> The error I get is Incremental requires an import.  So I'm assuming
> I'm doing this incorrectly w/ the incremental import, where it won't
> work w/ a query.
>
> So I tried this then:
>
> --connect 'jdbc:postgresql://www.example/dbname'  --username blah --
> query 'SELECT guidval::varchar(36), column2, datefield2,
> lastmodifieddate from table foo WHERE $CONDITIONS or
> (lastmodifieddate  > '2011-11-03')' --target-dir /hdfs/path --split-by
> datefield2
>
> But I get an error of:
> ERROR: cannot cast type integer to timestamp without time zone
>
> when i try to convert to timestamp, that doesn't work.
>
> trying this this:
>
> --connect 'jdbc:postgresql://www.example/dbname'  --username blah --
> query 'SELECT guidval::varchar(36), column2, datefield2,
> lastmodifieddate from table foo' --where "lastmodifieddate  >
> '2011-11-03' " --target-dir /hdfs/path --split-by datefield2
>
> But the error I get, requires that I put a $CONDITIONS clause, which
> doesn't work either.
>


+1 if there is a solution to this.  I have some legacy postgres types
that sqoop doesn't understand, but are easily
converted to char types.  Having to completely generate your own sql
to work around it defeats a decent portion
of (my) reason for using sqoop.


> I can't do the --table option, b/c one field
> is a UUID which apparently isn't supported by SQOOP as of yet.  If so,
> it kept failing, until I did a full select converting the UUID column
> to varchar(36).
> Also, the timestamp entered is not current timestamp, it can be any
> timestamp that will be passed as a parameter.
> Thanks
>
>