You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Linlin Chen(Sabrina)" <li...@hengtiansoft.com> on 2013/04/12 07:43:39 UTC
Hadoop update mysql table
Hi,all:
I'm trying to use Hadoop to read data, process and then write back to MySql. However, it seems as if I can only do insertion. If I try to write back with the same primary key(it means update), there will be an error like "Duplicate XXX for primary key'. Has anyone solved this problem? And is it possible to update? Thanks very much!!!
Linlin Chen
Re: Hadoop update mysql table
Posted by Sékine Coulibaly <sc...@gmail.com>.
Linlin,
A primary key means a unicity constraint being set on that column. You need
to update that line, not insert it, unless you use INSERT ... ON DUPLICATE
KEY UPDATE ... to perform your upserts.
This sounds more like a mySQL issue.
BR
2013/4/14 Jens Scheidtmann <je...@gmail.com>
> Dear Linlin,
>
> seems you are having a SQL problem and not a hadoop one.
>
> If you INSERT another row with the same primary key, you will get this
> error. Are you really using UPDATE for writing back?
>
> Also check if MERGE is more appropriate, see
> http://en.wikipedia.org/wiki/Merge_%28SQL%29
>
> Best regards,
>
> Jens
>
Re: Hadoop update mysql table
Posted by Sékine Coulibaly <sc...@gmail.com>.
Linlin,
A primary key means a unicity constraint being set on that column. You need
to update that line, not insert it, unless you use INSERT ... ON DUPLICATE
KEY UPDATE ... to perform your upserts.
This sounds more like a mySQL issue.
BR
2013/4/14 Jens Scheidtmann <je...@gmail.com>
> Dear Linlin,
>
> seems you are having a SQL problem and not a hadoop one.
>
> If you INSERT another row with the same primary key, you will get this
> error. Are you really using UPDATE for writing back?
>
> Also check if MERGE is more appropriate, see
> http://en.wikipedia.org/wiki/Merge_%28SQL%29
>
> Best regards,
>
> Jens
>
Re: Hadoop update mysql table
Posted by Sékine Coulibaly <sc...@gmail.com>.
Linlin,
A primary key means a unicity constraint being set on that column. You need
to update that line, not insert it, unless you use INSERT ... ON DUPLICATE
KEY UPDATE ... to perform your upserts.
This sounds more like a mySQL issue.
BR
2013/4/14 Jens Scheidtmann <je...@gmail.com>
> Dear Linlin,
>
> seems you are having a SQL problem and not a hadoop one.
>
> If you INSERT another row with the same primary key, you will get this
> error. Are you really using UPDATE for writing back?
>
> Also check if MERGE is more appropriate, see
> http://en.wikipedia.org/wiki/Merge_%28SQL%29
>
> Best regards,
>
> Jens
>
Re: Hadoop update mysql table
Posted by Sékine Coulibaly <sc...@gmail.com>.
Linlin,
A primary key means a unicity constraint being set on that column. You need
to update that line, not insert it, unless you use INSERT ... ON DUPLICATE
KEY UPDATE ... to perform your upserts.
This sounds more like a mySQL issue.
BR
2013/4/14 Jens Scheidtmann <je...@gmail.com>
> Dear Linlin,
>
> seems you are having a SQL problem and not a hadoop one.
>
> If you INSERT another row with the same primary key, you will get this
> error. Are you really using UPDATE for writing back?
>
> Also check if MERGE is more appropriate, see
> http://en.wikipedia.org/wiki/Merge_%28SQL%29
>
> Best regards,
>
> Jens
>
Re: Hadoop update mysql table
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Linlin,
seems you are having a SQL problem and not a hadoop one.
If you INSERT another row with the same primary key, you will get this
error. Are you really using UPDATE for writing back?
Also check if MERGE is more appropriate, see
http://en.wikipedia.org/wiki/Merge_%28SQL%29
Best regards,
Jens
Re: Hadoop update mysql table
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Linlin,
seems you are having a SQL problem and not a hadoop one.
If you INSERT another row with the same primary key, you will get this
error. Are you really using UPDATE for writing back?
Also check if MERGE is more appropriate, see
http://en.wikipedia.org/wiki/Merge_%28SQL%29
Best regards,
Jens
Re: Hadoop update mysql table
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Linlin,
seems you are having a SQL problem and not a hadoop one.
If you INSERT another row with the same primary key, you will get this
error. Are you really using UPDATE for writing back?
Also check if MERGE is more appropriate, see
http://en.wikipedia.org/wiki/Merge_%28SQL%29
Best regards,
Jens
Re: Hadoop update mysql table
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Linlin,
seems you are having a SQL problem and not a hadoop one.
If you INSERT another row with the same primary key, you will get this
error. Are you really using UPDATE for writing back?
Also check if MERGE is more appropriate, see
http://en.wikipedia.org/wiki/Merge_%28SQL%29
Best regards,
Jens
Re: Hadoop update mysql table
Posted by Harsh J <ha...@cloudera.com>.
You probably are doing an operation already supported and provided by
Apache Sqoop (http://sqoop.apache.org). Do try it out to see if its
updates feature solves your need (aside of its other features you can
simply reuse).
In either case the trouble is in how the program ends up writing to
the DB. If it has to update a row, such an instruction ought to go -
and if whatever you're using is not doing that, you'll need to modify
it to do so.
On Fri, Apr 12, 2013 at 11:13 AM, Linlin Chen(Sabrina)
<li...@hengtiansoft.com> wrote:
> Hi,all:
>
>
>
> I'm trying to use Hadoop to read data, process and then write back to MySql.
> However, it seems as if I can only do insertion. If I try to write back with
> the same primary key(it means update), there will be an error like
> "Duplicate XXX for primary key'. Has anyone solved this problem? And is it
> possible to update? Thanks very much!!!
>
>
>
> Linlin Chen
--
Harsh J
Re: Hadoop update mysql table
Posted by Harsh J <ha...@cloudera.com>.
You probably are doing an operation already supported and provided by
Apache Sqoop (http://sqoop.apache.org). Do try it out to see if its
updates feature solves your need (aside of its other features you can
simply reuse).
In either case the trouble is in how the program ends up writing to
the DB. If it has to update a row, such an instruction ought to go -
and if whatever you're using is not doing that, you'll need to modify
it to do so.
On Fri, Apr 12, 2013 at 11:13 AM, Linlin Chen(Sabrina)
<li...@hengtiansoft.com> wrote:
> Hi,all:
>
>
>
> I'm trying to use Hadoop to read data, process and then write back to MySql.
> However, it seems as if I can only do insertion. If I try to write back with
> the same primary key(it means update), there will be an error like
> "Duplicate XXX for primary key'. Has anyone solved this problem? And is it
> possible to update? Thanks very much!!!
>
>
>
> Linlin Chen
--
Harsh J
Re: Hadoop update mysql table
Posted by Harsh J <ha...@cloudera.com>.
You probably are doing an operation already supported and provided by
Apache Sqoop (http://sqoop.apache.org). Do try it out to see if its
updates feature solves your need (aside of its other features you can
simply reuse).
In either case the trouble is in how the program ends up writing to
the DB. If it has to update a row, such an instruction ought to go -
and if whatever you're using is not doing that, you'll need to modify
it to do so.
On Fri, Apr 12, 2013 at 11:13 AM, Linlin Chen(Sabrina)
<li...@hengtiansoft.com> wrote:
> Hi,all:
>
>
>
> I'm trying to use Hadoop to read data, process and then write back to MySql.
> However, it seems as if I can only do insertion. If I try to write back with
> the same primary key(it means update), there will be an error like
> "Duplicate XXX for primary key'. Has anyone solved this problem? And is it
> possible to update? Thanks very much!!!
>
>
>
> Linlin Chen
--
Harsh J
Re: Hadoop update mysql table
Posted by Harsh J <ha...@cloudera.com>.
You probably are doing an operation already supported and provided by
Apache Sqoop (http://sqoop.apache.org). Do try it out to see if its
updates feature solves your need (aside of its other features you can
simply reuse).
In either case the trouble is in how the program ends up writing to
the DB. If it has to update a row, such an instruction ought to go -
and if whatever you're using is not doing that, you'll need to modify
it to do so.
On Fri, Apr 12, 2013 at 11:13 AM, Linlin Chen(Sabrina)
<li...@hengtiansoft.com> wrote:
> Hi,all:
>
>
>
> I'm trying to use Hadoop to read data, process and then write back to MySql.
> However, it seems as if I can only do insertion. If I try to write back with
> the same primary key(it means update), there will be an error like
> "Duplicate XXX for primary key'. Has anyone solved this problem? And is it
> possible to update? Thanks very much!!!
>
>
>
> Linlin Chen
--
Harsh J