You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Eric Wadsworth <er...@wadhome.org> on 2011/12/05 19:46:23 UTC
couple of questions
Hi folks,
The company I work for (cj.com) will shortly be using sqoop heavily. I'm
working on getting it up and running for us. We've got a bunch of Oracle
databases that we're going to connecting to our shiny new HDFS and HBase
clusters. We're not using the Cloudera distributions, just apache hadoop.
I got 1.4.1-SNAPSHOT to build just fine. The tests all pass.
Questions:
1. Are there any plans for an IRC channel *#sqoop* on freenode?
2. Is this version 1.4.1 known to basically work? Should we use this for
production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
Thanks!
--- wad (Eric Wadsworth)
Re: couple of questions
Posted by Eric Wadsworth <er...@wadhome.org>.
Arvind,
Thanks for the info! It looks like I need to start trying to use it, and
see how it works. I'll probably bug you guys with questions and stuff,
and perhaps emit some patches along the way too.
--- wad
On 12/05/2011 11:15 AM, Arvind Prabhakar wrote:
> Hi Eric,
>
>> 1. Are there any plans for an IRC channel #sqoop on freenode?
> There is a #sqoop IRC channel on freenode but is not currently used.
> If there is interest from the community, we can start hanging out
> there on a regular basis. Maybe we can call a community to vote to see
> how many people are interested in reviving the channel.
>
>> Is this version 1.4.1 known to basically work? Should we use this for production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
> The most recent Sqoop release was 1.4.0-incubating which saw numerous
> sweeping changes to align the namespace from com.cloudera.sqoop to
> org.apache.sqoop. As is the case with any release that contains major
> changes, it is advisable to test out the binaries in your use case in
> a safe staging/testing area to ensure that it works for you. If you
> find issues, please file the appropriate JIRAs and we will be happy to
> assist. Patches are welcome too!
>
> Thanks,
> Arvind
>
>
>
> On Mon, Dec 5, 2011 at 10:46 AM, Eric Wadsworth<er...@wadhome.org> wrote:
>> Hi folks,
>>
>> The company I work for (cj.com) will shortly be using sqoop heavily. I'm
>> working on getting it up and running for us. We've got a bunch of Oracle
>> databases that we're going to connecting to our shiny new HDFS and HBase
>> clusters. We're not using the Cloudera distributions, just apache hadoop.
>>
>> I got 1.4.1-SNAPSHOT to build just fine. The tests all pass.
>>
>> Questions:
>> 1. Are there any plans for an IRC channel #sqoop on freenode?
>> 2. Is this version 1.4.1 known to basically work? Should we use this for
>> production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
>>
>> Thanks!
>>
>> --- wad (Eric Wadsworth)
Re: couple of questions
Posted by Arvind Prabhakar <ar...@apache.org>.
Hi Eric,
> 1. Are there any plans for an IRC channel #sqoop on freenode?
There is a #sqoop IRC channel on freenode but is not currently used.
If there is interest from the community, we can start hanging out
there on a regular basis. Maybe we can call a community to vote to see
how many people are interested in reviving the channel.
> Is this version 1.4.1 known to basically work? Should we use this for production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
The most recent Sqoop release was 1.4.0-incubating which saw numerous
sweeping changes to align the namespace from com.cloudera.sqoop to
org.apache.sqoop. As is the case with any release that contains major
changes, it is advisable to test out the binaries in your use case in
a safe staging/testing area to ensure that it works for you. If you
find issues, please file the appropriate JIRAs and we will be happy to
assist. Patches are welcome too!
Thanks,
Arvind
On Mon, Dec 5, 2011 at 10:46 AM, Eric Wadsworth <er...@wadhome.org> wrote:
> Hi folks,
>
> The company I work for (cj.com) will shortly be using sqoop heavily. I'm
> working on getting it up and running for us. We've got a bunch of Oracle
> databases that we're going to connecting to our shiny new HDFS and HBase
> clusters. We're not using the Cloudera distributions, just apache hadoop.
>
> I got 1.4.1-SNAPSHOT to build just fine. The tests all pass.
>
> Questions:
> 1. Are there any plans for an IRC channel #sqoop on freenode?
> 2. Is this version 1.4.1 known to basically work? Should we use this for
> production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
>
> Thanks!
>
> --- wad (Eric Wadsworth)
Re: couple of questions
Posted by Ken Krugler <kk...@transpac.com>.
Sorry, should have included that info.
Yes, it's the Cloudera release - actually a version of the code cut from the GitHub master before 1.3.0 was released, with a few patches applied.
Running it on CDH3u0.
-- Ken
On Dec 5, 2011, at 2:58pm, Eric Wadsworth wrote:
> Ken,
>
> Thanks for the datapoint! So, you're using the Cloudera version then, since it's prior to 1.4.0, right? Are you using a CDH as well, or an Apache Hadoop cluster?
>
> --- wad
>
> On 12/05/2011 12:08 PM, Ken Krugler wrote:
>>
>>
>> On Dec 5, 2011, at 10:46am, Eric Wadsworth wrote:
>>
>>> Hi folks,
>>>
>>> The company I work for (cj.com) will shortly be using sqoop heavily. I'm working on getting it up and running for us. We've got a bunch of Oracle databases that we're going to connecting to our shiny new HDFS and HBase clusters. We're not using the Cloudera distributions, just apache hadoop.
>>>
>>> I got 1.4.1-SNAPSHOT to build just fine. The tests all pass.
>>>
>>> Questions:
>>> 1. Are there any plans for an IRC channel #sqoop on freenode?
>>> 2. Is this version 1.4.1 known to basically work? Should we use this for production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
>>
>> The only data point I can provide is that we're using 1.3 with OraOop for pulling lots of data from Oracle DBs, and it's working well for many billions of records.
>>
>> We're importing into HDFS as text files, if that matters.
>>
>> -- Ken
>>
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Mahout & Solr
>>
>>
>>
>>
>
--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr
Re: couple of questions
Posted by Eric Wadsworth <er...@wadhome.org>.
Ken,
Thanks for the datapoint! So, you're using the Cloudera version then,
since it's prior to 1.4.0, right? Are you using a CDH as well, or an
Apache Hadoop cluster?
--- wad
On 12/05/2011 12:08 PM, Ken Krugler wrote:
>
> On Dec 5, 2011, at 10:46am, Eric Wadsworth wrote:
>
>> Hi folks,
>>
>> The company I work for (cj.com <http://cj.com>) will shortly be using
>> sqoop heavily. I'm working on getting it up and running for us. We've
>> got a bunch of Oracle databases that we're going to connecting to our
>> shiny new HDFS and HBase clusters. We're not using the Cloudera
>> distributions, just apache hadoop.
>>
>> I got 1.4.1-SNAPSHOT to build just fine. The tests all pass.
>>
>> Questions:
>> 1. Are there any plans for an IRC channel *#sqoop* on freenode?
>> 2. Is this version 1.4.1 known to basically work? Should we use this
>> for production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
>
> The only data point I can provide is that we're using 1.3 with OraOop
> for pulling lots of data from Oracle DBs, and it's working well for
> many billions of records.
>
> We're importing into HDFS as text files, if that matters.
>
> -- Ken
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
>
>
>
>
Re: couple of questions
Posted by Ken Krugler <kk...@transpac.com>.
On Dec 5, 2011, at 10:46am, Eric Wadsworth wrote:
> Hi folks,
>
> The company I work for (cj.com) will shortly be using sqoop heavily. I'm working on getting it up and running for us. We've got a bunch of Oracle databases that we're going to connecting to our shiny new HDFS and HBase clusters. We're not using the Cloudera distributions, just apache hadoop.
>
> I got 1.4.1-SNAPSHOT to build just fine. The tests all pass.
>
> Questions:
> 1. Are there any plans for an IRC channel #sqoop on freenode?
> 2. Is this version 1.4.1 known to basically work? Should we use this for production work, or fall back to Cloudera's 1.3, or the 1.4.0 branch?
The only data point I can provide is that we're using 1.3 with OraOop for pulling lots of data from Oracle DBs, and it's working well for many billions of records.
We're importing into HDFS as text files, if that matters.
-- Ken
--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr
Re: SQOOP AVRO ORAOOP some questions
Posted by Antoine Pieretti <an...@airfrance.fr>.
Hello,
Thanks for your help.
sorry for the delay.
I have created the JIRA : https://issues.apache.org/jira/browse/SQOOP-406
for the Avro update issue.
Regards
Antoine
Jarek Jarcec
Cecho
<jarcec@apache.or A
g> sqoop-user@incubator.apache.org
cc
07/12/2011 12:39
Objet
Re: SQOOP AVRO ORAOOP some
Veuillez répondre questions
à
sqoop-user@incuba
tor.apache.org
Hmm... I've just marked problem with default AVRO mapping of type DATE as
"Won't fix" in SQOOP-399 because it would break backward compatibility, but
I'm starting to think that it needs to be addressed anyway. I'll look on it
later this week. Can you please file JIRA for that?
We've recently added option to force sqoop use different than default
mapping, however version 1.4.0 do not support that for AVRO output. But
there is working patch attached to SQOOP-362 that has been committed to the
trunk and will be included in next release. Meantime you might consider
applying this patch to the base 1.4.0 release and rebuilding sqoop for you.
Can you also file JIRA for your second issue? You can also submit your
patch for review in case that you managed to fix that issue.
Thank you very much for your time during investigating the issues!
Jarcec
SQOOP-399 https://issues.apache.org/jira/browse/SQOOP-399
SQOOP-362 https://issues.apache.org/jira/browse/SQOOP-362
On Wed, Dec 07, 2011 at 11:48:24AM +0100, Antoine Pieretti wrote:
>
> Hi Kate,
>
> here the versions :
> Sqoop : 1.4.0
> Oraoop : oraoop-1.4.0.149
> Hadoop : hadoop-core-0.20.2-cdh3u2
>
> The first thing we found, is that the AvroSchemaGenerator, return a LONG
> type for a DATE in toAvroType, but ORAOOP always works with String.
>
> Please, find here the console log during an Update
> (See attached file: SQOOP_UPDATE_LOG-txt.zip)
>
> In debug, I found that in TextExportMapper, he call parse method, wich
> __loadFromFields with a List<String>, with the first value equals to the
> text "Objavro.schema?{"type":"record".
>
> If in the JdbcUpdateExportJob I implement the AvroExportMapper for an
Avro
> fyleType, it works.
>
> Thanks for your help.
>
> Please, also find an avro file example.
>
>
> (See attached file: part-m-00000-avro.zip)
>
> Regards, Antoine
>
>
>
>
>
>
>
> Kate Ting
> <kate@cloudera.co
> m>
A
> sqoop-user@incubator.apache.org
> 07/12/2011 09:27
cc
>
>
Objet
> Veuillez répondre Re: SQOOP AVRO ORAOOP some
> à questions
> sqoop-user@incuba
> tor.apache.org
>
>
>
>
>
>
>
>
> Hi Antoine,
>
> The following would be helpful in figuring out what could be causing the
> failed job:
>
> What Sqoop, OraOop, and Hadoop versions are you running?
> Please run Sqoop job with --verbose flag and then attach console log.
> Also, provide a representative input data set that triggers this problem.
>
> Regards, Kate
>
> On Tue, Dec 6, 2011 at 12:15 AM, Antoine Pieretti <
> anpieretti-ext@airfrance.fr> wrote:
>
>
> Hi folks,
>
> i am a newbie within Sqoop and Avro.
>
> I try to manipulate data from an Oracle Database, import, and
export
> Avro files.
>
> The import works well, the export (INSERT) also, but when i am
trying
> to Update data from an Avro file to the DB the job failed.
>
> Here my command line : export --connect jdbc:oracle:thin:@qxxxxxxx
> --username xxxxxxxxxxxxxxxxx_O --password xxxxxxxxxxx_O --table
> MY_TABLE --verbose --export-dir D:\\tmp\extraction\Rep --update-key
> My_COLUMN_KEY.
>
> The job calls the TextExportMapper (not the AvropExportMapper), and
> try to parse the file, wich contains the Avro schemas and parse it
as
> a data line.
>
> Is it a common issue ?
>
> Thanks for your help
>
> Antoine PIERETTI
>
>
>
> Accédez aux meilleurs tarifs Air France, gérez vos réservations et
> enregistrez-vous en ligne sur http://www.airfrance.com
> Find best Air France fares, manage your reservations and check in
> online at http://www.airfrance.com
>
>
> Les données et renseignements contenus dans ce message peuvent être
> de nature confidentielle et soumis au secret professionnel et sont
> destinés à l'usage exclusif du destinataire dont les coordonnées
> figurent ci-dessus. Si vous recevez cette communication par erreur,
> nous vous demandons de ne pas la copier, l'utiliser ou la
divulguer.
> Nous vous prions de notifier cette erreur à l'expéditeur et
d'effacer
> immédiatement cette communication de votre système. Société Air
> France - Société anonyme au capital de 1 901 231 625 euros - RCS
> Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG
> CEDEX
> The data and information contained in this message may be
> confidential and subject to professional secrecy and are intended
for
> the exclusive use of the recipient at the address shown above. If
you
> receive this message by mistake, we ask you not to copy, use or
> disclose it. Please notify this error to the sender immediately and
> delete this message from your system. Société Air France - Limited
> company with capital of 1,901,231,625 euros - Bobigny register of
> companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy
CDG
> CEDEX Pensez à l'environnement avant d'imprimer ce message.
> Think of the environment before printing this mail.
>
>
>
>
>
>
> Accédez aux meilleurs tarifs Air France, gérez vos réservations et
enregistrez-vous en ligne sur http://www.airfrance.com
> Find best Air France fares, manage your reservations and check in online
at http://www.airfrance.com Les données et renseignements contenus dans
ce message peuvent être de nature confidentielle et soumis au secret
professionnel et sont destinés à l'usage exclusif du destinataire dont les
coordonnées figurent ci-dessus. Si vous recevez cette communication par
erreur, nous vous demandons de ne pas la copier, l'utiliser ou la
divulguer. Nous vous prions de notifier cette erreur à l'expéditeur et
d'effacer immédiatement cette communication de votre système. Société Air
France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny
(France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX
> The data and information contained in this message may be confidential
and subject to professional secrecy and are intended for the exclusive use
of the recipient at the address shown above. If you receive this message by
mistake, we ask you not to copy, use or disclose it. Please notify this
error to the sender immediately and delete this message from your system.
Société Air France - Limited company with capital of 1,901,231,625 euros -
Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95
747 Roissy CDG CEDEX Pensez à l'environnement avant d'imprimer ce message.
> Think of the environment before printing this mail.
(See attached file: signature.asc)
Accédez aux meilleurs tarifs Air France, gérez vos réservations et enregistrez-vous en ligne sur http://www.airfrance.com
Find best Air France fares, manage your reservations and check in online at http://www.airfrance.com Les données et renseignements contenus dans ce message peuvent être de nature confidentielle et soumis au secret professionnel et sont destinés à l'usage exclusif du destinataire dont les coordonnées figurent ci-dessus. Si vous recevez cette communication par erreur, nous vous demandons de ne pas la copier, l'utiliser ou la divulguer. Nous vous prions de notifier cette erreur à l'expéditeur et d'effacer immédiatement cette communication de votre système. Société Air France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX
The data and information contained in this message may be confidential and subject to professional secrecy and are intended for the exclusive use of the recipient at the address shown above. If you receive this message by mistake, we ask you not to copy, use or disclose it. Please notify this error to the sender immediately and delete this message from your system. Société Air France - Limited company with capital of 1,901,231,625 euros - Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX Pensez à l'environnement avant d'imprimer ce message.
Think of the environment before printing this mail.
Re: SQOOP AVRO ORAOOP some questions
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Hmm... I've just marked problem with default AVRO mapping of type DATE as "Won't fix" in SQOOP-399 because it would break backward compatibility, but I'm starting to think that it needs to be addressed anyway. I'll look on it later this week. Can you please file JIRA for that?
We've recently added option to force sqoop use different than default mapping, however version 1.4.0 do not support that for AVRO output. But there is working patch attached to SQOOP-362 that has been committed to the trunk and will be included in next release. Meantime you might consider applying this patch to the base 1.4.0 release and rebuilding sqoop for you.
Can you also file JIRA for your second issue? You can also submit your patch for review in case that you managed to fix that issue.
Thank you very much for your time during investigating the issues!
Jarcec
SQOOP-399 https://issues.apache.org/jira/browse/SQOOP-399
SQOOP-362 https://issues.apache.org/jira/browse/SQOOP-362
On Wed, Dec 07, 2011 at 11:48:24AM +0100, Antoine Pieretti wrote:
>
> Hi Kate,
>
> here the versions :
> Sqoop : 1.4.0
> Oraoop : oraoop-1.4.0.149
> Hadoop : hadoop-core-0.20.2-cdh3u2
>
> The first thing we found, is that the AvroSchemaGenerator, return a LONG
> type for a DATE in toAvroType, but ORAOOP always works with String.
>
> Please, find here the console log during an Update
> (See attached file: SQOOP_UPDATE_LOG-txt.zip)
>
> In debug, I found that in TextExportMapper, he call parse method, wich
> __loadFromFields with a List<String>, with the first value equals to the
> text "Objavro.schema?{"type":"record".
>
> If in the JdbcUpdateExportJob I implement the AvroExportMapper for an Avro
> fyleType, it works.
>
> Thanks for your help.
>
> Please, also find an avro file example.
>
>
> (See attached file: part-m-00000-avro.zip)
>
> Regards, Antoine
>
>
>
>
>
>
>
> Kate Ting
> <kate@cloudera.co
> m> A
> sqoop-user@incubator.apache.org
> 07/12/2011 09:27 cc
>
> Objet
> Veuillez répondre Re: SQOOP AVRO ORAOOP some
> à questions
> sqoop-user@incuba
> tor.apache.org
>
>
>
>
>
>
>
>
> Hi Antoine,
>
> The following would be helpful in figuring out what could be causing the
> failed job:
>
> What Sqoop, OraOop, and Hadoop versions are you running?
> Please run Sqoop job with --verbose flag and then attach console log.
> Also, provide a representative input data set that triggers this problem.
>
> Regards, Kate
>
> On Tue, Dec 6, 2011 at 12:15 AM, Antoine Pieretti <
> anpieretti-ext@airfrance.fr> wrote:
>
>
> Hi folks,
>
> i am a newbie within Sqoop and Avro.
>
> I try to manipulate data from an Oracle Database, import, and export
> Avro files.
>
> The import works well, the export (INSERT) also, but when i am trying
> to Update data from an Avro file to the DB the job failed.
>
> Here my command line : export --connect jdbc:oracle:thin:@qxxxxxxx
> --username xxxxxxxxxxxxxxxxx_O --password xxxxxxxxxxx_O --table
> MY_TABLE --verbose --export-dir D:\\tmp\extraction\Rep --update-key
> My_COLUMN_KEY.
>
> The job calls the TextExportMapper (not the AvropExportMapper), and
> try to parse the file, wich contains the Avro schemas and parse it as
> a data line.
>
> Is it a common issue ?
>
> Thanks for your help
>
> Antoine PIERETTI
>
>
>
> Accédez aux meilleurs tarifs Air France, gérez vos réservations et
> enregistrez-vous en ligne sur http://www.airfrance.com
> Find best Air France fares, manage your reservations and check in
> online at http://www.airfrance.com
>
>
> Les données et renseignements contenus dans ce message peuvent être
> de nature confidentielle et soumis au secret professionnel et sont
> destinés à l'usage exclusif du destinataire dont les coordonnées
> figurent ci-dessus. Si vous recevez cette communication par erreur,
> nous vous demandons de ne pas la copier, l'utiliser ou la divulguer.
> Nous vous prions de notifier cette erreur à l'expéditeur et d'effacer
> immédiatement cette communication de votre système. Société Air
> France - Société anonyme au capital de 1 901 231 625 euros - RCS
> Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG
> CEDEX
> The data and information contained in this message may be
> confidential and subject to professional secrecy and are intended for
> the exclusive use of the recipient at the address shown above. If you
> receive this message by mistake, we ask you not to copy, use or
> disclose it. Please notify this error to the sender immediately and
> delete this message from your system. Société Air France - Limited
> company with capital of 1,901,231,625 euros - Bobigny register of
> companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG
> CEDEX Pensez à l'environnement avant d'imprimer ce message.
> Think of the environment before printing this mail.
>
>
>
>
>
>
> Accédez aux meilleurs tarifs Air France, gérez vos réservations et enregistrez-vous en ligne sur http://www.airfrance.com
> Find best Air France fares, manage your reservations and check in online at http://www.airfrance.com Les données et renseignements contenus dans ce message peuvent être de nature confidentielle et soumis au secret professionnel et sont destinés à l'usage exclusif du destinataire dont les coordonnées figurent ci-dessus. Si vous recevez cette communication par erreur, nous vous demandons de ne pas la copier, l'utiliser ou la divulguer. Nous vous prions de notifier cette erreur à l'expéditeur et d'effacer immédiatement cette communication de votre système. Société Air France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX
> The data and information contained in this message may be confidential and subject to professional secrecy and are intended for the exclusive use of the recipient at the address shown above. If you receive this message by mistake, we ask you not to copy, use or disclose it. Please notify this error to the sender immediately and delete this message from your system. Société Air France - Limited company with capital of 1,901,231,625 euros - Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX Pensez à l'environnement avant d'imprimer ce message.
> Think of the environment before printing this mail.
Re: SQOOP AVRO ORAOOP some questions
Posted by Antoine Pieretti <an...@airfrance.fr>.
Hi Kate,
here the versions :
Sqoop : 1.4.0
Oraoop : oraoop-1.4.0.149
Hadoop : hadoop-core-0.20.2-cdh3u2
The first thing we found, is that the AvroSchemaGenerator, return a LONG
type for a DATE in toAvroType, but ORAOOP always works with String.
Please, find here the console log during an Update
(See attached file: SQOOP_UPDATE_LOG-txt.zip)
In debug, I found that in TextExportMapper, he call parse method, wich
__loadFromFields with a List<String>, with the first value equals to the
text "Objavro.schema?{"type":"record".
If in the JdbcUpdateExportJob I implement the AvroExportMapper for an Avro
fyleType, it works.
Thanks for your help.
Please, also find an avro file example.
(See attached file: part-m-00000-avro.zip)
Regards, Antoine
Kate Ting
<kate@cloudera.co
m> A
sqoop-user@incubator.apache.org
07/12/2011 09:27 cc
Objet
Veuillez répondre Re: SQOOP AVRO ORAOOP some
à questions
sqoop-user@incuba
tor.apache.org
Hi Antoine,
The following would be helpful in figuring out what could be causing the
failed job:
What Sqoop, OraOop, and Hadoop versions are you running?
Please run Sqoop job with --verbose flag and then attach console log.
Also, provide a representative input data set that triggers this problem.
Regards, Kate
On Tue, Dec 6, 2011 at 12:15 AM, Antoine Pieretti <
anpieretti-ext@airfrance.fr> wrote:
Hi folks,
i am a newbie within Sqoop and Avro.
I try to manipulate data from an Oracle Database, import, and export
Avro files.
The import works well, the export (INSERT) also, but when i am trying
to Update data from an Avro file to the DB the job failed.
Here my command line : export --connect jdbc:oracle:thin:@qxxxxxxx
--username xxxxxxxxxxxxxxxxx_O --password xxxxxxxxxxx_O --table
MY_TABLE --verbose --export-dir D:\\tmp\extraction\Rep --update-key
My_COLUMN_KEY.
The job calls the TextExportMapper (not the AvropExportMapper), and
try to parse the file, wich contains the Avro schemas and parse it as
a data line.
Is it a common issue ?
Thanks for your help
Antoine PIERETTI
Accédez aux meilleurs tarifs Air France, gérez vos réservations et
enregistrez-vous en ligne sur http://www.airfrance.com
Find best Air France fares, manage your reservations and check in
online at http://www.airfrance.com
Les données et renseignements contenus dans ce message peuvent être
de nature confidentielle et soumis au secret professionnel et sont
destinés à l'usage exclusif du destinataire dont les coordonnées
figurent ci-dessus. Si vous recevez cette communication par erreur,
nous vous demandons de ne pas la copier, l'utiliser ou la divulguer.
Nous vous prions de notifier cette erreur à l'expéditeur et d'effacer
immédiatement cette communication de votre système. Société Air
France - Société anonyme au capital de 1 901 231 625 euros - RCS
Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG
CEDEX
The data and information contained in this message may be
confidential and subject to professional secrecy and are intended for
the exclusive use of the recipient at the address shown above. If you
receive this message by mistake, we ask you not to copy, use or
disclose it. Please notify this error to the sender immediately and
delete this message from your system. Société Air France - Limited
company with capital of 1,901,231,625 euros - Bobigny register of
companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG
CEDEX Pensez à l'environnement avant d'imprimer ce message.
Think of the environment before printing this mail.
Accédez aux meilleurs tarifs Air France, gérez vos réservations et enregistrez-vous en ligne sur http://www.airfrance.com
Find best Air France fares, manage your reservations and check in online at http://www.airfrance.com Les données et renseignements contenus dans ce message peuvent être de nature confidentielle et soumis au secret professionnel et sont destinés à l'usage exclusif du destinataire dont les coordonnées figurent ci-dessus. Si vous recevez cette communication par erreur, nous vous demandons de ne pas la copier, l'utiliser ou la divulguer. Nous vous prions de notifier cette erreur à l'expéditeur et d'effacer immédiatement cette communication de votre système. Société Air France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX
The data and information contained in this message may be confidential and subject to professional secrecy and are intended for the exclusive use of the recipient at the address shown above. If you receive this message by mistake, we ask you not to copy, use or disclose it. Please notify this error to the sender immediately and delete this message from your system. Société Air France - Limited company with capital of 1,901,231,625 euros - Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX Pensez à l'environnement avant d'imprimer ce message.
Think of the environment before printing this mail.
Re: SQOOP AVRO ORAOOP some questions
Posted by Kate Ting <ka...@cloudera.com>.
Hi Antoine,
The following would be helpful in figuring out what could be causing the
failed job:
What Sqoop, OraOop, and Hadoop versions are you running?
Please run Sqoop job with --verbose flag and then attach console log.
Also, provide a representative input data set that triggers this problem.
Regards, Kate
On Tue, Dec 6, 2011 at 12:15 AM, Antoine Pieretti <
anpieretti-ext@airfrance.fr> wrote:
> Hi folks,
>
> i am a newbie within Sqoop and Avro.
>
> I try to manipulate data from an Oracle Database, import, and export Avro
> files.
>
> The import works well, the export (INSERT) also, but when i am trying to
> Update data from an Avro file to the DB the job failed.
>
> Here my command line : export --connect jdbc:oracle:thin:@qxxxxxxx--username xxxxxxxxxxxxxxxxx_O --password xxxxxxxxxxx_O --table MY_TABLE
> --verbose --export-dir D:\\tmp\extraction\Rep --update-key My_COLUMN_KEY.
>
> The job calls the TextExportMapper (not the AvropExportMapper), and try to
> parse the file, wich contains the Avro schemas and parse it as a data line.
>
> Is it a common issue ?
>
> Thanks for your help
>
> Antoine PIERETTI
>
>
> Accédez aux meilleurs tarifs Air France, gérez vos réservations et
> enregistrez-vous en ligne sur http://www.airfrance.com
> Find best Air France fares, manage your reservations and check in online
> at http://www.airfrance.com
> ------------------------------
> Les données et renseignements contenus dans ce message peuvent être de
> nature confidentielle et soumis au secret professionnel et sont destinés à
> l'usage exclusif du destinataire dont les coordonnées figurent ci-dessus.
> Si vous recevez cette communication par erreur, nous vous demandons de ne
> pas la copier, l'utiliser ou la divulguer. Nous vous prions de notifier
> cette erreur à l'expéditeur et d'effacer immédiatement cette communication
> de votre système. Société Air France - Société anonyme au capital de 1 901
> 231 625 euros - RCS Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747
> Roissy CDG CEDEX
> The data and information contained in this message may be confidential and
> subject to professional secrecy and are intended for the exclusive use of
> the recipient at the address shown above. If you receive this message by
> mistake, we ask you not to copy, use or disclose it. Please notify this
> error to the sender immediately and delete this message from your system.
> Société Air France - Limited company with capital of 1,901,231,625 euros -
> Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95
> 747 Roissy CDG CEDEX
> ------------------------------
> Pensez à l'environnement avant d'imprimer ce message.
> Think of the environment before printing this mail.
>
>
SQOOP AVRO ORAOOP some questions
Posted by Antoine Pieretti <an...@airfrance.fr>.
Hi folks,
i am a newbie within Sqoop and Avro.
I try to manipulate data from an Oracle Database, import, and export Avro
files.
The import works well, the export (INSERT) also, but when i am trying to
Update data from an Avro file to the DB the job failed.
Here my command line : export --connect jdbc:oracle:thin:@qxxxxxxx
--username xxxxxxxxxxxxxxxxx_O --password xxxxxxxxxxx_O --table MY_TABLE
--verbose --export-dir D:\\tmp\extraction\Rep --update-key My_COLUMN_KEY.
The job calls the TextExportMapper (not the AvropExportMapper), and try to
parse the file, wich contains the Avro schemas and parse it as a data line.
Is it a common issue ?
Thanks for your help
Antoine PIERETTI
Accédez aux meilleurs tarifs Air France, gérez vos réservations et enregistrez-vous en ligne sur http://www.airfrance.com
Find best Air France fares, manage your reservations and check in online at http://www.airfrance.com Les données et renseignements contenus dans ce message peuvent être de nature confidentielle et soumis au secret professionnel et sont destinés à l'usage exclusif du destinataire dont les coordonnées figurent ci-dessus. Si vous recevez cette communication par erreur, nous vous demandons de ne pas la copier, l'utiliser ou la divulguer. Nous vous prions de notifier cette erreur à l'expéditeur et d'effacer immédiatement cette communication de votre système. Société Air France - Société anonyme au capital de 1 901 231 625 euros - RCS Bobigny (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX
The data and information contained in this message may be confidential and subject to professional secrecy and are intended for the exclusive use of the recipient at the address shown above. If you receive this message by mistake, we ask you not to copy, use or disclose it. Please notify this error to the sender immediately and delete this message from your system. Société Air France - Limited company with capital of 1,901,231,625 euros - Bobigny register of companies (France) 420 495 178 - 45, rue de Paris, 95 747 Roissy CDG CEDEX Pensez à l'environnement avant d'imprimer ce message.
Think of the environment before printing this mail.