You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Suraj Nayak <sn...@gmail.com> on 2015/07/16 03:42:01 UTC
Binary data transfer using Sqoop
Hi Sqoop Users and Developers,
How can i import a Binary data column in a table into HDFS without
converting it into String.
I have encrypted data in RDBMS, I need to import this column as is without
converting it into string. As of now, Sqoop is typecasting the data into
String/text and decryption is failing in Hadoop.
Can someone provide pointers to solve this? Any workaround?
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Importing the data via Sqoop as avro solved this problem.
Thanks for the help!
On Fri, Jul 17, 2015 at 2:58 AM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Abe,
>
> Yes, I had a large *_lob/large_obj_task_1436909150341_0086_m_0000000.lob* file
> with many blobs combined. The actual records had start offset and size
> assigned so I can parse it and read it. This is one way I can solve this.
>
> As you suggested, I tried Avro. It seems to be working(atleast am not
> seeing strings). I will run the decryption job soon and inform if the
> problem is solved.
>
> Thanks a lot for your help and quick replies! It helped a lot for solving
> this.
>
> On Thu, Jul 16, 2015 at 11:30 AM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> You should be able to view the file:
>> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*.
>>
>> Also, you can try the Avro or Parquet data formats for inline forms.
>>
>> Hopefully one of those options work.
>>
>> On Thu, Jul 16, 2015 at 2:51 AM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*
>>
>>
>>
>>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Importing the data via Sqoop as avro solved this problem.
Thanks for the help!
On Fri, Jul 17, 2015 at 2:58 AM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Abe,
>
> Yes, I had a large *_lob/large_obj_task_1436909150341_0086_m_0000000.lob* file
> with many blobs combined. The actual records had start offset and size
> assigned so I can parse it and read it. This is one way I can solve this.
>
> As you suggested, I tried Avro. It seems to be working(atleast am not
> seeing strings). I will run the decryption job soon and inform if the
> problem is solved.
>
> Thanks a lot for your help and quick replies! It helped a lot for solving
> this.
>
> On Thu, Jul 16, 2015 at 11:30 AM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> You should be able to view the file:
>> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*.
>>
>> Also, you can try the Avro or Parquet data formats for inline forms.
>>
>> Hopefully one of those options work.
>>
>> On Thu, Jul 16, 2015 at 2:51 AM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*
>>
>>
>>
>>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Yes, I had a large *_lob/large_obj_task_1436909150341_0086_m_0000000.lob* file
with many blobs combined. The actual records had start offset and size
assigned so I can parse it and read it. This is one way I can solve this.
As you suggested, I tried Avro. It seems to be working(atleast am not
seeing strings). I will run the decryption job soon and inform if the
problem is solved.
Thanks a lot for your help and quick replies! It helped a lot for solving
this.
On Thu, Jul 16, 2015 at 11:30 AM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> You should be able to view the file:
> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*.
>
> Also, you can try the Avro or Parquet data formats for inline forms.
>
> Hopefully one of those options work.
>
> On Thu, Jul 16, 2015 at 2:51 AM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*
>
>
>
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Yes, I had a large *_lob/large_obj_task_1436909150341_0086_m_0000000.lob* file
with many blobs combined. The actual records had start offset and size
assigned so I can parse it and read it. This is one way I can solve this.
As you suggested, I tried Avro. It seems to be working(atleast am not
seeing strings). I will run the decryption job soon and inform if the
problem is solved.
Thanks a lot for your help and quick replies! It helped a lot for solving
this.
On Thu, Jul 16, 2015 at 11:30 AM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> You should be able to view the file:
> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*.
>
> Also, you can try the Avro or Parquet data formats for inline forms.
>
> Hopefully one of those options work.
>
> On Thu, Jul 16, 2015 at 2:51 AM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*
>
>
>
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
You should be able to view the file:
*_lob/large_obj_task_1436909150341_0086_m_0000000.lob*.
Also, you can try the Avro or Parquet data formats for inline forms.
Hopefully one of those options work.
On Thu, Jul 16, 2015 at 2:51 AM, Suraj Nayak <sn...@gmail.com> wrote:
> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
You should be able to view the file:
*_lob/large_obj_task_1436909150341_0086_m_0000000.lob*.
Also, you can try the Avro or Parquet data formats for inline forms.
Hopefully one of those options work.
On Thu, Jul 16, 2015 at 2:51 AM, Suraj Nayak <sn...@gmail.com> wrote:
> *_lob/large_obj_task_1436909150341_0086_m_0000000.lob*
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Thanks for your input. With --inline-lob-limit 0, am able to get the data
as BLOB in external store
*_lob/large_obj_task_1436909150341_0086_m_0000000.lob*. The record
contain *externalLob(lf,_lob/large_obj_task_1436909150341_0086_m_0000000.lob,284,18)
*as data in the column. How can I read this?
Also, is there a way to get the actual data instead of this kind of
external reference? The size of the encrypted column is very small. I think
should not be more than 128 bytes.
Thanks
Suraj
On Wed, Jul 15, 2015 at 10:02 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_large_objects
>
> use --inline-lob-limit 0.
>
> -Abe
>
> On Wed, Jul 15, 2015 at 9:22 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Hi All,
>>
>> I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using
>> --query "select CAST(email as BLOB) from table_name where \$CONDITIONS",
>> and --map-column-java org.apache.hadoop.io.BytesWritable. Still the same
>> issue.
>>
>>
>> Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
>> Required output : [image: Inline image 1]
>>
>>
>>
>>
>>
>> On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>>>
>>> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>>>
>>>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>>
>>>>> Hi Abe,
>>>>>
>>>>> I think the 1st option suits my use case best. But, what is the
>>>>> method/way to tell Sqoop to use Large Object Loading?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey man,
>>>>>>
>>>>>> I think Sqoop does this intentionally:
>>>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>>>> .
>>>>>>
>>>>>> I see two things that may get you beyond this point:
>>>>>>
>>>>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>>>> Object Loading.
>>>>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Abe,
>>>>>>>
>>>>>>> Thanks for highlighting missing required info quickly. Below are the
>>>>>>> details:
>>>>>>>
>>>>>>> - *Version:* Sqoop 1.4.5
>>>>>>> - *Sqoop Command: *sqoop import --connect
>>>>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>>>> com.teradata.jdbc.TeraDriver
>>>>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>>>> email:binary, column5:varchar.
>>>>>>>
>>>>>>> Binary is used as the data is encrypted.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey man,
>>>>>>>>
>>>>>>>> Need some details to help:
>>>>>>>>
>>>>>>>> - What version of Sqoop?
>>>>>>>> - Sqoop command?
>>>>>>>> - Database table structure (preferably a describe on the
>>>>>>>> database)
>>>>>>>>
>>>>>>>> -Abe
>>>>>>>>
>>>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hi Sqoop Users and Developers,
>>>>>>>> >
>>>>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>>>>> > converting it into String.
>>>>>>>> >
>>>>>>>> > I have encrypted data in RDBMS, I need to import this column as
>>>>>>>> is without
>>>>>>>> > converting it into string. As of now, Sqoop is typecasting the
>>>>>>>> data into
>>>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>>>> >
>>>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Thanks
>>>>>>>> > Suraj Nayak M
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Suraj Nayak M
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Suraj Nayak M
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Thanks for your input. With --inline-lob-limit 0, am able to get the data
as BLOB in external store
*_lob/large_obj_task_1436909150341_0086_m_0000000.lob*. The record
contain *externalLob(lf,_lob/large_obj_task_1436909150341_0086_m_0000000.lob,284,18)
*as data in the column. How can I read this?
Also, is there a way to get the actual data instead of this kind of
external reference? The size of the encrypted column is very small. I think
should not be more than 128 bytes.
Thanks
Suraj
On Wed, Jul 15, 2015 at 10:02 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_large_objects
>
> use --inline-lob-limit 0.
>
> -Abe
>
> On Wed, Jul 15, 2015 at 9:22 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Hi All,
>>
>> I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using
>> --query "select CAST(email as BLOB) from table_name where \$CONDITIONS",
>> and --map-column-java org.apache.hadoop.io.BytesWritable. Still the same
>> issue.
>>
>>
>> Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
>> Required output : [image: Inline image 1]
>>
>>
>>
>>
>>
>> On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>>>
>>> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>>>
>>>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>>
>>>>> Hi Abe,
>>>>>
>>>>> I think the 1st option suits my use case best. But, what is the
>>>>> method/way to tell Sqoop to use Large Object Loading?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey man,
>>>>>>
>>>>>> I think Sqoop does this intentionally:
>>>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>>>> .
>>>>>>
>>>>>> I see two things that may get you beyond this point:
>>>>>>
>>>>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>>>> Object Loading.
>>>>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Abe,
>>>>>>>
>>>>>>> Thanks for highlighting missing required info quickly. Below are the
>>>>>>> details:
>>>>>>>
>>>>>>> - *Version:* Sqoop 1.4.5
>>>>>>> - *Sqoop Command: *sqoop import --connect
>>>>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>>>> com.teradata.jdbc.TeraDriver
>>>>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>>>> email:binary, column5:varchar.
>>>>>>>
>>>>>>> Binary is used as the data is encrypted.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey man,
>>>>>>>>
>>>>>>>> Need some details to help:
>>>>>>>>
>>>>>>>> - What version of Sqoop?
>>>>>>>> - Sqoop command?
>>>>>>>> - Database table structure (preferably a describe on the
>>>>>>>> database)
>>>>>>>>
>>>>>>>> -Abe
>>>>>>>>
>>>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hi Sqoop Users and Developers,
>>>>>>>> >
>>>>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>>>>> > converting it into String.
>>>>>>>> >
>>>>>>>> > I have encrypted data in RDBMS, I need to import this column as
>>>>>>>> is without
>>>>>>>> > converting it into string. As of now, Sqoop is typecasting the
>>>>>>>> data into
>>>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>>>> >
>>>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Thanks
>>>>>>>> > Suraj Nayak M
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks
>>>>>>> Suraj Nayak M
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Suraj Nayak M
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_large_objects
use --inline-lob-limit 0.
-Abe
On Wed, Jul 15, 2015 at 9:22 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi All,
>
> I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using --query
> "select CAST(email as BLOB) from table_name where \$CONDITIONS", and
> --map-column-java org.apache.hadoop.io.BytesWritable. Still the same
> issue.
>
>
> Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
> Required output : [image: Inline image 1]
>
>
>
>
>
> On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>>
>> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>>
>>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>>> Hi Abe,
>>>>
>>>> I think the 1st option suits my use case best. But, what is the
>>>> method/way to tell Sqoop to use Large Object Loading?
>>>>
>>>> Thanks!
>>>>
>>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey man,
>>>>>
>>>>> I think Sqoop does this intentionally:
>>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>>> .
>>>>>
>>>>> I see two things that may get you beyond this point:
>>>>>
>>>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>>> Object Loading.
>>>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>>>
>>>>> -Abe
>>>>>
>>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Abe,
>>>>>>
>>>>>> Thanks for highlighting missing required info quickly. Below are the
>>>>>> details:
>>>>>>
>>>>>> - *Version:* Sqoop 1.4.5
>>>>>> - *Sqoop Command: *sqoop import --connect
>>>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>>> com.teradata.jdbc.TeraDriver
>>>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>>> email:binary, column5:varchar.
>>>>>>
>>>>>> Binary is used as the data is encrypted.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey man,
>>>>>>>
>>>>>>> Need some details to help:
>>>>>>>
>>>>>>> - What version of Sqoop?
>>>>>>> - Sqoop command?
>>>>>>> - Database table structure (preferably a describe on the database)
>>>>>>>
>>>>>>> -Abe
>>>>>>>
>>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi Sqoop Users and Developers,
>>>>>>> >
>>>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>>>> > converting it into String.
>>>>>>> >
>>>>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>>>>> without
>>>>>>> > converting it into string. As of now, Sqoop is typecasting the
>>>>>>> data into
>>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>>> >
>>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>>> >
>>>>>>> > --
>>>>>>> > Thanks
>>>>>>> > Suraj Nayak M
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks
>>>>>> Suraj Nayak M
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_large_objects
use --inline-lob-limit 0.
-Abe
On Wed, Jul 15, 2015 at 9:22 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi All,
>
> I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using --query
> "select CAST(email as BLOB) from table_name where \$CONDITIONS", and
> --map-column-java org.apache.hadoop.io.BytesWritable. Still the same
> issue.
>
>
> Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
> Required output : [image: Inline image 1]
>
>
>
>
>
> On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>>
>> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>>
>>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>>> Hi Abe,
>>>>
>>>> I think the 1st option suits my use case best. But, what is the
>>>> method/way to tell Sqoop to use Large Object Loading?
>>>>
>>>> Thanks!
>>>>
>>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey man,
>>>>>
>>>>> I think Sqoop does this intentionally:
>>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>>> .
>>>>>
>>>>> I see two things that may get you beyond this point:
>>>>>
>>>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>>> Object Loading.
>>>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>>>
>>>>> -Abe
>>>>>
>>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Abe,
>>>>>>
>>>>>> Thanks for highlighting missing required info quickly. Below are the
>>>>>> details:
>>>>>>
>>>>>> - *Version:* Sqoop 1.4.5
>>>>>> - *Sqoop Command: *sqoop import --connect
>>>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>>> com.teradata.jdbc.TeraDriver
>>>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>>> email:binary, column5:varchar.
>>>>>>
>>>>>> Binary is used as the data is encrypted.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey man,
>>>>>>>
>>>>>>> Need some details to help:
>>>>>>>
>>>>>>> - What version of Sqoop?
>>>>>>> - Sqoop command?
>>>>>>> - Database table structure (preferably a describe on the database)
>>>>>>>
>>>>>>> -Abe
>>>>>>>
>>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi Sqoop Users and Developers,
>>>>>>> >
>>>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>>>> > converting it into String.
>>>>>>> >
>>>>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>>>>> without
>>>>>>> > converting it into string. As of now, Sqoop is typecasting the
>>>>>>> data into
>>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>>> >
>>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>>> >
>>>>>>> > --
>>>>>>> > Thanks
>>>>>>> > Suraj Nayak M
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks
>>>>>> Suraj Nayak M
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi All,
I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using --query
"select CAST(email as BLOB) from table_name where \$CONDITIONS", and
--map-column-java org.apache.hadoop.io.BytesWritable. Still the same issue.
Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
Required output : [image: Inline image 1]
On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <sn...@gmail.com> wrote:
> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>
> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>
>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> Hi Abe,
>>>
>>> I think the 1st option suits my use case best. But, what is the
>>> method/way to tell Sqoop to use Large Object Loading?
>>>
>>> Thanks!
>>>
>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>>> wrote:
>>>
>>>> Hey man,
>>>>
>>>> I think Sqoop does this intentionally:
>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>> .
>>>>
>>>> I see two things that may get you beyond this point:
>>>>
>>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>> Object Loading.
>>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>>
>>>> -Abe
>>>>
>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>>
>>>>> Hi Abe,
>>>>>
>>>>> Thanks for highlighting missing required info quickly. Below are the
>>>>> details:
>>>>>
>>>>> - *Version:* Sqoop 1.4.5
>>>>> - *Sqoop Command: *sqoop import --connect
>>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>> com.teradata.jdbc.TeraDriver
>>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>> email:binary, column5:varchar.
>>>>>
>>>>> Binary is used as the data is encrypted.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey man,
>>>>>>
>>>>>> Need some details to help:
>>>>>>
>>>>>> - What version of Sqoop?
>>>>>> - Sqoop command?
>>>>>> - Database table structure (preferably a describe on the database)
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi Sqoop Users and Developers,
>>>>>> >
>>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>>> > converting it into String.
>>>>>> >
>>>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>>>> without
>>>>>> > converting it into string. As of now, Sqoop is typecasting the data
>>>>>> into
>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>> >
>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>> >
>>>>>> > --
>>>>>> > Thanks
>>>>>> > Suraj Nayak M
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Suraj Nayak M
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi All,
I tried --map-column-java com.cloudera.sqoop.lib.BlobRef by using --query
"select CAST(email as BLOB) from table_name where \$CONDITIONS", and
--map-column-java org.apache.hadoop.io.BytesWritable. Still the same issue.
Sqoop Output: 20 16 3a 34 fa 40 f6 fc 24 3d 5f b9 c6 63 b5 5f 54 77
Required output : [image: Inline image 1]
On Wed, Jul 15, 2015 at 9:04 PM, Suraj Nayak <sn...@gmail.com> wrote:
> One correction in Sqoop version. Its 1.4.4 not 1.4.5.
>
> On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>>
>> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> Hi Abe,
>>>
>>> I think the 1st option suits my use case best. But, what is the
>>> method/way to tell Sqoop to use Large Object Loading?
>>>
>>> Thanks!
>>>
>>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>>> wrote:
>>>
>>>> Hey man,
>>>>
>>>> I think Sqoop does this intentionally:
>>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>>> .
>>>>
>>>> I see two things that may get you beyond this point:
>>>>
>>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>>> Object Loading.
>>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>>
>>>> -Abe
>>>>
>>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>>
>>>>> Hi Abe,
>>>>>
>>>>> Thanks for highlighting missing required info quickly. Below are the
>>>>> details:
>>>>>
>>>>> - *Version:* Sqoop 1.4.5
>>>>> - *Sqoop Command: *sqoop import --connect
>>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>>> com.teradata.jdbc.TeraDriver
>>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>>> email:binary, column5:varchar.
>>>>>
>>>>> Binary is used as the data is encrypted.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hey man,
>>>>>>
>>>>>> Need some details to help:
>>>>>>
>>>>>> - What version of Sqoop?
>>>>>> - Sqoop command?
>>>>>> - Database table structure (preferably a describe on the database)
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi Sqoop Users and Developers,
>>>>>> >
>>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>>> > converting it into String.
>>>>>> >
>>>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>>>> without
>>>>>> > converting it into string. As of now, Sqoop is typecasting the data
>>>>>> into
>>>>>> > String/text and decryption is failing in Hadoop.
>>>>>> >
>>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>>> >
>>>>>> > --
>>>>>> > Thanks
>>>>>> > Suraj Nayak M
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Suraj Nayak M
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
One correction in Sqoop version. Its 1.4.4 not 1.4.5.
On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>
> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Hi Abe,
>>
>> I think the 1st option suits my use case best. But, what is the
>> method/way to tell Sqoop to use Large Object Loading?
>>
>> Thanks!
>>
>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>> wrote:
>>
>>> Hey man,
>>>
>>> I think Sqoop does this intentionally:
>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>> .
>>>
>>> I see two things that may get you beyond this point:
>>>
>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>> Object Loading.
>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>
>>> -Abe
>>>
>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>>> Hi Abe,
>>>>
>>>> Thanks for highlighting missing required info quickly. Below are the
>>>> details:
>>>>
>>>> - *Version:* Sqoop 1.4.5
>>>> - *Sqoop Command: *sqoop import --connect
>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>> com.teradata.jdbc.TeraDriver
>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>> email:binary, column5:varchar.
>>>>
>>>> Binary is used as the data is encrypted.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey man,
>>>>>
>>>>> Need some details to help:
>>>>>
>>>>> - What version of Sqoop?
>>>>> - Sqoop command?
>>>>> - Database table structure (preferably a describe on the database)
>>>>>
>>>>> -Abe
>>>>>
>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi Sqoop Users and Developers,
>>>>> >
>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>> > converting it into String.
>>>>> >
>>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>>> without
>>>>> > converting it into string. As of now, Sqoop is typecasting the data
>>>>> into
>>>>> > String/text and decryption is failing in Hadoop.
>>>>> >
>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>> >
>>>>> > --
>>>>> > Thanks
>>>>> > Suraj Nayak M
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
One correction in Sqoop version. Its 1.4.4 not 1.4.5.
On Wed, Jul 15, 2015 at 8:56 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
>
> On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Hi Abe,
>>
>> I think the 1st option suits my use case best. But, what is the
>> method/way to tell Sqoop to use Large Object Loading?
>>
>> Thanks!
>>
>> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
>> wrote:
>>
>>> Hey man,
>>>
>>> I think Sqoop does this intentionally:
>>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>>> .
>>>
>>> I see two things that may get you beyond this point:
>>>
>>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>>> Object Loading.
>>> 2. Try the --as-avrodatafile option and write as Avro.
>>>
>>> -Abe
>>>
>>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>>> Hi Abe,
>>>>
>>>> Thanks for highlighting missing required info quickly. Below are the
>>>> details:
>>>>
>>>> - *Version:* Sqoop 1.4.5
>>>> - *Sqoop Command: *sqoop import --connect
>>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>>> com.teradata.jdbc.TeraDriver
>>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>>> email:binary, column5:varchar.
>>>>
>>>> Binary is used as the data is encrypted.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey man,
>>>>>
>>>>> Need some details to help:
>>>>>
>>>>> - What version of Sqoop?
>>>>> - Sqoop command?
>>>>> - Database table structure (preferably a describe on the database)
>>>>>
>>>>> -Abe
>>>>>
>>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi Sqoop Users and Developers,
>>>>> >
>>>>> > How can i import a Binary data column in a table into HDFS without
>>>>> > converting it into String.
>>>>> >
>>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>>> without
>>>>> > converting it into string. As of now, Sqoop is typecasting the data
>>>>> into
>>>>> > String/text and decryption is failing in Hadoop.
>>>>> >
>>>>> > Can someone provide pointers to solve this? Any workaround?
>>>>> >
>>>>> > --
>>>>> > Thanks
>>>>> > Suraj Nayak M
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Suraj Nayak M
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Abe,
>
> I think the 1st option suits my use case best. But, what is the method/way
> to tell Sqoop to use Large Object Loading?
>
> Thanks!
>
> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Hey man,
>>
>> I think Sqoop does this intentionally:
>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>> .
>>
>> I see two things that may get you beyond this point:
>>
>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>> Object Loading.
>> 2. Try the --as-avrodatafile option and write as Avro.
>>
>> -Abe
>>
>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> Hi Abe,
>>>
>>> Thanks for highlighting missing required info quickly. Below are the
>>> details:
>>>
>>> - *Version:* Sqoop 1.4.5
>>> - *Sqoop Command: *sqoop import --connect
>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>> com.teradata.jdbc.TeraDriver
>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>> email:binary, column5:varchar.
>>>
>>> Binary is used as the data is encrypted.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>> wrote:
>>>
>>>> Hey man,
>>>>
>>>> Need some details to help:
>>>>
>>>> - What version of Sqoop?
>>>> - Sqoop command?
>>>> - Database table structure (preferably a describe on the database)
>>>>
>>>> -Abe
>>>>
>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>>
>>>> > Hi Sqoop Users and Developers,
>>>> >
>>>> > How can i import a Binary data column in a table into HDFS without
>>>> > converting it into String.
>>>> >
>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>> without
>>>> > converting it into string. As of now, Sqoop is typecasting the data
>>>> into
>>>> > String/text and decryption is failing in Hadoop.
>>>> >
>>>> > Can someone provide pointers to solve this? Any workaround?
>>>> >
>>>> > --
>>>> > Thanks
>>>> > Suraj Nayak M
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Is it by setting --map-column-java email=com.cloudera.sqoop.lib.BlobRef?
On Wed, Jul 15, 2015 at 8:11 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Abe,
>
> I think the 1st option suits my use case best. But, what is the method/way
> to tell Sqoop to use Large Object Loading?
>
> Thanks!
>
> On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Hey man,
>>
>> I think Sqoop does this intentionally:
>> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
>> .
>>
>> I see two things that may get you beyond this point:
>>
>> 1. You can try type casting to a BLOB and tell Sqoop to use Large
>> Object Loading.
>> 2. Try the --as-avrodatafile option and write as Avro.
>>
>> -Abe
>>
>> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>>> Hi Abe,
>>>
>>> Thanks for highlighting missing required info quickly. Below are the
>>> details:
>>>
>>> - *Version:* Sqoop 1.4.5
>>> - *Sqoop Command: *sqoop import --connect
>>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>>> com.teradata.jdbc.TeraDriver
>>> - *Table structure:* id:varchar, count:int, first_name:binary,
>>> email:binary, column5:varchar.
>>>
>>> Binary is used as the data is encrypted.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>>> wrote:
>>>
>>>> Hey man,
>>>>
>>>> Need some details to help:
>>>>
>>>> - What version of Sqoop?
>>>> - Sqoop command?
>>>> - Database table structure (preferably a describe on the database)
>>>>
>>>> -Abe
>>>>
>>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>>
>>>> > Hi Sqoop Users and Developers,
>>>> >
>>>> > How can i import a Binary data column in a table into HDFS without
>>>> > converting it into String.
>>>> >
>>>> > I have encrypted data in RDBMS, I need to import this column as is
>>>> without
>>>> > converting it into string. As of now, Sqoop is typecasting the data
>>>> into
>>>> > String/text and decryption is failing in Hadoop.
>>>> >
>>>> > Can someone provide pointers to solve this? Any workaround?
>>>> >
>>>> > --
>>>> > Thanks
>>>> > Suraj Nayak M
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Suraj Nayak M
>>>
>>
>>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
I think the 1st option suits my use case best. But, what is the method/way
to tell Sqoop to use Large Object Loading?
Thanks!
On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Hey man,
>
> I think Sqoop does this intentionally:
> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
> .
>
> I see two things that may get you beyond this point:
>
> 1. You can try type casting to a BLOB and tell Sqoop to use Large
> Object Loading.
> 2. Try the --as-avrodatafile option and write as Avro.
>
> -Abe
>
> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Hi Abe,
>>
>> Thanks for highlighting missing required info quickly. Below are the
>> details:
>>
>> - *Version:* Sqoop 1.4.5
>> - *Sqoop Command: *sqoop import --connect
>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>> com.teradata.jdbc.TeraDriver
>> - *Table structure:* id:varchar, count:int, first_name:binary,
>> email:binary, column5:varchar.
>>
>> Binary is used as the data is encrypted.
>>
>> Thanks!
>>
>>
>>
>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>> wrote:
>>
>>> Hey man,
>>>
>>> Need some details to help:
>>>
>>> - What version of Sqoop?
>>> - Sqoop command?
>>> - Database table structure (preferably a describe on the database)
>>>
>>> -Abe
>>>
>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>> > Hi Sqoop Users and Developers,
>>> >
>>> > How can i import a Binary data column in a table into HDFS without
>>> > converting it into String.
>>> >
>>> > I have encrypted data in RDBMS, I need to import this column as is
>>> without
>>> > converting it into string. As of now, Sqoop is typecasting the data
>>> into
>>> > String/text and decryption is failing in Hadoop.
>>> >
>>> > Can someone provide pointers to solve this? Any workaround?
>>> >
>>> > --
>>> > Thanks
>>> > Suraj Nayak M
>>> >
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
I think the 1st option suits my use case best. But, what is the method/way
to tell Sqoop to use Large Object Loading?
Thanks!
On Wed, Jul 15, 2015 at 7:50 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Hey man,
>
> I think Sqoop does this intentionally:
> https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
> .
>
> I see two things that may get you beyond this point:
>
> 1. You can try type casting to a BLOB and tell Sqoop to use Large
> Object Loading.
> 2. Try the --as-avrodatafile option and write as Avro.
>
> -Abe
>
> On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
>> Hi Abe,
>>
>> Thanks for highlighting missing required info quickly. Below are the
>> details:
>>
>> - *Version:* Sqoop 1.4.5
>> - *Sqoop Command: *sqoop import --connect
>> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
>> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
>> 1 --target-dir /tmp/ENC_TBL --connection-manager
>> "org.apache.sqoop.manager.GenericJdbcManager" --driver
>> com.teradata.jdbc.TeraDriver
>> - *Table structure:* id:varchar, count:int, first_name:binary,
>> email:binary, column5:varchar.
>>
>> Binary is used as the data is encrypted.
>>
>> Thanks!
>>
>>
>>
>> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
>> wrote:
>>
>>> Hey man,
>>>
>>> Need some details to help:
>>>
>>> - What version of Sqoop?
>>> - Sqoop command?
>>> - Database table structure (preferably a describe on the database)
>>>
>>> -Abe
>>>
>>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>>
>>> > Hi Sqoop Users and Developers,
>>> >
>>> > How can i import a Binary data column in a table into HDFS without
>>> > converting it into String.
>>> >
>>> > I have encrypted data in RDBMS, I need to import this column as is
>>> without
>>> > converting it into string. As of now, Sqoop is typecasting the data
>>> into
>>> > String/text and decryption is failing in Hadoop.
>>> >
>>> > Can someone provide pointers to solve this? Any workaround?
>>> >
>>> > --
>>> > Thanks
>>> > Suraj Nayak M
>>> >
>>>
>>
>>
>>
>> --
>> Thanks
>> Suraj Nayak M
>>
>
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey man,
I think Sqoop does this intentionally:
https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
.
I see two things that may get you beyond this point:
1. You can try type casting to a BLOB and tell Sqoop to use Large Object
Loading.
2. Try the --as-avrodatafile option and write as Avro.
-Abe
On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Abe,
>
> Thanks for highlighting missing required info quickly. Below are the
> details:
>
> - *Version:* Sqoop 1.4.5
> - *Sqoop Command: *sqoop import --connect
> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
> 1 --target-dir /tmp/ENC_TBL --connection-manager
> "org.apache.sqoop.manager.GenericJdbcManager" --driver
> com.teradata.jdbc.TeraDriver
> - *Table structure:* id:varchar, count:int, first_name:binary,
> email:binary, column5:varchar.
>
> Binary is used as the data is encrypted.
>
> Thanks!
>
>
>
> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Hey man,
>>
>> Need some details to help:
>>
>> - What version of Sqoop?
>> - Sqoop command?
>> - Database table structure (preferably a describe on the database)
>>
>> -Abe
>>
>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>> > Hi Sqoop Users and Developers,
>> >
>> > How can i import a Binary data column in a table into HDFS without
>> > converting it into String.
>> >
>> > I have encrypted data in RDBMS, I need to import this column as is
>> without
>> > converting it into string. As of now, Sqoop is typecasting the data into
>> > String/text and decryption is failing in Hadoop.
>> >
>> > Can someone provide pointers to solve this? Any workaround?
>> >
>> > --
>> > Thanks
>> > Suraj Nayak M
>> >
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey man,
I think Sqoop does this intentionally:
https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java#L65
.
I see two things that may get you beyond this point:
1. You can try type casting to a BLOB and tell Sqoop to use Large Object
Loading.
2. Try the --as-avrodatafile option and write as Avro.
-Abe
On Wed, Jul 15, 2015 at 7:23 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Abe,
>
> Thanks for highlighting missing required info quickly. Below are the
> details:
>
> - *Version:* Sqoop 1.4.5
> - *Sqoop Command: *sqoop import --connect
> jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
> --password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
> 1 --target-dir /tmp/ENC_TBL --connection-manager
> "org.apache.sqoop.manager.GenericJdbcManager" --driver
> com.teradata.jdbc.TeraDriver
> - *Table structure:* id:varchar, count:int, first_name:binary,
> email:binary, column5:varchar.
>
> Binary is used as the data is encrypted.
>
> Thanks!
>
>
>
> On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com>
> wrote:
>
>> Hey man,
>>
>> Need some details to help:
>>
>> - What version of Sqoop?
>> - Sqoop command?
>> - Database table structure (preferably a describe on the database)
>>
>> -Abe
>>
>> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>>
>> > Hi Sqoop Users and Developers,
>> >
>> > How can i import a Binary data column in a table into HDFS without
>> > converting it into String.
>> >
>> > I have encrypted data in RDBMS, I need to import this column as is
>> without
>> > converting it into string. As of now, Sqoop is typecasting the data into
>> > String/text and decryption is failing in Hadoop.
>> >
>> > Can someone provide pointers to solve this? Any workaround?
>> >
>> > --
>> > Thanks
>> > Suraj Nayak M
>> >
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Thanks for highlighting missing required info quickly. Below are the
details:
- *Version:* Sqoop 1.4.5
- *Sqoop Command: *sqoop import --connect
jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
--password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
1 --target-dir /tmp/ENC_TBL --connection-manager
"org.apache.sqoop.manager.GenericJdbcManager" --driver
com.teradata.jdbc.TeraDriver
- *Table structure:* id:varchar, count:int, first_name:binary,
email:binary, column5:varchar.
Binary is used as the data is encrypted.
Thanks!
On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Hey man,
>
> Need some details to help:
>
> - What version of Sqoop?
> - Sqoop command?
> - Database table structure (preferably a describe on the database)
>
> -Abe
>
> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
> > Hi Sqoop Users and Developers,
> >
> > How can i import a Binary data column in a table into HDFS without
> > converting it into String.
> >
> > I have encrypted data in RDBMS, I need to import this column as is
> without
> > converting it into string. As of now, Sqoop is typecasting the data into
> > String/text and decryption is failing in Hadoop.
> >
> > Can someone provide pointers to solve this? Any workaround?
> >
> > --
> > Thanks
> > Suraj Nayak M
> >
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Abe,
Thanks for highlighting missing required info quickly. Below are the
details:
- *Version:* Sqoop 1.4.5
- *Sqoop Command: *sqoop import --connect
jdbc:teradata://aa.bb.cc.internal/DATABASE=someDB --username sqoop_usr
--password sqoop_usr --table ENCRYPTED_TBL --fields-terminated-by \\001 -m
1 --target-dir /tmp/ENC_TBL --connection-manager
"org.apache.sqoop.manager.GenericJdbcManager" --driver
com.teradata.jdbc.TeraDriver
- *Table structure:* id:varchar, count:int, first_name:binary,
email:binary, column5:varchar.
Binary is used as the data is encrypted.
Thanks!
On Wed, Jul 15, 2015 at 6:44 PM, Abraham Elmahrek <ab...@cloudera.com> wrote:
> Hey man,
>
> Need some details to help:
>
> - What version of Sqoop?
> - Sqoop command?
> - Database table structure (preferably a describe on the database)
>
> -Abe
>
> On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
>
> > Hi Sqoop Users and Developers,
> >
> > How can i import a Binary data column in a table into HDFS without
> > converting it into String.
> >
> > I have encrypted data in RDBMS, I need to import this column as is
> without
> > converting it into string. As of now, Sqoop is typecasting the data into
> > String/text and decryption is failing in Hadoop.
> >
> > Can someone provide pointers to solve this? Any workaround?
> >
> > --
> > Thanks
> > Suraj Nayak M
> >
>
--
Thanks
Suraj Nayak M
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey man,
Need some details to help:
- What version of Sqoop?
- Sqoop command?
- Database table structure (preferably a describe on the database)
-Abe
On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Sqoop Users and Developers,
>
> How can i import a Binary data column in a table into HDFS without
> converting it into String.
>
> I have encrypted data in RDBMS, I need to import this column as is without
> converting it into string. As of now, Sqoop is typecasting the data into
> String/text and decryption is failing in Hadoop.
>
> Can someone provide pointers to solve this? Any workaround?
>
> --
> Thanks
> Suraj Nayak M
>
Re: Binary data transfer using Sqoop
Posted by Abraham Elmahrek <ab...@cloudera.com>.
Hey man,
Need some details to help:
- What version of Sqoop?
- Sqoop command?
- Database table structure (preferably a describe on the database)
-Abe
On Wed, Jul 15, 2015 at 6:42 PM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Sqoop Users and Developers,
>
> How can i import a Binary data column in a table into HDFS without
> converting it into String.
>
> I have encrypted data in RDBMS, I need to import this column as is without
> converting it into string. As of now, Sqoop is typecasting the data into
> String/text and decryption is failing in Hadoop.
>
> Can someone provide pointers to solve this? Any workaround?
>
> --
> Thanks
> Suraj Nayak M
>