You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@atlas.apache.org by Reed Villanueva <rv...@ucera.org> on 2019/08/08 01:43:14 UTC
Sqoop not creating entities in Atlas
Using sqoop on hdp 3.1 to import Oracle DB data into HDFS as parquet and
expecting it to appear in atlas as an entity (I assume it should be the
default sqoop_process type). However, after running the sqoop job, nothing
appears in atlas. Sqoop job looks like...
sqoop import \
-Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \
-Dyarn.app.mapreduce.am.log.level=DEBUG \
-Dmapreduce.map.log.level=DEBUG \
-Dmapreduce.reduce.log.level=DEBUG \
-Dmapred.job.name="Ora import table $tablename" \
-Djava.security.egd=file:///dev/urandom \
-Djava.security.egd=file:///dev/urandom \
-Doraoop.timestamp.string=false \
-Dmapreduce.map.max.attempts=10 \
$oracle_cnxn_str \
--as-parquetfile \
--target-dir $importdir \
-query "select a.*, current_date as etl_date from $tablename a
where 1=1 AND \$CONDITIONS" \
--split-by $splitby \
--where "1=1" \
--num-mappers 12 \
--delete-target-dir
Looking in the hdp ambari interface for atlas, I can see that the sqoop
plugin for atlas is configured[image: enter image description here]
<https://i.stack.imgur.com/U7qYp.png>(did not initially do any manual
tweaking to it after installing atlas according to the docs
<https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/installing-ranger/content/ranger_install_ranger_plugin_settings.html>).
Adding the property[image: enter image description here]
<https://i.stack.imgur.com/LRukd.png>based on another Hortonworks post
<http://community.hortonworks.com/answers/52257/view.html> and attempting
the sqoop job again still shows nothing in the atlas UI. Anyone with more
experience with hdp and atlas have any debugging suggestions or fixes?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Sqoop not creating entities in Atlas
Posted by Reed Villanueva <rv...@ucera.org>.
Not totally sure what exactly I'm looking to see, but I think that the hook
libs are already there / symlinked. What I see is...
➜ ~# cd /usr/hdp/current/sqoop-server/lib
➜ lib# ls
aircompressor-0.10.jar
ant-contrib-1.0b3.jar
ant-eclipse-1.0-jvm1.2.jar
atlas-plugin-classloader-1.1.0.3.1.0.0-78.jar
...
sqoop-bridge-shim-1.1.0.3.1.0.0-78.jar
xz-1.5.jar
➜ lib# ls /usr/hdp/current/atlas-server/hook/sqoop/
atlas-plugin-classloader-1.1.0.3.1.0.0-78.jar
atlas-sqoop-plugin-impl
sqoop-bridge-shim-1.1.0.3.1.0.0-78.jar
So it looks like the jars in the dir you recommend symlinking are already
in the location they should be symlinked to.
What specifically in the sqoop logs would you be looking for in for more
info on it hooking into atlas?
Thanks
On Fri, Aug 16, 2019 at 12:45 AM Nixon Rodrigues <
nixon.rodrigues@freestoneinfotech.com> wrote:
> Reed,
>
> Looks like atlas sqoop hook libs are not present while running sqoop job.
> logs while running sqoop job would give more insights about this.
>
> Can you please check if Atlas sqoop hook libs are symlinked to sqoop libs.
>
> ls /usr/hdp/current/sqoop-server/lib .
>
> If atlas hook libs are not linked. create sym link for Atlas sqoop hook
> lib
>
> cd /usr/hdp/current/sqoop-server/lib
> ln -s /usr/hdp/current/atlas-server/hook/sqoop/ .
>
> HTH
>
> Regards,
> Nixon
>
>
>
>
> On Thu, Aug 8, 2019 at 7:15 AM Reed Villanueva <rv...@ucera.org>
> wrote:
>
>> Using sqoop on hdp 3.1 to import Oracle DB data into HDFS as parquet and
>> expecting it to appear in atlas as an entity (I assume it should be the
>> default sqoop_process type). However, after running the sqoop job,
>> nothing appears in atlas. Sqoop job looks like...
>>
>> sqoop import \
>> -Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \
>> -Dyarn.app.mapreduce.am.log.level=DEBUG \
>> -Dmapreduce.map.log.level=DEBUG \
>> -Dmapreduce.reduce.log.level=DEBUG \
>> -Dmapred.job.name="Ora import table $tablename" \
>> -Djava.security.egd=file:///dev/urandom \
>> -Djava.security.egd=file:///dev/urandom \
>> -Doraoop.timestamp.string=false \
>> -Dmapreduce.map.max.attempts=10 \
>> $oracle_cnxn_str \
>> --as-parquetfile \
>> --target-dir $importdir \
>> -query "select a.*, current_date as etl_date from $tablename a where 1=1 AND \$CONDITIONS" \
>> --split-by $splitby \
>> --where "1=1" \
>> --num-mappers 12 \
>> --delete-target-dir
>>
>> Looking in the hdp ambari interface for atlas, I can see that the sqoop
>> plugin for atlas is configured[image: enter image description here]
>> <https://i.stack.imgur.com/U7qYp.png>(did not initially do any manual
>> tweaking to it after installing atlas according to the docs
>> <https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/installing-ranger/content/ranger_install_ranger_plugin_settings.html>).
>> Adding the property[image: enter image description here]
>> <https://i.stack.imgur.com/LRukd.png>based on another Hortonworks post
>> <http://community.hortonworks.com/answers/52257/view.html> and
>> attempting the sqoop job again still shows nothing in the atlas UI. Anyone
>> with more experience with hdp and atlas have any debugging suggestions or
>> fixes?
>>
>> This electronic message is intended only for the named
>> recipient, and may contain information that is confidential or
>> privileged. If you are not the intended recipient, you are
>> hereby notified that any disclosure, copying, distribution or
>> use of the contents of this message is strictly prohibited. If
>> you have received this message in error or are not the named
>> recipient, please notify us immediately by contacting the
>> sender at the electronic mail address noted above, and delete
>> and destroy all copies of this message. Thank you.
>>
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: Sqoop not creating entities in Atlas
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
Reed,
Looks like atlas sqoop hook libs are not present while running sqoop job.
logs while running sqoop job would give more insights about this.
Can you please check if Atlas sqoop hook libs are symlinked to sqoop libs.
ls /usr/hdp/current/sqoop-server/lib .
If atlas hook libs are not linked. create sym link for Atlas sqoop hook lib
cd /usr/hdp/current/sqoop-server/lib
ln -s /usr/hdp/current/atlas-server/hook/sqoop/ .
HTH
Regards,
Nixon
On Thu, Aug 8, 2019 at 7:15 AM Reed Villanueva <rv...@ucera.org>
wrote:
> Using sqoop on hdp 3.1 to import Oracle DB data into HDFS as parquet and
> expecting it to appear in atlas as an entity (I assume it should be the
> default sqoop_process type). However, after running the sqoop job,
> nothing appears in atlas. Sqoop job looks like...
>
> sqoop import \
> -Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m \
> -Dyarn.app.mapreduce.am.log.level=DEBUG \
> -Dmapreduce.map.log.level=DEBUG \
> -Dmapreduce.reduce.log.level=DEBUG \
> -Dmapred.job.name="Ora import table $tablename" \
> -Djava.security.egd=file:///dev/urandom \
> -Djava.security.egd=file:///dev/urandom \
> -Doraoop.timestamp.string=false \
> -Dmapreduce.map.max.attempts=10 \
> $oracle_cnxn_str \
> --as-parquetfile \
> --target-dir $importdir \
> -query "select a.*, current_date as etl_date from $tablename a where 1=1 AND \$CONDITIONS" \
> --split-by $splitby \
> --where "1=1" \
> --num-mappers 12 \
> --delete-target-dir
>
> Looking in the hdp ambari interface for atlas, I can see that the sqoop
> plugin for atlas is configured[image: enter image description here]
> <https://i.stack.imgur.com/U7qYp.png>(did not initially do any manual
> tweaking to it after installing atlas according to the docs
> <https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/installing-ranger/content/ranger_install_ranger_plugin_settings.html>).
> Adding the property[image: enter image description here]
> <https://i.stack.imgur.com/LRukd.png>based on another Hortonworks post
> <http://community.hortonworks.com/answers/52257/view.html> and attempting
> the sqoop job again still shows nothing in the atlas UI. Anyone with more
> experience with hdp and atlas have any debugging suggestions or fixes?
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
>