Copying non-jar files to the job

[Kristopher Kane <kk...@gmail.com>]

Does anyone have a pointer to how I can copy non-jar files from a
storage handler such that they are accessible by the map task executor
in usercache?

Thanks,

Kris

Re: Copying non-jar files to the job

[Kristopher Kane <kk...@gmail.com>]

'usercache' Might not be completely accurate.

I need JKS files to be accessible to a Kafka consumer from HDFS. In
the 'launcher_container.sh' I see even the Storage Handler itself
getting a link.  How can I get non-jar files here such they are
basically in the directory of the launching main class jar?

echo "Setting up job resources"
ln -sf "/grid/0/hadoop/yarn/local/usercache/<user>/filecache/55/kafka-handler-4.0.0-SNAPSHOT.jar"
"kafka-handler-4.0.0-SNAPSHOT.jar"

I have attempted to use:

// Should be hdfs://
Job job = Job.getInstance(this.configuration);
job.addCacheFile(new URI("hdfs://tmp/my.truststore"));

.. and the Distributed Cache directly but I do not see them in the
directly listing of a Tez log.

On Tue, Aug 6, 2019 at 1:44 PM Kristopher Kane <kk...@gmail.com> wrote:
>
> Does anyone have a pointer to how I can copy non-jar files from a
> storage handler such that they are accessible by the map task executor
> in usercache?
>
> Thanks,
>
> Kris