adding a local jar for UDF test

[Alex Bohr <al...@gradientx.com>]

Hi,
I'm developing a new UDF.
I want to be able to test the new jar as I develop without having to copy
the Jar up to HDFS every time I change code and recompile the Jar.

I'm using Hive CLI for testing, and adding this command:
add jar 'file:///home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';

I've also used variations on the local file system protocol:
add jar 'file://home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';
add jar 'file:/home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';
add jar '/home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';

but keep getting this error:
Query returned non-zero code: 1,
cause: '/home/abohr/test/hive-udf-1.0-SNAPSHOT.jar' does not exist.

Is it not possible to test UDFs with a jar on the local filesystem?
I don't want to add the jar to our configured aux-jars path that
HiveServer2 reads from until I'm done testing.


Thanks

Re: adding a local jar for UDF test

[Alexander Pivovarov <ap...@gmail.com>]

I can suggest 3 options

1. you can use JUnit test to test your UDF (e.g. TestGenericUDFLastDay)

2. you can create q file and test your UDF via mvn (look at udf_last_day.q)
mvn clean install -DskipTests -Phadoop-2
cd itest/qtest
mvn test -Dtest=TestCliDriver -Dqfile=udf_last_day.q
-Dtest.output.overwrite=true -Phadoop-2

3. you can build hive with dist profile and run hive cli locally
mvn clean install -DskipTests -Phadoop-2,dist
cd
packaging/target/apache-hive-1.2.0-SNAPSHOT-bin/apache-hive-1.2.0-SNAPSHOT-bin/
bin/hive

Maybe you can run hive using local engine but I decided to have hdfs and
yarn running locally

After you have hdfs and yarn running locally you can create hive-site.xml
similar to the following

sample hive config
$ echo $HIVE_CONF_DIR
/etc/hive/apivovarov-conf

$ cat /etc/hive/apivovarov-conf/hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>hdfs://localhost/apps/apivovarov/warehouse</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:derby:;databaseName=/tmp/apivovarov_metastore_db;create=true</value>
  </property>
</configuration>

On Wed, Apr 1, 2015 at 10:14 AM, Alex Bohr <al...@gradientx.com> wrote:

> Hi,
> I'm developing a new UDF.
> I want to be able to test the new jar as I develop without having to copy
> the Jar up to HDFS every time I change code and recompile the Jar.
>
> I'm using Hive CLI for testing, and adding this command:
> add jar 'file:///home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';
>
> I've also used variations on the local file system protocol:
> add jar 'file://home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';
> add jar 'file:/home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';
> add jar '/home/abohr/test/hive-udf-1.0-SNAPSHOT.jar';
>
> but keep getting this error:
> Query returned non-zero code: 1,
> cause: '/home/abohr/test/hive-udf-1.0-SNAPSHOT.jar' does not exist.
>
> Is it not possible to test UDFs with a jar on the local filesystem?
> I don't want to add the jar to our configured aux-jars path that
> HiveServer2 reads from until I'm done testing.
>
>
> Thanks
>