You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "luat (JIRA)" <ji...@apache.org> on 2016/12/21 08:05:58 UTC
[jira] [Comment Edited] (SPARK-18941) Spark thrift server, Spark
2.0.2, The "drop table" command doesn't delete the directory associated
with the Hive table (not EXTERNAL table) from the HDFS file system
[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766407#comment-15766407 ]
luat edited comment on SPARK-18941 at 12/21/16 8:05 AM:
--------------------------------------------------------
Hi [~dongjoon],
Thank for your reply.
I download spark 2.0.2 from link http://spark.apache.org/downloads.html
Then, I unzip and start spark thrift server and testing as below steps:
[root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./sbin/start-thriftserver.sh \
> --hiveconf hive.server2.thrift.port=10005 \
> --hiveconf hive.server2.thrift.bind.host=10.30.132.76 \
> --master yarn
[root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./bin/beeline
Beeline version 1.2.1.spark2 by Apache Hive
beeline> !connect jdbc:hive2://10.30.132.76:10005
Connecting to jdbc:hive2://10.30.132.76:10005
Enter username for jdbc:hive2://10.30.132.76:10005: hdfs
Enter password for jdbc:hive2://10.30.132.76:10005:
16/12/21 14:36:54 INFO jdbc.Utils: Supplied authorities: 10.30.132.76:10005
16/12/21 14:36:54 INFO jdbc.Utils: Resolved authority: 10.30.132.76:10005
16/12/21 14:36:54 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://10.30.132.76:10005
Connected to: Spark SQL (version 2.0.2)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://10.30.132.76:10005>
0: jdbc:hive2://10.30.132.76:10005> create table test(a String) stored as orc location '/test';
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.11 seconds)
0: jdbc:hive2://10.30.132.76:10005> insert into test values("abc");
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.294 seconds)
0: jdbc:hive2://10.30.132.76:10005> select * from test;
+------+--+
| a |
+------+--+
| abc |
+------+--+
1 row selected (0.158 seconds)
Checking result on the hdfs file system:
[hdfs@namenode76 ~]$ hdfs dfs -ls /test
Found 3 items
drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4
-rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000
-rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001
Then, I try to drop table:
0: jdbc:hive2://10.30.132.76:10005> drop table test;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.112 seconds)
0: jdbc:hive2://10.30.132.76:10005> select * from test;
Error: org.apache.spark.sql.AnalysisException: Table or view not found: test; line 1 pos 14 (state=,code=0)
0: jdbc:hive2://10.30.132.76:10005>
Checking again result on the hdfs file system:
[hdfs@namenode76 ~]$ hdfs dfs -ls /test
Found 3 items
drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4
-rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000
-rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001
I'm running on HDP 2.5 cluster (Hadoop 2.7.3).
What is wrong with my testing?
was (Author: luatnc):
Hi Dongjoon Hyun,
Thank for your reply.
I download spark 2.0.2 from link http://spark.apache.org/downloads.html
Then, I unzip and start spark thrift server and testing as below steps:
[root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./sbin/start-thriftserver.sh \
> --hiveconf hive.server2.thrift.port=10005 \
> --hiveconf hive.server2.thrift.bind.host=10.30.132.76 \
> --master yarn
[root@namenode76 spark-2.0.2-bin-hadoop2.7]$ ./bin/beeline
Beeline version 1.2.1.spark2 by Apache Hive
beeline> !connect jdbc:hive2://10.30.132.76:10005
Connecting to jdbc:hive2://10.30.132.76:10005
Enter username for jdbc:hive2://10.30.132.76:10005: hdfs
Enter password for jdbc:hive2://10.30.132.76:10005:
16/12/21 14:36:54 INFO jdbc.Utils: Supplied authorities: 10.30.132.76:10005
16/12/21 14:36:54 INFO jdbc.Utils: Resolved authority: 10.30.132.76:10005
16/12/21 14:36:54 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://10.30.132.76:10005
Connected to: Spark SQL (version 2.0.2)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://10.30.132.76:10005>
0: jdbc:hive2://10.30.132.76:10005> create table test(a String) stored as orc location '/test';
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.11 seconds)
0: jdbc:hive2://10.30.132.76:10005> insert into test values("abc");
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.294 seconds)
0: jdbc:hive2://10.30.132.76:10005> select * from test;
+------+--+
| a |
+------+--+
| abc |
+------+--+
1 row selected (0.158 seconds)
Checking result on the hdfs file system:
[hdfs@namenode76 ~]$ hdfs dfs -ls /test
Found 3 items
drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4
-rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000
-rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001
Then, I try to drop table:
0: jdbc:hive2://10.30.132.76:10005> drop table test;
+---------+--+
| Result |
+---------+--+
+---------+--+
No rows selected (0.112 seconds)
0: jdbc:hive2://10.30.132.76:10005> select * from test;
Error: org.apache.spark.sql.AnalysisException: Table or view not found: test; line 1 pos 14 (state=,code=0)
0: jdbc:hive2://10.30.132.76:10005>
Checking again result on the hdfs file system:
[hdfs@namenode76 ~]$ hdfs dfs -ls /test
Found 3 items
drwxr-xr-x - hdfs hdfs 0 2016-12-21 14:37 /test/.hive-staging_hive_2016-12-21_14-37-04_802_1747227594649313497-4
-rwxr-xr-x 3 hdfs hdfs 49 2016-12-21 14:37 /test/part-00000
-rwxr-xr-x 3 hdfs hdfs 219 2016-12-21 14:37 /test/part-00001
I'm running on HDP 2.5 cluster (Hadoop 2.7.3).
What is wrong with my testing?
> Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-18941
> URL: https://issues.apache.org/jira/browse/SPARK-18941
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.0.2
> Reporter: luat
>
> Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org