You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashok Kumar <as...@yahoo.com.INVALID> on 2016/03/25 23:12:41 UTC
Finding out the time a table was created
Experts,
I would like to know when a table was created in Hive database using Spark shell?
Thanks
Re: Finding out the time a table was created
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Ted,
I moved to Spark 1.6
Still the same issue outstanding
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java
1.7.0_25)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
SQL context available as sqlContext.
scala> sql("describe formatted test.t14").collect.foreach(println)
16/03/26 08:51:23 ERROR Hive: Table test not found: default.test table not
found
16/03/26 08:51:23 ERROR Hive: Table test not found: default.test table not
found
[# col_name data_type comment ]
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 March 2016 at 22:49, Mich Talebzadeh <mi...@gmail.com>
wrote:
> mine is version 1.5.2 Ted
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 25 March 2016 at 22:45, Ted Yu <yu...@gmail.com> wrote:
>
>> Strange: the JIRAs below were marked Fixed in 1.5.0
>>
>> On Fri, Mar 25, 2016 at 3:43 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Is this 1.6 Ted?
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 25 March 2016 at 22:40, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Looks like database support was fixed by:
>>>>
>>>> [SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes
>>>> multi-database support
>>>>
>>>> On Fri, Mar 25, 2016 at 3:35 PM, Ashok Kumar <as...@yahoo.com>
>>>> wrote:
>>>>
>>>>> 1.5.2 Ted.
>>>>>
>>>>> Those two lines I don't know where they come. It finds and gets the
>>>>> table info OK
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>> On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Which release of Spark do you use, Mich ?
>>>>>
>>>>> In master branch, the message is more accurate
>>>>> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
>>>>>
>>>>> override def getMessage: String = s"Table $table not found in
>>>>> database $db"
>>>>>
>>>>>
>>>>> On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that
>>>>> info.
>>>>>
>>>>> This is based on the same command in Hive however, it throws two
>>>>> erroneous error lines as shown below (don't see them in Hive DESCRIBE ...)
>>>>>
>>>>> Example
>>>>>
>>>>> scala> sql("describe formatted test.t14").collect.foreach(println)
>>>>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table
>>>>> not found
>>>>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table
>>>>> not found
>>>>> [# col_name data_type comment ]
>>>>> [ ]
>>>>> [invoicenumber int ]
>>>>> [paymentdate date ]
>>>>> [net decimal(20,2) ]
>>>>> [vat decimal(20,2) ]
>>>>> [total decimal(20,2) ]
>>>>> [ ]
>>>>> [# Detailed Table Information ]
>>>>> [Database: test ]
>>>>> [Owner: hduser ]
>>>>> [
>>>>> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
>>>>> UNKNOWN ]
>>>>> [Protect Mode: None ]
>>>>> [Retention: 0 ]
>>>>> [Location:
>>>>> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
>>>>> [Table Type: MANAGED_TABLE ]
>>>>> [Table Parameters: ]
>>>>> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
>>>>> [ comment from csv file from excel sheet]
>>>>> [ numFiles 2 ]
>>>>> [ orc.compress ZLIB ]
>>>>> [ totalSize 1090 ]
>>>>> [ transient_lastDdlTime 1458944025 ]
>>>>> [ ]
>>>>> [# Storage Information ]
>>>>> [SerDe Library:
>>>>> org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
>>>>> [InputFormat:
>>>>> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
>>>>> [OutputFormat:
>>>>> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
>>>>> [Compressed: No ]
>>>>> [Num Buckets: -1 ]
>>>>> [Bucket Columns: [] ]
>>>>> [Sort Columns: [] ]
>>>>> [Storage Desc Params: ]
>>>>> [ serialization.format 1 ]
>>>>>
>>>>> HTH
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
>>>>> wrote:
>>>>>
>>>>> Experts,
>>>>>
>>>>> I would like to know when a table was created in Hive database using
>>>>> Spark shell?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Finding out the time a table was created
Posted by Mich Talebzadeh <mi...@gmail.com>.
mine is version 1.5.2 Ted
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 March 2016 at 22:45, Ted Yu <yu...@gmail.com> wrote:
> Strange: the JIRAs below were marked Fixed in 1.5.0
>
> On Fri, Mar 25, 2016 at 3:43 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Is this 1.6 Ted?
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 25 March 2016 at 22:40, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Looks like database support was fixed by:
>>>
>>> [SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes
>>> multi-database support
>>>
>>> On Fri, Mar 25, 2016 at 3:35 PM, Ashok Kumar <as...@yahoo.com>
>>> wrote:
>>>
>>>> 1.5.2 Ted.
>>>>
>>>> Those two lines I don't know where they come. It finds and gets the
>>>> table info OK
>>>>
>>>> HTH
>>>>
>>>>
>>>> On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>
>>>> Which release of Spark do you use, Mich ?
>>>>
>>>> In master branch, the message is more accurate
>>>> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
>>>>
>>>> override def getMessage: String = s"Table $table not found in
>>>> database $db"
>>>>
>>>>
>>>> On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
>>>>
>>>> This is based on the same command in Hive however, it throws two
>>>> erroneous error lines as shown below (don't see them in Hive DESCRIBE ...)
>>>>
>>>> Example
>>>>
>>>> scala> sql("describe formatted test.t14").collect.foreach(println)
>>>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
>>>> found
>>>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
>>>> found
>>>> [# col_name data_type comment ]
>>>> [ ]
>>>> [invoicenumber int ]
>>>> [paymentdate date ]
>>>> [net decimal(20,2) ]
>>>> [vat decimal(20,2) ]
>>>> [total decimal(20,2) ]
>>>> [ ]
>>>> [# Detailed Table Information ]
>>>> [Database: test ]
>>>> [Owner: hduser ]
>>>> [
>>>> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
>>>> UNKNOWN ]
>>>> [Protect Mode: None ]
>>>> [Retention: 0 ]
>>>> [Location:
>>>> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
>>>> [Table Type: MANAGED_TABLE ]
>>>> [Table Parameters: ]
>>>> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
>>>> [ comment from csv file from excel sheet]
>>>> [ numFiles 2 ]
>>>> [ orc.compress ZLIB ]
>>>> [ totalSize 1090 ]
>>>> [ transient_lastDdlTime 1458944025 ]
>>>> [ ]
>>>> [# Storage Information ]
>>>> [SerDe Library:
>>>> org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
>>>> [InputFormat:
>>>> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
>>>> [OutputFormat:
>>>> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
>>>> [Compressed: No ]
>>>> [Num Buckets: -1 ]
>>>> [Bucket Columns: [] ]
>>>> [Sort Columns: [] ]
>>>> [Storage Desc Params: ]
>>>> [ serialization.format 1 ]
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
>>>> wrote:
>>>>
>>>> Experts,
>>>>
>>>> I would like to know when a table was created in Hive database using
>>>> Spark shell?
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Finding out the time a table was created
Posted by Ted Yu <yu...@gmail.com>.
Strange: the JIRAs below were marked Fixed in 1.5.0
On Fri, Mar 25, 2016 at 3:43 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Is this 1.6 Ted?
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 25 March 2016 at 22:40, Ted Yu <yu...@gmail.com> wrote:
>
>> Looks like database support was fixed by:
>>
>> [SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes
>> multi-database support
>>
>> On Fri, Mar 25, 2016 at 3:35 PM, Ashok Kumar <as...@yahoo.com>
>> wrote:
>>
>>> 1.5.2 Ted.
>>>
>>> Those two lines I don't know where they come. It finds and gets the
>>> table info OK
>>>
>>> HTH
>>>
>>>
>>> On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>
>>> Which release of Spark do you use, Mich ?
>>>
>>> In master branch, the message is more accurate
>>> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
>>>
>>> override def getMessage: String = s"Table $table not found in database
>>> $db"
>>>
>>>
>>> On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
>>>
>>> This is based on the same command in Hive however, it throws two
>>> erroneous error lines as shown below (don't see them in Hive DESCRIBE ...)
>>>
>>> Example
>>>
>>> scala> sql("describe formatted test.t14").collect.foreach(println)
>>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
>>> found
>>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
>>> found
>>> [# col_name data_type comment ]
>>> [ ]
>>> [invoicenumber int ]
>>> [paymentdate date ]
>>> [net decimal(20,2) ]
>>> [vat decimal(20,2) ]
>>> [total decimal(20,2) ]
>>> [ ]
>>> [# Detailed Table Information ]
>>> [Database: test ]
>>> [Owner: hduser ]
>>> [
>>> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
>>> UNKNOWN ]
>>> [Protect Mode: None ]
>>> [Retention: 0 ]
>>> [Location:
>>> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
>>> [Table Type: MANAGED_TABLE ]
>>> [Table Parameters: ]
>>> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
>>> [ comment from csv file from excel sheet]
>>> [ numFiles 2 ]
>>> [ orc.compress ZLIB ]
>>> [ totalSize 1090 ]
>>> [ transient_lastDdlTime 1458944025 ]
>>> [ ]
>>> [# Storage Information ]
>>> [SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
>>> ]
>>> [InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
>>> ]
>>> [OutputFormat:
>>> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
>>> [Compressed: No ]
>>> [Num Buckets: -1 ]
>>> [Bucket Columns: [] ]
>>> [Sort Columns: [] ]
>>> [Storage Desc Params: ]
>>> [ serialization.format 1 ]
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
>>> wrote:
>>>
>>> Experts,
>>>
>>> I would like to know when a table was created in Hive database using
>>> Spark shell?
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Finding out the time a table was created
Posted by Mich Talebzadeh <mi...@gmail.com>.
Is this 1.6 Ted?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 March 2016 at 22:40, Ted Yu <yu...@gmail.com> wrote:
> Looks like database support was fixed by:
>
> [SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes
> multi-database support
>
> On Fri, Mar 25, 2016 at 3:35 PM, Ashok Kumar <as...@yahoo.com> wrote:
>
>> 1.5.2 Ted.
>>
>> Those two lines I don't know where they come. It finds and gets the table
>> info OK
>>
>> HTH
>>
>>
>> On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
>>
>>
>> Which release of Spark do you use, Mich ?
>>
>> In master branch, the message is more accurate
>> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
>>
>> override def getMessage: String = s"Table $table not found in database
>> $db"
>>
>>
>> On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
>>
>> This is based on the same command in Hive however, it throws two
>> erroneous error lines as shown below (don't see them in Hive DESCRIBE ...)
>>
>> Example
>>
>> scala> sql("describe formatted test.t14").collect.foreach(println)
>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
>> found
>> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
>> found
>> [# col_name data_type comment ]
>> [ ]
>> [invoicenumber int ]
>> [paymentdate date ]
>> [net decimal(20,2) ]
>> [vat decimal(20,2) ]
>> [total decimal(20,2) ]
>> [ ]
>> [# Detailed Table Information ]
>> [Database: test ]
>> [Owner: hduser ]
>> [
>> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
>> UNKNOWN ]
>> [Protect Mode: None ]
>> [Retention: 0 ]
>> [Location:
>> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
>> [Table Type: MANAGED_TABLE ]
>> [Table Parameters: ]
>> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
>> [ comment from csv file from excel sheet]
>> [ numFiles 2 ]
>> [ orc.compress ZLIB ]
>> [ totalSize 1090 ]
>> [ transient_lastDdlTime 1458944025 ]
>> [ ]
>> [# Storage Information ]
>> [SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
>> [InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
>> [OutputFormat:
>> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
>> [Compressed: No ]
>> [Num Buckets: -1 ]
>> [Bucket Columns: [] ]
>> [Sort Columns: [] ]
>> [Storage Desc Params: ]
>> [ serialization.format 1 ]
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
>> wrote:
>>
>> Experts,
>>
>> I would like to know when a table was created in Hive database using
>> Spark shell?
>>
>> Thanks
>>
>>
>>
>>
>>
>>
>
Re: Finding out the time a table was created
Posted by Ted Yu <yu...@gmail.com>.
Looks like database support was fixed by:
[SPARK-7943] [SPARK-8105] [SPARK-8435] [SPARK-8714] [SPARK-8561] Fixes
multi-database support
On Fri, Mar 25, 2016 at 3:35 PM, Ashok Kumar <as...@yahoo.com> wrote:
> 1.5.2 Ted.
>
> Those two lines I don't know where they come. It finds and gets the table
> info OK
>
> HTH
>
>
> On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
>
>
> Which release of Spark do you use, Mich ?
>
> In master branch, the message is more accurate
> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
>
> override def getMessage: String = s"Table $table not found in database
> $db"
>
>
> On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
>
> This is based on the same command in Hive however, it throws two erroneous
> error lines as shown below (don't see them in Hive DESCRIBE ...)
>
> Example
>
> scala> sql("describe formatted test.t14").collect.foreach(println)
> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
> found
> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
> found
> [# col_name data_type comment ]
> [ ]
> [invoicenumber int ]
> [paymentdate date ]
> [net decimal(20,2) ]
> [vat decimal(20,2) ]
> [total decimal(20,2) ]
> [ ]
> [# Detailed Table Information ]
> [Database: test ]
> [Owner: hduser ]
> [
> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
> UNKNOWN ]
> [Protect Mode: None ]
> [Retention: 0 ]
> [Location:
> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
> [Table Type: MANAGED_TABLE ]
> [Table Parameters: ]
> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
> [ comment from csv file from excel sheet]
> [ numFiles 2 ]
> [ orc.compress ZLIB ]
> [ totalSize 1090 ]
> [ transient_lastDdlTime 1458944025 ]
> [ ]
> [# Storage Information ]
> [SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
> [InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
> [OutputFormat:
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
> [Compressed: No ]
> [Num Buckets: -1 ]
> [Bucket Columns: [] ]
> [Sort Columns: [] ]
> [Storage Desc Params: ]
> [ serialization.format 1 ]
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
> http://talebzadehmich.wordpress.com
>
>
> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
> wrote:
>
> Experts,
>
> I would like to know when a table was created in Hive database using Spark
> shell?
>
> Thanks
>
>
>
>
>
>
Re: Finding out the time a table was created
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hive gets it OK. That is what you saw as well I Guess Ashok, Mine is 1.5.2
as well
hive> describe formatted test.t14;
OK
# col_name data_type comment
invoicenumber int
paymentdate date
net decimal(20,2)
vat decimal(20,2)
total decimal(20,2)
# Detailed Table Information
Database: test
Owner: hduser
CreateTime: Fri Mar 25 22:13:44 GMT 2016
LastAccessTime: UNKNOWN
Retention: 0
Location: hdfs://rhes564:9000/user/hive/warehouse/test.db/t14
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
comment from csv file from excel sheet
numFiles 2
orc.compress ZLIB
totalSize 1090
transient_lastDdlTime 1458944025
# Storage Information
SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
Time taken: 0.047 seconds, Fetched: 34 row(s)
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 March 2016 at 22:35, Ashok Kumar <as...@yahoo.com> wrote:
> 1.5.2 Ted.
>
> Those two lines I don't know where they come. It finds and gets the table
> info OK
>
> HTH
>
>
> On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
>
>
> Which release of Spark do you use, Mich ?
>
> In master branch, the message is more accurate
> (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
>
> override def getMessage: String = s"Table $table not found in database
> $db"
>
>
> On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
>
> This is based on the same command in Hive however, it throws two erroneous
> error lines as shown below (don't see them in Hive DESCRIBE ...)
>
> Example
>
> scala> sql("describe formatted test.t14").collect.foreach(println)
> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
> found
> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
> found
> [# col_name data_type comment ]
> [ ]
> [invoicenumber int ]
> [paymentdate date ]
> [net decimal(20,2) ]
> [vat decimal(20,2) ]
> [total decimal(20,2) ]
> [ ]
> [# Detailed Table Information ]
> [Database: test ]
> [Owner: hduser ]
> [
> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
> UNKNOWN ]
> [Protect Mode: None ]
> [Retention: 0 ]
> [Location:
> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
> [Table Type: MANAGED_TABLE ]
> [Table Parameters: ]
> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
> [ comment from csv file from excel sheet]
> [ numFiles 2 ]
> [ orc.compress ZLIB ]
> [ totalSize 1090 ]
> [ transient_lastDdlTime 1458944025 ]
> [ ]
> [# Storage Information ]
> [SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
> [InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
> [OutputFormat:
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
> [Compressed: No ]
> [Num Buckets: -1 ]
> [Bucket Columns: [] ]
> [Sort Columns: [] ]
> [Storage Desc Params: ]
> [ serialization.format 1 ]
>
> HTH
>
> Dr Mich Talebzadeh
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
> http://talebzadehmich.wordpress.com
>
>
> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
> wrote:
>
> Experts,
>
> I would like to know when a table was created in Hive database using Spark
> shell?
>
> Thanks
>
>
>
>
>
>
Re: Finding out the time a table was created
Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
1.5.2 Ted.
Those two lines I don't know where they come. It finds and gets the table info OK
HTH
On Friday, 25 March 2016, 22:32, Ted Yu <yu...@gmail.com> wrote:
Which release of Spark do you use, Mich ?
In master branch, the message is more accurate (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
override def getMessage: String = s"Table $table not found in database $db"
On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <mi...@gmail.com> wrote:
You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
This is based on the same command in Hive however, it throws two erroneous error lines as shown below (don't see them in Hive DESCRIBE ...)
Example
scala> sql("describe formatted test.t14").collect.foreach(println)
16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not found
16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not found
[# col_name data_type comment ]
[ ]
[invoicenumber int ]
[paymentdate date ]
[net decimal(20,2) ]
[vat decimal(20,2) ]
[total decimal(20,2) ]
[ ]
[# Detailed Table Information ]
[Database: test ]
[Owner: hduser ]
[CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]
[LastAccessTime: UNKNOWN ]
[Protect Mode: None ]
[Retention: 0 ]
[Location: hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
[Table Type: MANAGED_TABLE ]
[Table Parameters: ]
[ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
[ comment from csv file from excel sheet]
[ numFiles 2 ]
[ orc.compress ZLIB ]
[ totalSize 1090 ]
[ transient_lastDdlTime 1458944025 ]
[ ]
[# Storage Information ]
[SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
[InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
[OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
[Compressed: No ]
[Num Buckets: -1 ]
[Bucket Columns: [] ]
[Sort Columns: [] ]
[Storage Desc Params: ]
[ serialization.format 1 ]
HTH
Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com
On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid> wrote:
Experts,
I would like to know when a table was created in Hive database using Spark shell?
Thanks
Re: Finding out the time a table was created
Posted by Ted Yu <yu...@gmail.com>.
Which release of Spark do you use, Mich ?
In master branch, the message is more accurate
(sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala):
override def getMessage: String = s"Table $table not found in database
$db"
On Fri, Mar 25, 2016 at 3:21 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
>
> This is based on the same command in Hive however, it throws two erroneous
> error lines as shown below (don't see them in Hive DESCRIBE ...)
>
> Example
>
> scala> sql("describe formatted test.t14").collect.foreach(println)
> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
> found
> 16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
> found
> [# col_name data_type comment ]
> [ ]
> [invoicenumber int ]
> [paymentdate date ]
> [net decimal(20,2) ]
> [vat decimal(20,2) ]
> [total decimal(20,2) ]
> [ ]
> [# Detailed Table Information ]
> [Database: test ]
> [Owner: hduser ]
> [
> *CreateTime: Fri Mar 25 22:13:44 GMT 2016 ]*[LastAccessTime:
> UNKNOWN ]
> [Protect Mode: None ]
> [Retention: 0 ]
> [Location:
> hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
> [Table Type: MANAGED_TABLE ]
> [Table Parameters: ]
> [ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
> [ comment from csv file from excel sheet]
> [ numFiles 2 ]
> [ orc.compress ZLIB ]
> [ totalSize 1090 ]
> [ transient_lastDdlTime 1458944025 ]
> [ ]
> [# Storage Information ]
> [SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
> [InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
> [OutputFormat:
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
> [Compressed: No ]
> [Num Buckets: -1 ]
> [Bucket Columns: [] ]
> [Sort Columns: [] ]
> [Storage Desc Params: ]
> [ serialization.format 1 ]
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid>
> wrote:
>
>> Experts,
>>
>> I would like to know when a table was created in Hive database using
>> Spark shell?
>>
>> Thanks
>>
>
>
Re: Finding out the time a table was created
Posted by Mich Talebzadeh <mi...@gmail.com>.
You can use DESCRIBE FORMATTED <DATABASE>.<TABLE_NAME> to get that info.
This is based on the same command in Hive however, it throws two erroneous
error lines as shown below (don't see them in Hive DESCRIBE ...)
Example
scala> sql("describe formatted test.t14").collect.foreach(println)
16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
found
16/03/25 22:32:38 ERROR Hive: Table test not found: test.test table not
found
[# col_name data_type comment ]
[ ]
[invoicenumber int ]
[paymentdate date ]
[net decimal(20,2) ]
[vat decimal(20,2) ]
[total decimal(20,2) ]
[ ]
[# Detailed Table Information ]
[Database: test ]
[Owner: hduser ]
[
*CreateTime: Fri Mar 25 22:13:44 GMT 2016
]*[LastAccessTime:
UNKNOWN ]
[Protect Mode: None ]
[Retention: 0 ]
[Location:
hdfs://rhes564:9000/user/hive/warehouse/test.db/t14 ]
[Table Type: MANAGED_TABLE ]
[Table Parameters: ]
[ COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}]
[ comment from csv file from excel sheet]
[ numFiles 2 ]
[ orc.compress ZLIB ]
[ totalSize 1090 ]
[ transient_lastDdlTime 1458944025 ]
[ ]
[# Storage Information ]
[SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde ]
[InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat ]
[OutputFormat:
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat ]
[Compressed: No ]
[Num Buckets: -1 ]
[Bucket Columns: [] ]
[Sort Columns: [] ]
[Storage Desc Params: ]
[ serialization.format 1 ]
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 25 March 2016 at 22:12, Ashok Kumar <as...@yahoo.com.invalid> wrote:
> Experts,
>
> I would like to know when a table was created in Hive database using Spark
> shell?
>
> Thanks
>