You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Shivam Sharma <28...@gmail.com> on 2018/07/02 16:41:53 UTC
Regarding external metastore like HIVE
Hi,
I have read Flink documentation that Flink supports Metastore which is
currently InMemory. Is Flink community thinking to implement external
Metastore like Hive?
Thanks
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*
Re: Regarding external metastore like HIVE
Posted by Rong Rong <wa...@gmail.com>.
+1 on this feature, there have been a lot of pains for us trying to connect
to external catalog / metastore as well.
@shivam can you comment on the tickets regarding the specific use case and
the type of external catalogs you are interested ?
Thanks,
Rong
On Tue, Jul 3, 2018 at 3:16 AM Shivam Sharma <28...@gmail.com>
wrote:
> Thanks Timo, Fabian
>
> I will follow this.
>
> Best
>
> On Tue, Jul 3, 2018 at 3:01 PM Timo Walther <tw...@apache.org> wrote:
>
>> Hi,
>>
>> you can follow the progress here:
>> https://issues.apache.org/jira/browse/FLINK-9171
>>
>> Regards,
>> Timo
>>
>>
>> Am 03.07.18 um 10:32 schrieb Fabian Hueske:
>>
>> Hi,
>>
>> The docs explain that the ExternalCatalog interface *can* be used to
>> implement a catalog for HCatalog or Metastore.
>> However, there is no such implementation in Flink yet. You would need to
>> implement such as catalog connector yourself.
>>
>> I think there would be quite a few people interested in such a connector.
>>
>> Best, Fabian
>>
>> 2018-07-03 10:24 GMT+02:00 Shivam Sharma <28...@gmail.com>:
>>
>>> Hi,
>>>
>>> Please find the documentation link here:
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog
>>> .
>>>
>>> Currently, Flink provides an InMemoryExternalCatalog for demo and
>>>> testing purposes. However, the ExternalCatalog interface can also be
>>>> used to connect catalogs like *HCatalog or Metastore* to the Table API.
>>>
>>>
>>> Thanks
>>>
>>>
>>> On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <ni...@gmail.com> wrote:
>>>
>>>> @ashwin
>>>>
>>>> I cannot find the documention about `metastore`
>>>>
>>>> Could you give the reference?
>>>>
>>>> Ashwin Sinha <as...@go-mmt.com> 于2018年7月3日周二 下午4:08写道：
>>>>
>>>>> By metastore we mean catalog where table information is stored.
>>>>>
>>>>> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <ch...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> What do you mean with "Metastore"? Are you referring to state
>>>>>> backends
>>>>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>
>>>>>> ?
>>>>>>
>>>>>> On 02.07.2018 18:41, Shivam Sharma wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have read Flink documentation that Flink supports Metastore which
>>>>>> is currently InMemory. Is Flink community thinking to implement external
>>>>>> Metastore like Hive?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> --
>>>>>> Shivam Sharma
>>>>>> Data Engineer @ Goibibo
>>>>>> Indian Institute Of Information Technology, Design and Manufacturing
>>>>>> Jabalpur
>>>>>> Mobile No- (+91) 8882114744
>>>>>> Email:- 28shivamsharma@gmail.com
>>>>>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>>>>>> <https://www.linkedin.com/in/28shivamsharma>*
>>>>>>
>>>>>>
>>>>>>
>>>>> ::DISCLAIMER::
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> This message is intended only for the use of the addressee and may
>>>>> contain information that is privileged, confidential and exempt from
>>>>> disclosure under applicable law. If the reader of this message is not the
>>>>> intended recipient, or the employee or agent responsible for delivering the
>>>>> message to the intended recipient, you are hereby notified that any
>>>>> dissemination, distribution or copying of this communication is strictly
>>>>> prohibited. If you have received this e-mail in error, please notify us
>>>>> immediately by return e-mail and delete this e-mail and all attachments
>>>>> from your system.
>>>>>
>>>>
>>>
>>> --
>>> Shivam Sharma
>>> Data Engineer @ Goibibo
>>> Indian Institute Of Information Technology, Design and Manufacturing
>>> Jabalpur
>>> Mobile No- (+91) 8882114744
>>> Email:- 28shivamsharma@gmail.com
>>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>>> <https://www.linkedin.com/in/28shivamsharma>*
>>>
>>
>>
>>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>
Re: Regarding external metastore like HIVE
Posted by Shivam Sharma <28...@gmail.com>.
Thanks Timo, Fabian
I will follow this.
Best
On Tue, Jul 3, 2018 at 3:01 PM Timo Walther <tw...@apache.org> wrote:
> Hi,
>
> you can follow the progress here:
> https://issues.apache.org/jira/browse/FLINK-9171
>
> Regards,
> Timo
>
>
> Am 03.07.18 um 10:32 schrieb Fabian Hueske:
>
> Hi,
>
> The docs explain that the ExternalCatalog interface *can* be used to
> implement a catalog for HCatalog or Metastore.
> However, there is no such implementation in Flink yet. You would need to
> implement such as catalog connector yourself.
>
> I think there would be quite a few people interested in such a connector.
>
> Best, Fabian
>
> 2018-07-03 10:24 GMT+02:00 Shivam Sharma <28...@gmail.com>:
>
>> Hi,
>>
>> Please find the documentation link here:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog
>> .
>>
>> Currently, Flink provides an InMemoryExternalCatalog for demo and
>>> testing purposes. However, the ExternalCatalog interface can also be
>>> used to connect catalogs like *HCatalog or Metastore* to the Table API.
>>
>>
>> Thanks
>>
>>
>> On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <ni...@gmail.com> wrote:
>>
>>> @ashwin
>>>
>>> I cannot find the documention about `metastore`
>>>
>>> Could you give the reference?
>>>
>>> Ashwin Sinha <as...@go-mmt.com> 于2018年7月3日周二 下午4:08写道：
>>>
>>>> By metastore we mean catalog where table information is stored.
>>>>
>>>> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <ch...@apache.org> wrote:
>>>>
>>>>> What do you mean with "Metastore"? Are you referring to state backends
>>>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>
>>>>> ?
>>>>>
>>>>> On 02.07.2018 18:41, Shivam Sharma wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have read Flink documentation that Flink supports Metastore which is
>>>>> currently InMemory. Is Flink community thinking to implement external
>>>>> Metastore like Hive?
>>>>>
>>>>> Thanks
>>>>>
>>>>> --
>>>>> Shivam Sharma
>>>>> Data Engineer @ Goibibo
>>>>> Indian Institute Of Information Technology, Design and Manufacturing
>>>>> Jabalpur
>>>>> Mobile No- (+91) 8882114744
>>>>> Email:- 28shivamsharma@gmail.com
>>>>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>>>>> <https://www.linkedin.com/in/28shivamsharma>*
>>>>>
>>>>>
>>>>>
>>>> ::DISCLAIMER::
>>>>
>>>>
>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> This message is intended only for the use of the addressee and may
>>>> contain information that is privileged, confidential and exempt from
>>>> disclosure under applicable law. If the reader of this message is not the
>>>> intended recipient, or the employee or agent responsible for delivering the
>>>> message to the intended recipient, you are hereby notified that any
>>>> dissemination, distribution or copying of this communication is strictly
>>>> prohibited. If you have received this e-mail in error, please notify us
>>>> immediately by return e-mail and delete this e-mail and all attachments
>>>> from your system.
>>>>
>>>
>>
>> --
>> Shivam Sharma
>> Data Engineer @ Goibibo
>> Indian Institute Of Information Technology, Design and Manufacturing
>> Jabalpur
>> Mobile No- (+91) 8882114744
>> Email:- 28shivamsharma@gmail.com
>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>> <https://www.linkedin.com/in/28shivamsharma>*
>>
>
>
>
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*
Re: Regarding external metastore like HIVE
Posted by Timo Walther <tw...@apache.org>.
Hi,
you can follow the progress here:
https://issues.apache.org/jira/browse/FLINK-9171
Regards,
Timo
Am 03.07.18 um 10:32 schrieb Fabian Hueske:
> Hi,
>
> The docs explain that the ExternalCatalog interface *can* be used to
> implement a catalog for HCatalog or Metastore.
> However, there is no such implementation in Flink yet. You would need
> to implement such as catalog connector yourself.
>
> I think there would be quite a few people interested in such a connector.
>
> Best, Fabian
>
> 2018-07-03 10:24 GMT+02:00 Shivam Sharma <28shivamsharma@gmail.com
> <ma...@gmail.com>>:
>
> Hi,
>
> Please find the documentation link here:
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog
> <https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog>.
>
> Currently, Flink provides an|InMemoryExternalCatalog|for demo
> and testing purposes. However, the|ExternalCatalog|interface
> can also be used to connect catalogs like *HCatalog or
> Metastore* to the Table API.
>
>
> Thanks
>
>
> On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <nichuanlei@gmail.com
> <ma...@gmail.com>> wrote:
>
> @ashwin
>
> I cannot find the documention about `metastore`
>
> Could you give the reference?
>
> Ashwin Sinha <ashwin.sinha@go-mmt.com
> <ma...@go-mmt.com>> 于2018年7月3日周二
> 下午4:08写道：
>
> By metastore we mean catalog where table information is
> stored.
>
> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler,
> <chesnay@apache.org <ma...@apache.org>> wrote:
>
> What do you mean with "Metastore"? Are you referring
> to state backends
> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>?
>
> On 02.07.2018 18:41, Shivam Sharma wrote:
>> Hi,
>>
>> I have read Flink documentation that Flink supports
>> Metastore which is currently InMemory. Is Flink
>> community thinking to implement external Metastore
>> like Hive?
>>
>> Thanks
>>
>> --
>> Shivam Sharma
>> Data Engineer @ Goibibo
>> Indian Institute Of Information Technology, Design
>> and Manufacturing Jabalpur
>> Mobile No- (+91) 8882114744
>> Email:- 28shivamsharma@gmail.com
>> <ma...@gmail.com>
>> LinkedIn:-_https://www.linkedin.com/in/28shivamsharma
>> <https://www.linkedin.com/in/28shivamsharma>_
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> This message is intended only for the use of the addressee
> and may contain information that is privileged,
> confidential and exempt from disclosure under applicable
> law. If the reader of this message is not the intended
> recipient, or the employee or agent responsible for
> delivering the message to the intended recipient, you are
> hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If
> you have received this e-mail in error, please notify us
> immediately by return e-mail and delete this e-mail and
> all attachments from your system.
>
>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and
> Manufacturing Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com <ma...@gmail.com>
> LinkedIn:-_https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>_
>
>
Re: Regarding external metastore like HIVE
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
The docs explain that the ExternalCatalog interface *can* be used to
implement a catalog for HCatalog or Metastore.
However, there is no such implementation in Flink yet. You would need to
implement such as catalog connector yourself.
I think there would be quite a few people interested in such a connector.
Best, Fabian
2018-07-03 10:24 GMT+02:00 Shivam Sharma <28...@gmail.com>:
> Hi,
>
> Please find the documentation link here: https://ci.apache.org/
> projects/flink/flink-docs-release-1.5/dev/table/common.
> html#register-an-external-catalog.
>
> Currently, Flink provides an InMemoryExternalCatalog for demo and testing
>> purposes. However, the ExternalCatalog interface can also be used to
>> connect catalogs like *HCatalog or Metastore* to the Table API.
>
>
> Thanks
>
>
> On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <ni...@gmail.com> wrote:
>
>> @ashwin
>>
>> I cannot find the documention about `metastore`
>>
>> Could you give the reference?
>>
>> Ashwin Sinha <as...@go-mmt.com> 于2018年7月3日周二 下午4:08写道：
>>
>>> By metastore we mean catalog where table information is stored.
>>>
>>> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <ch...@apache.org> wrote:
>>>
>>>> What do you mean with "Metastore"? Are you referring to state backends
>>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>
>>>> ?
>>>>
>>>> On 02.07.2018 18:41, Shivam Sharma wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have read Flink documentation that Flink supports Metastore which is
>>>> currently InMemory. Is Flink community thinking to implement external
>>>> Metastore like Hive?
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> Shivam Sharma
>>>> Data Engineer @ Goibibo
>>>> Indian Institute Of Information Technology, Design and Manufacturing
>>>> Jabalpur
>>>> Mobile No- (+91) 8882114744
>>>> Email:- 28shivamsharma@gmail.com
>>>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>>>> <https://www.linkedin.com/in/28shivamsharma>*
>>>>
>>>>
>>>>
>>> ::DISCLAIMER::
>>>
>>> ------------------------------------------------------------
>>> ------------------------------------------------------------
>>> ----------------------------
>>>
>>>
>>> This message is intended only for the use of the addressee and may
>>> contain information that is privileged, confidential and exempt from
>>> disclosure under applicable law. If the reader of this message is not the
>>> intended recipient, or the employee or agent responsible for delivering the
>>> message to the intended recipient, you are hereby notified that any
>>> dissemination, distribution or copying of this communication is strictly
>>> prohibited. If you have received this e-mail in error, please notify us
>>> immediately by return e-mail and delete this e-mail and all attachments
>>> from your system.
>>>
>>
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>
Re: Regarding external metastore like HIVE
Posted by Shivam Sharma <28...@gmail.com>.
Hi,
Please find the documentation link here:
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/table/common.html#register-an-external-catalog
.
Currently, Flink provides an InMemoryExternalCatalog for demo and testing
> purposes. However, the ExternalCatalog interface can also be used to
> connect catalogs like *HCatalog or Metastore* to the Table API.
Thanks
On Tue, Jul 3, 2018 at 1:41 PM Chuanlei Ni <ni...@gmail.com> wrote:
> @ashwin
>
> I cannot find the documention about `metastore`
>
> Could you give the reference?
>
> Ashwin Sinha <as...@go-mmt.com> 于2018年7月3日周二 下午4:08写道：
>
>> By metastore we mean catalog where table information is stored.
>>
>> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <ch...@apache.org> wrote:
>>
>>> What do you mean with "Metastore"? Are you referring to state backends
>>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>
>>> ?
>>>
>>> On 02.07.2018 18:41, Shivam Sharma wrote:
>>>
>>> Hi,
>>>
>>> I have read Flink documentation that Flink supports Metastore which is
>>> currently InMemory. Is Flink community thinking to implement external
>>> Metastore like Hive?
>>>
>>> Thanks
>>>
>>> --
>>> Shivam Sharma
>>> Data Engineer @ Goibibo
>>> Indian Institute Of Information Technology, Design and Manufacturing
>>> Jabalpur
>>> Mobile No- (+91) 8882114744
>>> Email:- 28shivamsharma@gmail.com
>>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>>> <https://www.linkedin.com/in/28shivamsharma>*
>>>
>>>
>>>
>> ::DISCLAIMER::
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is not the
>> intended recipient, or the employee or agent responsible for delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify us
>> immediately by return e-mail and delete this e-mail and all attachments
>> from your system.
>>
>
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*
Re: Regarding external metastore like HIVE
Posted by Chuanlei Ni <ni...@gmail.com>.
@ashwin
I cannot find the documention about `metastore`
Could you give the reference?
Ashwin Sinha <as...@go-mmt.com> 于2018年7月3日周二 下午4:08写道：
> By metastore we mean catalog where table information is stored.
>
> On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <ch...@apache.org> wrote:
>
>> What do you mean with "Metastore"? Are you referring to state backends
>> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>
>> ?
>>
>> On 02.07.2018 18:41, Shivam Sharma wrote:
>>
>> Hi,
>>
>> I have read Flink documentation that Flink supports Metastore which is
>> currently InMemory. Is Flink community thinking to implement external
>> Metastore like Hive?
>>
>> Thanks
>>
>> --
>> Shivam Sharma
>> Data Engineer @ Goibibo
>> Indian Institute Of Information Technology, Design and Manufacturing
>> Jabalpur
>> Mobile No- (+91) 8882114744
>> Email:- 28shivamsharma@gmail.com
>> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
>> <https://www.linkedin.com/in/28shivamsharma>*
>>
>>
>>
> ::DISCLAIMER::
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> This message is intended only for the use of the addressee and may contain
> information that is privileged, confidential and exempt from disclosure
> under applicable law. If the reader of this message is not the intended
> recipient, or the employee or agent responsible for delivering the message
> to the intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this e-mail in error, please notify us immediately by
> return e-mail and delete this e-mail and all attachments from your system.
>
Re: Regarding external metastore like HIVE
Posted by Ashwin Sinha <as...@go-mmt.com>.
By metastore we mean catalog where table information is stored.
On Tue 3 Jul, 2018, 13:23 Chesnay Schepler, <ch...@apache.org> wrote:
> What do you mean with "Metastore"? Are you referring to state backends
> <https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>
> ?
>
> On 02.07.2018 18:41, Shivam Sharma wrote:
>
> Hi,
>
> I have read Flink documentation that Flink supports Metastore which is
> currently InMemory. Is Flink community thinking to implement external
> Metastore like Hive?
>
> Thanks
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>
>
>
--
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
This message is intended only for the use of the addressee and may
contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the
intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us
immediately by return e-mail and delete this e-mail and all attachments
from your system.
Re: Regarding external metastore like HIVE
Posted by Chesnay Schepler <ch...@apache.org>.
What do you mean with "Metastore"? Are you referring to state backends
<https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html>?
On 02.07.2018 18:41, Shivam Sharma wrote:
> Hi,
>
> I have read Flink documentation that Flink supports Metastore which is
> currently InMemory. Is Flink community thinking to implement external
> Metastore like Hive?
>
> Thanks
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com <ma...@gmail.com>
> LinkedIn:-_https://www.linkedin.com/in/28shivamsharma_