You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by "Kukreja, Tarun" <ta...@deloitte.com.INVALID> on 2022/10/19 13:54:29 UTC
Option on Different Cloud
Hello,
I know, We can use Athena + S3 to work with Iceberg table.
Is there any such option available on Azure and GCP?
Regards,
Tarun Kukreja
+91-9910998689
This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited.
Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about to learn more.
v.E.1
Re: Option on Different Cloud
Posted by Jon Fritz <jo...@dremio.com>.
For AWS, Azure, and GCP - Dremio can also query Iceberg tables in S3, GCS, and ADLS:
https://docs.dremio.com/software/data-formats/apache-iceberg/
On AWS, there is a fully managed solution where you can get started in a few clicks: https://www.dremio.com/platform/
> On Oct 20, 2022, at 1:14 PM, Austin Bennett <wh...@gmail.com> wrote:
>
> GCP -->
>
> https://cloud.google.com/biglake <https://cloud.google.com/biglake>
>
> Effectively BigQuery can read Iceberg.
>
> Also, see a video from last week's announcement: https://www.youtube.com/watch?v=iQilh-PHDvs <https://www.youtube.com/watch?v=iQilh-PHDvs>
>
> Cheers -
>
> On Thu, Oct 20, 2022 at 3:15 PM Alexander Jo <alex.jo@starburstdata.com <ma...@starburstdata.com>> wrote:
> Since you mentioned Athena, I throw in that the open source Trino connector supports GCS and Azure.
> I know the documentation here says Hive, but it also works for Iceberg,
> - https://trino.io/docs/current/connector/hive-gcs-tutorial.html <https://trino.io/docs/current/connector/hive-gcs-tutorial.html>
> - https://trino.io/docs/current/connector/hive-azure.html <https://trino.io/docs/current/connector/hive-azure.html>
>
> I'll also suggest Starburst Galaxy, which can be deployed in Azure/GCS and provides a pretty out of the box experience like Athena does.
> https://www.starburst.io/platform/starburst-galaxy/ <https://www.starburst.io/platform/starburst-galaxy/>
> On Thu, Oct 20, 2022 at 6:55 PM Micah Kornfield <emkornfield@gmail.com <ma...@gmail.com>> wrote:
>
> For Querying on GCP, BigQuery just recently announced preview support for querying Iceberg tables (more docs should be getting published soon with details).
>
> On Wed, Oct 19, 2022 at 9:57 AM Ryan Blue <blue@tabular.io <ma...@tabular.io>> wrote:
> In addition to the native FileIO implementations, you can use any Hadoop FileSystem with Iceberg through the default FIleIO implementation, HadoopFileIO. If there's Hadoop support for a back-end, it'll just work.
>
> The reason why we have direct implementations is that we can relax some of the constraints and avoid a lot of requests just to ensure the FileSystem abstraction is consistent. But you don't actually require a FileIO implementation.
>
> Ryan
>
> On Wed, Oct 19, 2022 at 9:48 AM Eduard Tudenhoefner <eduard@tabular.io <ma...@tabular.io>> wrote:
> For GCP you could use GCSFileIO <https://github.com/apache/iceberg/blob/master/gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java#L49> and for Azure there is an open PR <https://github.com/apache/iceberg/pull/4465> to provide AzureFileIO.
>
> On Wed, Oct 19, 2022 at 6:33 PM Kukreja, Tarun <ta...@deloitte.com.invalid> wrote:
> Hello,
>
>
>
> I know, We can use Athena + S3 to work with Iceberg table.
>
>
>
> Is there any such option available on Azure and GCP?
>
>
>
> Regards,
>
> Tarun Kukreja
>
> +91-9910998689
>
>
>
> This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited.
>
> Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about <http://www.deloitte.com/about> to learn more.
>
> v.E.1
>
>
>
> --
> Ryan Blue
> Tabular
Re: Option on Different Cloud
Posted by Austin Bennett <wh...@gmail.com>.
GCP -->
https://cloud.google.com/biglake
Effectively BigQuery can read Iceberg.
Also, see a video from last week's announcement:
https://www.youtube.com/watch?v=iQilh-PHDvs
Cheers -
On Thu, Oct 20, 2022 at 3:15 PM Alexander Jo <al...@starburstdata.com>
wrote:
> Since you mentioned Athena, I throw in that the open source Trino
> connector supports GCS and Azure.
> I know the documentation here says Hive, but it also works for Iceberg,
> - https://trino.io/docs/current/connector/hive-gcs-tutorial.html
> - https://trino.io/docs/current/connector/hive-azure.html
>
> I'll also suggest Starburst Galaxy, which can be deployed in Azure/GCS and
> provides a pretty out of the box experience like Athena does.
> https://www.starburst.io/platform/starburst-galaxy/
>
> On Thu, Oct 20, 2022 at 6:55 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>>
>> For Querying on GCP, BigQuery just recently announced preview support for
>> querying Iceberg tables (more docs should be getting published soon with
>> details).
>>
>> On Wed, Oct 19, 2022 at 9:57 AM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> In addition to the native FileIO implementations, you can use any Hadoop
>>> FileSystem with Iceberg through the default FIleIO implementation,
>>> HadoopFileIO. If there's Hadoop support for a back-end, it'll just work.
>>>
>>> The reason why we have direct implementations is that we can relax some
>>> of the constraints and avoid a lot of requests just to ensure the
>>> FileSystem abstraction is consistent. But you don't actually require a
>>> FileIO implementation.
>>>
>>> Ryan
>>>
>>> On Wed, Oct 19, 2022 at 9:48 AM Eduard Tudenhoefner <ed...@tabular.io>
>>> wrote:
>>>
>>>> For GCP you could use GCSFileIO
>>>> <https://github.com/apache/iceberg/blob/master/gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java#L49>
>>>> and for Azure there is an open PR
>>>> <https://github.com/apache/iceberg/pull/4465> to provide AzureFileIO.
>>>>
>>>> On Wed, Oct 19, 2022 at 6:33 PM Kukreja, Tarun
>>>> <ta...@deloitte.com.invalid> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>
>>>>> I know, We can use Athena + S3 to work with Iceberg table.
>>>>>
>>>>>
>>>>>
>>>>> Is there any such option available on Azure and GCP?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Tarun Kukreja
>>>>>
>>>>> +91-9910998689
>>>>>
>>>>>
>>>>>
>>>>> This message (including any attachments) contains confidential
>>>>> information intended for a specific individual and purpose, and is
>>>>> protected by law. If you are not the intended recipient, you should delete
>>>>> this message and any disclosure, copying, or distribution of this message,
>>>>> or the taking of any action based on it, by you is strictly prohibited.
>>>>>
>>>>> Deloitte refers to a Deloitte member firm, one of its related
>>>>> entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte
>>>>> member firm is a separate legal entity and a member of DTTL. DTTL does not
>>>>> provide services to clients. Please see www.deloitte.com/about to
>>>>> learn more.
>>>>>
>>>>> v.E.1
>>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
Re: Option on Different Cloud
Posted by Alexander Jo <al...@starburstdata.com>.
Since you mentioned Athena, I throw in that the open source Trino connector
supports GCS and Azure.
I know the documentation here says Hive, but it also works for Iceberg,
- https://trino.io/docs/current/connector/hive-gcs-tutorial.html
- https://trino.io/docs/current/connector/hive-azure.html
I'll also suggest Starburst Galaxy, which can be deployed in Azure/GCS and
provides a pretty out of the box experience like Athena does.
https://www.starburst.io/platform/starburst-galaxy/
On Thu, Oct 20, 2022 at 6:55 PM Micah Kornfield <em...@gmail.com>
wrote:
>
> For Querying on GCP, BigQuery just recently announced preview support for
> querying Iceberg tables (more docs should be getting published soon with
> details).
>
> On Wed, Oct 19, 2022 at 9:57 AM Ryan Blue <bl...@tabular.io> wrote:
>
>> In addition to the native FileIO implementations, you can use any Hadoop
>> FileSystem with Iceberg through the default FIleIO implementation,
>> HadoopFileIO. If there's Hadoop support for a back-end, it'll just work.
>>
>> The reason why we have direct implementations is that we can relax some
>> of the constraints and avoid a lot of requests just to ensure the
>> FileSystem abstraction is consistent. But you don't actually require a
>> FileIO implementation.
>>
>> Ryan
>>
>> On Wed, Oct 19, 2022 at 9:48 AM Eduard Tudenhoefner <ed...@tabular.io>
>> wrote:
>>
>>> For GCP you could use GCSFileIO
>>> <https://github.com/apache/iceberg/blob/master/gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java#L49>
>>> and for Azure there is an open PR
>>> <https://github.com/apache/iceberg/pull/4465> to provide AzureFileIO.
>>>
>>> On Wed, Oct 19, 2022 at 6:33 PM Kukreja, Tarun
>>> <ta...@deloitte.com.invalid> wrote:
>>>
>>>> Hello,
>>>>
>>>>
>>>>
>>>> I know, We can use Athena + S3 to work with Iceberg table.
>>>>
>>>>
>>>>
>>>> Is there any such option available on Azure and GCP?
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Tarun Kukreja
>>>>
>>>> +91-9910998689
>>>>
>>>>
>>>>
>>>> This message (including any attachments) contains confidential
>>>> information intended for a specific individual and purpose, and is
>>>> protected by law. If you are not the intended recipient, you should delete
>>>> this message and any disclosure, copying, or distribution of this message,
>>>> or the taking of any action based on it, by you is strictly prohibited.
>>>>
>>>> Deloitte refers to a Deloitte member firm, one of its related entities,
>>>> or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is
>>>> a separate legal entity and a member of DTTL. DTTL does not provide
>>>> services to clients. Please see www.deloitte.com/about to learn more.
>>>>
>>>> v.E.1
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>
Re: Option on Different Cloud
Posted by Micah Kornfield <em...@gmail.com>.
For Querying on GCP, BigQuery just recently announced preview support for
querying Iceberg tables (more docs should be getting published soon with
details).
On Wed, Oct 19, 2022 at 9:57 AM Ryan Blue <bl...@tabular.io> wrote:
> In addition to the native FileIO implementations, you can use any Hadoop
> FileSystem with Iceberg through the default FIleIO implementation,
> HadoopFileIO. If there's Hadoop support for a back-end, it'll just work.
>
> The reason why we have direct implementations is that we can relax some of
> the constraints and avoid a lot of requests just to ensure the FileSystem
> abstraction is consistent. But you don't actually require a FileIO
> implementation.
>
> Ryan
>
> On Wed, Oct 19, 2022 at 9:48 AM Eduard Tudenhoefner <ed...@tabular.io>
> wrote:
>
>> For GCP you could use GCSFileIO
>> <https://github.com/apache/iceberg/blob/master/gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java#L49>
>> and for Azure there is an open PR
>> <https://github.com/apache/iceberg/pull/4465> to provide AzureFileIO.
>>
>> On Wed, Oct 19, 2022 at 6:33 PM Kukreja, Tarun
>> <ta...@deloitte.com.invalid> wrote:
>>
>>> Hello,
>>>
>>>
>>>
>>> I know, We can use Athena + S3 to work with Iceberg table.
>>>
>>>
>>>
>>> Is there any such option available on Azure and GCP?
>>>
>>>
>>>
>>> Regards,
>>>
>>> Tarun Kukreja
>>>
>>> +91-9910998689
>>>
>>>
>>>
>>> This message (including any attachments) contains confidential
>>> information intended for a specific individual and purpose, and is
>>> protected by law. If you are not the intended recipient, you should delete
>>> this message and any disclosure, copying, or distribution of this message,
>>> or the taking of any action based on it, by you is strictly prohibited.
>>>
>>> Deloitte refers to a Deloitte member firm, one of its related entities,
>>> or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is
>>> a separate legal entity and a member of DTTL. DTTL does not provide
>>> services to clients. Please see www.deloitte.com/about to learn more.
>>>
>>> v.E.1
>>>
>>
>
> --
> Ryan Blue
> Tabular
>
Re: Option on Different Cloud
Posted by Ryan Blue <bl...@tabular.io>.
In addition to the native FileIO implementations, you can use any Hadoop
FileSystem with Iceberg through the default FIleIO implementation,
HadoopFileIO. If there's Hadoop support for a back-end, it'll just work.
The reason why we have direct implementations is that we can relax some of
the constraints and avoid a lot of requests just to ensure the FileSystem
abstraction is consistent. But you don't actually require a FileIO
implementation.
Ryan
On Wed, Oct 19, 2022 at 9:48 AM Eduard Tudenhoefner <ed...@tabular.io>
wrote:
> For GCP you could use GCSFileIO
> <https://github.com/apache/iceberg/blob/master/gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java#L49>
> and for Azure there is an open PR
> <https://github.com/apache/iceberg/pull/4465> to provide AzureFileIO.
>
> On Wed, Oct 19, 2022 at 6:33 PM Kukreja, Tarun
> <ta...@deloitte.com.invalid> wrote:
>
>> Hello,
>>
>>
>>
>> I know, We can use Athena + S3 to work with Iceberg table.
>>
>>
>>
>> Is there any such option available on Azure and GCP?
>>
>>
>>
>> Regards,
>>
>> Tarun Kukreja
>>
>> +91-9910998689
>>
>>
>>
>> This message (including any attachments) contains confidential
>> information intended for a specific individual and purpose, and is
>> protected by law. If you are not the intended recipient, you should delete
>> this message and any disclosure, copying, or distribution of this message,
>> or the taking of any action based on it, by you is strictly prohibited.
>>
>> Deloitte refers to a Deloitte member firm, one of its related entities,
>> or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is
>> a separate legal entity and a member of DTTL. DTTL does not provide
>> services to clients. Please see www.deloitte.com/about to learn more.
>>
>> v.E.1
>>
>
--
Ryan Blue
Tabular
Re: Option on Different Cloud
Posted by Eduard Tudenhoefner <ed...@tabular.io>.
For GCP you could use GCSFileIO
<https://github.com/apache/iceberg/blob/master/gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java#L49>
and for Azure there is an open PR
<https://github.com/apache/iceberg/pull/4465> to provide AzureFileIO.
On Wed, Oct 19, 2022 at 6:33 PM Kukreja, Tarun
<ta...@deloitte.com.invalid> wrote:
> Hello,
>
>
>
> I know, We can use Athena + S3 to work with Iceberg table.
>
>
>
> Is there any such option available on Azure and GCP?
>
>
>
> Regards,
>
> Tarun Kukreja
>
> +91-9910998689
>
>
>
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law. If
> you are not the intended recipient, you should delete this message and any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, by you is strictly prohibited.
>
> Deloitte refers to a Deloitte member firm, one of its related entities, or
> Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a
> separate legal entity and a member of DTTL. DTTL does not provide services
> to clients. Please see www.deloitte.com/about to learn more.
>
> v.E.1
>