Re: [EXT] Re: implementation in other languages

[Casey Lucas <Ca...@Dynata.com>]

Hi Ryan,
Thanks for the clarification. Will do.


Casey Lucas
Director, Engineering

dynata.com

The information contained in this e-mail message is intended 
for the use of the recipient(s) named above and is privileged and 
confidential. If you are not the intended recipient, you are formally 
notified that you have received this message in error and that any review, 
dissemination, distribution, or copying of the message is strictly 
prohibited. If you have received this communication in error, please 
notify us immediately by e-mail and delete the original message.
From: Ryan Blue <rb...@netflix.com.INVALID>
Date: Tuesday, January 5, 2021 at 4:31 PM
To: Iceberg Dev List <de...@iceberg.apache.org>
Subject: [EXT] Re: implementation in other languages
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
 
Hi Casey, 

We wrote the spec for Iceberg intending for it to be implemented in other languages, so we purposely avoided anything that would tie it to Java, like using Java `hashCode` implementations. It should be portable, but please let us know if you run into any issues.

I'm not aware of an existing go implementation. If you start one, please let the community know!

rb

On Sun, Jan 3, 2021 at 1:20 PM Driesprong, Fokko <fo...@driesprong.frl> wrote:
Hi Casey,

1. Personally I don't know any other implementation than in the repository itself, so JVM and Python.
2. There aren't any limitations. Most of the underlying tech used is compatible with many languages. One thing to keep in mind is that you should be able to read Avro files, which are also available in many languages. Apart from that, the data itself is written in Avro/Parquet/ORC which is also widely used and supported.

I've been working on an Apache Beam implementation (very early stage), which also has a Go API. But that will take some time to get it to a certain maturity level. Hope this helps!

Cheers, Fokko

Op zo 3 jan. 2021 om 19:39 schreef Casey Lucas <ma...@dynata.com>:
I'm just learning about Iceberg. Thanks for the work that has gone into it thus far. It seems like a promising technology.

I understand why the core implementation is Java based and saw that there are some aspects of the spec implemented in Python.

1. Is anyone aware of implementations in other languages such as go?
2. Is there anything that would fundamentally limit the use of other implementations being able to read and write Iceberg data that is also being manipulated by the JVM implementation? I assume no based on reading the spec, but any insight is appreciated.


I'm curious if reading and writing Iceberg data from a spec-compliant implementation without necessarily involving Presto or Spark is an expected use-case.

Thanks,

Casey Lucas
Director, Engineering

https://nam10.safelinks.protection.outlook.com/?url=http://dynata.com/&data=04|01|Casey.Lucas@dynata.com|cd8b44683a464b2b755208d8b1c99d86|f0ff917dab8c4129b13f33be267a153b|0|0|637454826780075160|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0=|1000&sdata=EO6zIyEgnNMC0Tw1OyMuq6K8A/7VMZc/9CbRScvG2Ko=&reserved=0

The information contained in this e-mail message is intended 
for the use of the recipient(s) named above and is privileged and 
confidential. If you are not the intended recipient, you are formally 
notified that you have received this message in error and that any review, 
dissemination, distribution, or copying of the message is strictly 
prohibited. If you have received this communication in error, please 
notify us immediately by e-mail and delete the original message.



-- 
Ryan Blue 
Software Engineer
Netflix