You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by "John D. Ament" <jo...@apache.org> on 2024/05/16 17:53:59 UTC
Can I get started with a plain java app?
Completely naive question since I'm not familiar at all with the
technologies. I wanted to demonstrate using Iceberg files as a way to
ingest lots of data and persist it to S3. It seems like it can do this,
but I have a feeling I need tools like Spark to do it. is that true? or
can I hook it up to just a generic REST API that loads the data in?
I have a feeling Spark will be part of the final solution, but I don't want
to get into that complexity quite yet, and it's not clear from the docs.
Also apologies for posting this to your dev@ list. No user(s)@ list.
John
Re: Can I get started with a plain java app?
Posted by Steven Wu <st...@gmail.com>.
Iceberg has a Java library, which is the most complete implementation of
the spec (compared to other languages like Python, Rust) at the moment. You
can certainly use the Java library directly to write and commit data to
Iceberg. But you will likely need to implement quite a bit of code for
things that are already taken care of by the engines like Spark, Flink,
Trino etc.
You also mentioned lots of data. You would need a distributed engine (like
Spark etc.) to handle larger scales. Re-implementing your own distributed
engine is likely not a good investment of your time.
On Thu, May 16, 2024 at 10:54 AM John D. Ament <jo...@apache.org>
wrote:
> Completely naive question since I'm not familiar at all with the
> technologies. I wanted to demonstrate using Iceberg files as a way to
> ingest lots of data and persist it to S3. It seems like it can do this,
> but I have a feeling I need tools like Spark to do it. is that true? or
> can I hook it up to just a generic REST API that loads the data in?
>
> I have a feeling Spark will be part of the final solution, but I don't
> want to get into that complexity quite yet, and it's not clear from the
> docs.
>
> Also apologies for posting this to your dev@ list. No user(s)@ list.
>
> John
>
Re: Can I get started with a plain java app?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
I'm working on a Camel Component that use purely java to ingest data
(and also scan table):
https://github.com/jbonofre/iceberg/blob/CAMEL/camel/camel-iceberg/src/main/java/org/apache/iceberg/camel/IcebergProducer.java
NB: it's still a WIP, I'm working on it and I will share a proposal on
the mailing list next week.
Regards
JB
On Thu, May 16, 2024 at 7:53 PM John D. Ament <jo...@apache.org> wrote:
>
> Completely naive question since I'm not familiar at all with the technologies. I wanted to demonstrate using Iceberg files as a way to ingest lots of data and persist it to S3. It seems like it can do this, but I have a feeling I need tools like Spark to do it. is that true? or can I hook it up to just a generic REST API that loads the data in?
>
> I have a feeling Spark will be part of the final solution, but I don't want to get into that complexity quite yet, and it's not clear from the docs.
>
> Also apologies for posting this to your dev@ list. No user(s)@ list.
>
> John