You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Ajantha Bhat <aj...@gmail.com> on 2024/12/10 10:15:15 UTC
[Discussion] Maintain vendor neutrality on the quickstart page
The quickstart <https://iceberg.apache.org/spark-quickstart/> page is a
critical touchpoint for new users and plays a key role in driving project
adoption.
Currently, it references *tabulario/spark-iceberg* and
*tabulario/iceberg-rest*
We’ve already replaced *tabulario/iceberg-rest* with the
community-maintained Docker image, *apache/iceberg-rest-fixture*, based on
the REST TCK fixture.
However, *tabulario/spark-iceberg* seems outdated, and doesn't use the
latest Iceberg version.
To enhance the user experience and keep the quickstart aligned with project
standards, I suggest hosting it either under the /docker folder in the
Iceberg repository
or as a subproject called *apache/iceberg-playground* where users can
contribute to maintain other docker images.
The quickstart page should ideally reference images maintained by the
community rather than vendor-specific open-source projects.
Thoughts?
- Ajantha
Re: [Discussion] Maintain vendor neutrality on the quickstart page
Posted by Yong Zheng <yz...@apache.org>.
I also think that is a good idea and interested to take this up. But before I do so, here are couple clarifications i would like. Based on what I recalled from tabulario/spark-iceberg, it has couple additional things are not there compared to official spark image:
1. couple additional python libs (e.g. jupyter notebook, pyiceberg, matplotlib, etc.)
- for this one, if we switched to official spark image, do we want to build our custom image on top of that and also include those? they are not technically required as we don't mention jupyter notebook on our page nor README.md
2. couple notebooks published by Tabular earlier for simulating various behaviors with iceberg
- if we still want to use those notebooks, we will need to create a form of those or we can do our own?
3. there is an ipython line cell magic used in jupyter notebook
- same as above, only needed if we want to use the same set of notebooks.
4. they included IJava and scala kernels in jupyter notebook
- same as above, only needed if we want to use the same set of notebooks.
5. couple parquet files for dummy dataset
- same as above, only needed if we want to use the same set of notebooks.
That being said, if I would want to take this, should we remove the dependency on jupyter notebook and those extra notebooks/kernerl/libs etc.?
On 2024/12/10 11:29:06 Fokko Driesprong wrote:
> Yes, that's exactly my motivation (sorry for not stating this explicitly
> earlier). Looking at the fact that the quickstart is currently outdated, I
> would be reluctant to introduce additional Docker images and/or
> repositories, since we need to update those as well.
>
> Kind regards,
> Fokko
>
> Op di 10 dec 2024 om 11:48 schreef Ajantha Bhat <aj...@gmail.com>:
>
> > That's a good suggestion Fokko.
> > It would avoid maintaining one more docker image. We can update the
> > quickstart to use the docker image provided by Spark.
> >
> > - Ajantha
> >
> > On Tue, Dec 10, 2024 at 4:08 PM Fokko Driesprong <fo...@apache.org> wrote:
> >
> >> Hey Ajantha,
> >>
> >> Thanks for bringing this up, we should both remove the vendor reference
> >> and bring this back up to date. My preference would be to rely on the Spark
> >> image <https://hub.docker.com/r/apache/spark> provided by the Apache
> >> Spark project, similar to what we do for the Hive
> >> <https://iceberg.apache.org/hive-quickstart/> quickstart. We should be
> >> able to load all the Iceberg-specific JARs through the
> >> spark.jars.packages configuration
> >> <https://spark.apache.org/docs/3.5.1/configuration.html>.
> >>
> >> Kind regards,
> >> Fokko
> >>
> >> Op di 10 dec 2024 om 11:16 schreef Ajantha Bhat <aj...@gmail.com>:
> >>
> >>> The quickstart <https://iceberg.apache.org/spark-quickstart/> page is a
> >>> critical touchpoint for new users and plays a key role in driving project
> >>> adoption.
> >>> Currently, it references *tabulario/spark-iceberg* and
> >>> *tabulario/iceberg-rest*
> >>>
> >>> We’ve already replaced *tabulario/iceberg-rest* with the
> >>> community-maintained Docker image, *apache/iceberg-rest-fixture*, based
> >>> on the REST TCK fixture.
> >>>
> >>> However, *tabulario/spark-iceberg* seems outdated, and doesn't use the
> >>> latest Iceberg version.
> >>> To enhance the user experience and keep the quickstart aligned with
> >>> project standards, I suggest hosting it either under the /docker folder in
> >>> the Iceberg repository
> >>> or as a subproject called *apache/iceberg-playground* where users can
> >>> contribute to maintain other docker images.
> >>>
> >>> The quickstart page should ideally reference images maintained by the
> >>> community rather than vendor-specific open-source projects.
> >>>
> >>> Thoughts?
> >>>
> >>> - Ajantha
> >>>
> >>
>
Re: [Discussion] Maintain vendor neutrality on the quickstart page
Posted by Fokko Driesprong <fo...@apache.org>.
Yes, that's exactly my motivation (sorry for not stating this explicitly
earlier). Looking at the fact that the quickstart is currently outdated, I
would be reluctant to introduce additional Docker images and/or
repositories, since we need to update those as well.
Kind regards,
Fokko
Op di 10 dec 2024 om 11:48 schreef Ajantha Bhat <aj...@gmail.com>:
> That's a good suggestion Fokko.
> It would avoid maintaining one more docker image. We can update the
> quickstart to use the docker image provided by Spark.
>
> - Ajantha
>
> On Tue, Dec 10, 2024 at 4:08 PM Fokko Driesprong <fo...@apache.org> wrote:
>
>> Hey Ajantha,
>>
>> Thanks for bringing this up, we should both remove the vendor reference
>> and bring this back up to date. My preference would be to rely on the Spark
>> image <https://hub.docker.com/r/apache/spark> provided by the Apache
>> Spark project, similar to what we do for the Hive
>> <https://iceberg.apache.org/hive-quickstart/> quickstart. We should be
>> able to load all the Iceberg-specific JARs through the
>> spark.jars.packages configuration
>> <https://spark.apache.org/docs/3.5.1/configuration.html>.
>>
>> Kind regards,
>> Fokko
>>
>> Op di 10 dec 2024 om 11:16 schreef Ajantha Bhat <aj...@gmail.com>:
>>
>>> The quickstart <https://iceberg.apache.org/spark-quickstart/> page is a
>>> critical touchpoint for new users and plays a key role in driving project
>>> adoption.
>>> Currently, it references *tabulario/spark-iceberg* and
>>> *tabulario/iceberg-rest*
>>>
>>> We’ve already replaced *tabulario/iceberg-rest* with the
>>> community-maintained Docker image, *apache/iceberg-rest-fixture*, based
>>> on the REST TCK fixture.
>>>
>>> However, *tabulario/spark-iceberg* seems outdated, and doesn't use the
>>> latest Iceberg version.
>>> To enhance the user experience and keep the quickstart aligned with
>>> project standards, I suggest hosting it either under the /docker folder in
>>> the Iceberg repository
>>> or as a subproject called *apache/iceberg-playground* where users can
>>> contribute to maintain other docker images.
>>>
>>> The quickstart page should ideally reference images maintained by the
>>> community rather than vendor-specific open-source projects.
>>>
>>> Thoughts?
>>>
>>> - Ajantha
>>>
>>
Re: [Discussion] Maintain vendor neutrality on the quickstart page
Posted by Ajantha Bhat <aj...@gmail.com>.
That's a good suggestion Fokko.
It would avoid maintaining one more docker image. We can update the
quickstart to use the docker image provided by Spark.
- Ajantha
On Tue, Dec 10, 2024 at 4:08 PM Fokko Driesprong <fo...@apache.org> wrote:
> Hey Ajantha,
>
> Thanks for bringing this up, we should both remove the vendor reference
> and bring this back up to date. My preference would be to rely on the Spark
> image <https://hub.docker.com/r/apache/spark> provided by the Apache
> Spark project, similar to what we do for the Hive
> <https://iceberg.apache.org/hive-quickstart/> quickstart. We should be
> able to load all the Iceberg-specific JARs through the spark.jars.packages
> configuration <https://spark.apache.org/docs/3.5.1/configuration.html>.
>
> Kind regards,
> Fokko
>
> Op di 10 dec 2024 om 11:16 schreef Ajantha Bhat <aj...@gmail.com>:
>
>> The quickstart <https://iceberg.apache.org/spark-quickstart/> page is a
>> critical touchpoint for new users and plays a key role in driving project
>> adoption.
>> Currently, it references *tabulario/spark-iceberg* and
>> *tabulario/iceberg-rest*
>>
>> We’ve already replaced *tabulario/iceberg-rest* with the
>> community-maintained Docker image, *apache/iceberg-rest-fixture*, based
>> on the REST TCK fixture.
>>
>> However, *tabulario/spark-iceberg* seems outdated, and doesn't use the
>> latest Iceberg version.
>> To enhance the user experience and keep the quickstart aligned with
>> project standards, I suggest hosting it either under the /docker folder in
>> the Iceberg repository
>> or as a subproject called *apache/iceberg-playground* where users can
>> contribute to maintain other docker images.
>>
>> The quickstart page should ideally reference images maintained by the
>> community rather than vendor-specific open-source projects.
>>
>> Thoughts?
>>
>> - Ajantha
>>
>
Re: [Discussion] Maintain vendor neutrality on the quickstart page
Posted by Fokko Driesprong <fo...@apache.org>.
Hey Ajantha,
Thanks for bringing this up, we should both remove the vendor reference and
bring this back up to date. My preference would be to rely on the Spark
image <https://hub.docker.com/r/apache/spark> provided by the Apache Spark
project, similar to what we do for the Hive
<https://iceberg.apache.org/hive-quickstart/> quickstart. We should be able
to load all the Iceberg-specific JARs through the spark.jars.packages
configuration <https://spark.apache.org/docs/3.5.1/configuration.html>.
Kind regards,
Fokko
Op di 10 dec 2024 om 11:16 schreef Ajantha Bhat <aj...@gmail.com>:
> The quickstart <https://iceberg.apache.org/spark-quickstart/> page is a
> critical touchpoint for new users and plays a key role in driving project
> adoption.
> Currently, it references *tabulario/spark-iceberg* and
> *tabulario/iceberg-rest*
>
> We’ve already replaced *tabulario/iceberg-rest* with the
> community-maintained Docker image, *apache/iceberg-rest-fixture*, based
> on the REST TCK fixture.
>
> However, *tabulario/spark-iceberg* seems outdated, and doesn't use the
> latest Iceberg version.
> To enhance the user experience and keep the quickstart aligned with
> project standards, I suggest hosting it either under the /docker folder in
> the Iceberg repository
> or as a subproject called *apache/iceberg-playground* where users can
> contribute to maintain other docker images.
>
> The quickstart page should ideally reference images maintained by the
> community rather than vendor-specific open-source projects.
>
> Thoughts?
>
> - Ajantha
>